Hi Rick,

Looking for some thoughts related to character encoding issues. We do a fair amount of generating CSV files in our WWC app using straightforward VFP string manipulation/textmerge. This can cause pain for our Mac-based users as the upper ASCII characters in Windows don't line up in MacOS. Do you have any magic tricks or advice for dealing with multi-platform character encoding when generating plain, old CSV (text) files?

TIA

If you're dealing with a single charset/codepage, can't you just generate using your local FoxPro char set and then Utf-8 encode afterwards?

If you need multiple character sets at the same time, then your SOL with FoxPro and into pure Unicode translation hell. There are ways to make this work, but it's major pain in the ass.

This is really old, but still relevant:

Using Unicode in Visual FoxPro Web and Desktop Applications

Today you can also use .NET for the Unicode conversions which is a little more approachable.

Regardless, I can't recommend doing any of that because it's so easy to fuck it up if you forget to encode correctly at some point in the data chain.

If you can go from data or files directly, you can perhaps use .NET to read and write out CSV which removes any of the charset limitations as everything is Unicode. You can then write out as UTF-8 which is universally readable. There are several really good OSS .NET CSV libraries available to do this but the key is the data has to come directly in and not from FoxPro to avoid the ?? active charset 'violations'.

+++ Rick ---

Thanks. All the data lives in SQL, and I think part of the longer term solution is to change the default codepage in my SQL DBs away from SQL_Latin1_General_CP1_CI_AS. I am, indeed, in encoding hell.

I remember reading that post. Probably have it bookmarked. 😃 I shall go revisit that to see if it fits my situation.

There's a VFPx CSVProcessor project which I might be able to use.

Probably won't be able to solve the character encoding issue if you're dealing with data coming from FoxPro and it's using native foxpro code to access that data. Unless it uses some other mechanism for getting the data that doesn't involve FoxPro touching the string data, you'll end up with the same issues I describe.

+++ Rick ---

Thanks. All the data lives in SQL, and I think part of the longer term solution is to change the default codepage in my SQL DBs away from SQL_Latin1_General_CP1_CI_AS. I am, indeed, in encoding hell.

Yes ideally you'll want to write the data out as Unicode because that's pretty much what everything else uses. There are flags you can set on the SQL interfaces in FoxPro to write multi-byte data into the database more or less transparently.

That still doesn't solve the multi-character set issue because FoxPro just can't manage more than one character set at the same time.

+++ Rick ---

True. But in theory I can detect if the client request came from a macOS/iOS system, then strconv to the appropriate regionalID/regionalIDType before saving out the file to disk. But that remains to be tested.

That works as long as you stick with specific character sets that are supported. And you have to map charsets to codepages in FoxPro which is also tricky given the limited code page support.

+++ Rick ---