Hi All,
About a year ago, I added a vendor price list update feature to my VFP9 Desktop App using WWWC Client Tools. It submits an XML Request to the vendor's server and the vendor returns the price list updates in XML. It worked fine for several months, and then started failing. To my knowledge, nothing changed on my end.
I spoke with the vendor and they indicated my upload contains an invalid "Majic Character" in the first position based on the following error message returned from their server:
{"error":{"code":500,"name":"Error","message":"Non-whitespace before first tag.\nLine: 0\nColumn: 1\nChar: \u001f","stack":["Error: Non-whitespace before first tag.","Line: 0","Column: 1","Char: \u001f","at error (/node_modules/sax/lib/sax.js:651:10)
Here's what the vendor said:
"I think we found the issue, when you submit your XML, it seems you have a bad characters '001f' at the beginning. It is look like a magic character..
Message : "Non-whitespace before first tag.\nLine: 0\nColumn: 1\nChar: \u001f"
Hereafter the answer I got from one of our specialist "The character 001f is passed as the first character, which is not allowed. You do not see it because it is an invisible character, and you will not get it either with a copy / paste of XML."
Maybe your variable who contains the XML is not initialized before and for any reason that character has been place in your XML."
Has anyone seen this issue? The vendor said my XML is fine. I checked and I do initialize my lcXML = '' variable. Any idea why or how this 001f character is getting placed into the upload request?
TIA,
Steve
UPDATE - I may have found the issue. I disabled GZip and it started working. 😕
The API Spec says the server supports GZip, and this is how it was originally setup. I disabled GZip as a test and boom. Time to talk to the vendor again.
Thanks,
Steve
Not sure if this is related to your problem or not but I had similar issue with one of my .html files which was the product of two .xml + .xsl files.
It used to be saved and treated as "UTF-8 with BOM" and the unwanted BOM character (") used to come up on my page.
After hours of time spending on that, it got fixed as soon as I moved my <script> tag from <head> tag to the <body> tag!
Hope this can save some time for someone in future.
All the best,
Kathy
I think you're running into the UTF-8 BOM which is a pre-amble that indicates to readers that the file is encoded as UTF-8 (or some other encoding like various Unicode flavors).
If you the raw file as is the BOM will get sent and that likely causes the problem for result data. So, if you do this you can use File2Var() from the client tools (in wwutils.prg) it automatically strips off the BOM. You can look at the code to see how to do that - basically you check for common BOM values in the string and strip them off.
I guess some of this depends on how you are sending the data, but my guess is you're using FILETOSTR() and that doesn't strip off any BOM, but just gives you the raw file.
+++ Rick ---
Yes, I am using FILETOSTR(). I will check out the File2Var() function.
When I disable GZip on the upload file, it works fine. Any theory on why it works with GZip disabled? In general, what happens if I GZip the upload but GZip is disabled on the server?
Thanks again,
Steve
GZip likely has nothing to do with it if it's what you describe.
The BOM gets added by editors when they save. It 'worked' before because you probably edited the documents in your editor of choice, but then somebody else or you with a different editor edited the documents and the BOM got added. FILETOSTR() doesn't strip the BOM so there's your invalid file content then. You need to fix the BOM issue. GZip on or off is likely just coincidental.
+++ Rick ---
Rick,
Is there any way to see the UTF-8 BOM using Fiddler? I incorporated code to strip out the UTF-8 BOM info when present, but it still fails unless GZip is disabled. Don't really see anything in Fiddler that looks like the BOM.
TIA,
Steve
What exactly are you GZipping?
If you're looking in Fiddler you have to look at the decoded content after GZip is removed (Fiddler will let you do that in the UI)...
+++ Rick ---
I thought that perhaps Visual Studio, which I used to edit the XML Template File, may have embedded something weird into the file. So, yesterday I edited it using Notepad to see if that would fix this issue. It did not.
Editing the file with Notepad would clear out any UTF-8 DOM, wouldn't it?
Thanks for your feedback!
Steve
No it wouldn't.
Most editors have a way to set the file encoding. Choose UTF-8 (No BOM) or something like that. In Visual Studio it's Save As... then Save button dropdown to Save with Encoding and choose (I think) Unicode (UTF-8 without signature).
NotePad preserves the original encoding I believe, but there too you can save with a specific encoding.
Again you shouldn't do this here though. Whatever file processing you use to retrieve the files should be smart enough to strip off the BOM if it has one. FoxPro unfortunately doesn't because it wasn't designed with Unicode and UTF-8 in mind and it treats files like ANSI double byte files - raw content basically. So whatever file routine you use should look at that. Again File2Var() automatically strips off a BOM UTF-8 and Unicode BOMs when reading files into memory.
+++ Rick ---