Hi,

we have encountered problems with encoding of http post requests after upgrading client tools from version 7.23 to 7.27.

After upgrade, API server started to respond with unspecified error. We don't have access to server, so I tried to send the same request to https://webhook.site. There I can see, that all characters with diacritics (czech language) are replaced by question marks. After removing all diacritics from request, it was successfully accepted by original API server.

We use windows-1250 encoding. Content-type of request should be application/json; charset=utf-8.

Could you please help me to solve it?

Here is code of our method sending request:

Lparameters lcUrl, loRequest, lcPropertyNameOverrides

Do wwhttp
Do wwJsonSerializer

loHttp			= Createobject("wwHttp")
loHttp.cContentType	= "application/json"
lcKey			= Strconv(Trim(This.cUserName) + ":" + Trim(This.cPassword), 13)
loHttp.AddHeader("Authorization", "Basic " + lcKey)

loClient = Createobject("wwJsonServiceClient")
loClient.createwwhttp(loHttp)

If Type("m.lcPropertyNameOverrides") = "C" And Not Empty(m.lcPropertyNameOverrides)
	loSer				= Createobject("wwJsonSerializer")
	loSer.PropertyNameOverrides	= m.lcPropertyNameOverrides
	loClient.CreateSerializer(loSer)
Endif

m.lcHttpVerb = "POST"
loResponse	 = loClient.CallService(lcUrl, m.loRequest, m.lcHttpVerb)

Some example of problematic characters: í, á, ž, c, ...

Thank you.

You're using wwJsonServiceClient I'm guessing?

There was a change in that class to automatically encode/decode content to and from UTF-8 which was not done previously - it was a bug that cause ? content in extended characters.

So if your data returned from the server is not UTF-8 encoded this change will cause a problem (in which case I'd argue there's something wrong with the service since it should return UTF-8).

You can set the loService.lNoUtf8Decoding = .T. flag to force the service client to not UTF-8 decode the response.

Kind of surprised nobody had reported this issue with UTF-8 encoding not being automatic! With English text you don't run into many issues which is likely why I missed this, but in Euro languages with special high order characters that problem shows up a lot more prominently with extended characters that'll show ? when deserialized.

+++ Rick ---

Maybe I didn't write it clearly. I don't have problem with processing response. Main problem is in request. In the message I am sending to API server.

Now I tried not to use wwJsonServiceClient. Instead of it I used wwJsonSerializer to create json string and then I used STRCONV() to manually convert it from Windows 1250 to UTF-8. Then I sent it with wwHttp and it finally works.

Do wwhttp
Do wwJsonSerializer

loSer  = Createobject("wwJsonSerializer")
lcData = loSer.serialize(m.loRequest)

* CONVERTING WINDOWS-1250 to UTF-8
lcData  = Strconv(Strconv(m.lcData,1,1029),9,1029)

loHttp				= Createobject("wwHttp")
loHttp.cContentType	= "application/json"
lcKey = Strconv(Trim(m.lcUserName) + ":" + Trim(m.lcPassword), 13)
loHttp.AddHeader("Authorization", "Basic " + lcKey)

lcResponse = loHttp.post(m.lcUrl, m.lcData)

So I can make it work with Client Tools 7.27, but it is quite problem for our existing code which is using wwJsonServiceClient and which was ok without using STRCONV() with Client Tools 7.23.

Is it rather bug or new feature or behavior? Should we update our existing code or it could be somehow solved by Client Tools? Is there any way to make conversion using wwJsonServiceClient?

Thank you.

Yup - you're spot on. The request was not UTF-8 encoded, but not quite sure how that happened. I'm looking through my Git history and I don't see that value changed, except that perhaps an old version was used in wwClient, compared to the Web Connection libraries.

What has changed in the recent version was that response data is auto-decoded by default. IOW, previously there was no encoding/decoding happening at all, but decoding is happening now (by default).

IAC, I fixed this when I originally looked at your message so now both inbound and outbound are encoding/decoding and put back the encoding for the outbound data to UTF-8.

I've updated the wwClient packages (both registered and shareware) with updated versions of the wwJsonSerializer.prg file that includes the UTF-8 encoding.

https://client-tools.west-wind.com

If you want to fix this without updating it's this code wwJsonServiceClient.CallService():

*** Serialize single value/object
IF !IsNullOrEmpty(lvPostData)
    loSer = THIS.CreateSerializer()
    lcJson = loSer.Serialize(lvPostData)
    loHttp.cContentType = "application/json; charset=utf-8"	
    loHttp.AddPostKey(STRCONV(lcJson,9))  && <--- THIS
ENDIF

+++ Rick ---

Thank you. It seems to be ok.