Rick,

We use the wwJSONSerializer class to serialize cursors into JSON strings. Normally works great. However, we have a memo field with a base64 string that is erroring out. I've tracked it down to the wwJsonSerializer.writeString() method.

The line that's erroring out is: lcOutput = REPLICATE(CHR(0),LEN(lcValue) * 6 + 3)

The error message is: String is too long to fit. Proc: writestring Line No: 254 Line Contents: lcOutput = REPLICATE(CHR(0),LEN(lcValue) * 6 + 3)

The problem is when you do LEN(lcValue) you get 7750323 and then you are multiplying that by 6 which gives you 46,501,938 which is well beyond VFP's string max length.

I can manually get in and do something like:

FUNCTION WriteString(lcValue)
LOCAL lnK, lcResult, lcValue,lcOutput

IF THIS.TrimStringValues
   lcValue = TRIM(m.lcValue)
ENDIF

IF EMPTY(lcValue)
	this.cOutput = this.cOutput + [""]
	RETURN 
ENDIF	

*** Optimized for perf with C code in wwIPStuff.dll
*** JsonEncodeString
LOCAL lcOutput 
* Ricks original Line:
* lcOutput = REPLICATE(CHR(0),LEN(lcValue) * 6 + 3)
*** Start My 'Hack' ***
IF LEN(lcValue) *6 +3 >= 1500000
     lcOutput = REPLICATE(CHR(0),1500000)
ELSE
     lcOutput = REPLICATE(CHR(0),LEN(lcValue) * 6 + 3)
ENDIF
*** End Hack ***

lnPointer = JsonEncodeString(lcValue,@lcOutput)
this.cOutput = this.cOutput +  WinApi_NullString(@lcOutput)

ENDFUNC
* WriteString

But I'm not sure if this breaks anything? The other problem is, once I apply an update from you I lose my hack. Any thoughts on the best way to tackle this?

Thanks,

Scott

The value is probably overly agressive. This basically accounts for 6 characters for each string character (which are unicode encoded characters) and which assumes every character would be encoded. That's unlikely to ever happen even less so with FoxPro's ANSI character set which doesn't include extended characters.

The problem is the buffer has to be big enough to accommodate the target string - if it's not big enough it'll bomb. So for small strings this is probably a valid calculation because you could have 2 unicode or control characters. For larger strings the ratio probably be easily dropped to or even 2 times.

Your idea of just using the max buffer if it's bigger is good - at least that way we're maxing it out. Note your sample uses 1.5 megs not 15 😃

There are also a few tricks we could do with this:

*** Works  - max size ~16.75 megs
lcOutput = REPLICATE(CHR(0), 16750000)
? LEN(lcOutput)

*** This also works to create a string as big as it needs to be: 20,000,000  (20megs)
lcOutput = REPLICATE(CHR(0),10000000)
lcOutput = lcOutput + REPLICATE(CHR(0),10000000)
? LEN(lcOutput)

Unfortunately though the JSON Serializer uses an internal this.cOutput property which cannot exceed 16megs so the max size for full JSON is 16megs.

Anyway I've updated the code to do the following:

FUNCTION WriteString(lcValue)
LOCAL lnK, lcResult, lcValue,lcOutput

IF THIS.TrimStringValues
   lcValue = TRIM(m.lcValue)
ENDIF

IF EMPTY(lcValue)
	this.cOutput = this.cOutput + [""]
	RETURN 
ENDIF	

*** Optimized for perf with C code in wwIPStuff.dll
*** JsonEncodeString
LOCAL lcOutput, lnBufferSize
lnBufferSize = LEN(lcValue) * 5 + 3
IF (lnBufferSize < 16750000)
   lcOutput = REPLICATE(CHR(0),lnBufferSize)
ELSE 
    *** Set to max string size and hope for the best!
    *** The calculated value is overly pessimistic so
    *** as long as the output is under 16 megs it'll still work
    lcOutput = REPLICATE(CHR(0),16750000)
ENDIF	

PUBLIC __JsonEncodeStringAPI
IF !__JsonEncodeStringAPI
	DECLARE INTEGER JsonEncodeString IN wwipstuff.dll string  json,string@  output
	__JsonEncodeStringAPI = .T.
ENDIF	

try
   *** we'll now crash if the buffer is overrun by the output
   lnPointer = JsonEncodeString(lcValue,@lcOutput)
CATCH 
   ERROR "JSON Encoding failed: Most likely the string output exceeds 16 megs."
ENDTRY

this.cOutput = this.cOutput  + WinApi_NullString(@lcOutput)
ENDFUNC
* WriteString

This now works:

DO wwJsonSerializer

*** 16.75 megs of data
lcData = REPLICATE("1234567890", 1000000)
lcData = lcData + REPLICATE("1234567890", 675000) 

loSer = CREATEOBJECT("wwJsonSerializer")
lcResult = loSer.Serialize(lcData)

*** 16,750,002 bytes (quotes are the 2)
? LEN(lcResult)

All this said it's a really bad idea to create massive JSON documents that are multi-megabytes in size because parsing those documents is going to be slow-ish. I guess it's a good thing that we can't create larger documents in Fox, although it's might actually be possible to read larger documents since the actual processing for JSON is done in .NET and we simply can read JSON from a file and pass it to the parser.

+++ Rick ---

I also added some additional check code into the JSON API call so it doesn't overrun the buffer. If you pass to small of a buffer and a large input string that would fail with a buffer overrun.

The update will capture that explicitly by checking the size of the output buffer before assigning the generated string to it. Either way larger than 16.75 megs encoded final results will fail though but more gracefully.

There's a change that goes along with that which is that the data passed in has to be a spaces rather than CHR(0) since there's no good way to get the size of the buffer off the pointer alone. Passing a " " makes it a string that can be checked for size. if CHR(0) is passed the old behavior still works but if the buffer exceeds 16 megs it'll overrun the buffer just like before which is not good.

+++ Rick ---

Rick,

Yes, once you pointed that out I realized I missed a 0 and only did 1.5 MB instead of 15.

Thanks! Code works great. Appreciate your help on this. I understand the performance issues. Unfortunately, the base64 doc is not something I can get away from right now due to a clients needs.

Thanks,

Scott