Hi Rick,

I'm working with a new (for me) API call here. I'm running into an issue serializing the response object that's coming back to me.

The response object has the following structure:

After I get the response object, here's the bit of my code where I attempt to interpret that object:

IF NOT EMPTY(.oProxy.cErrorMsg)
  .apiErrorMessage=.oProxy.cErrorMsg
  .logEvent(.apiErrorMessage)
ELSE 
  .oSerializer.formattedOutput=.lMakeDebugLog
  .jsonResetPWReport=.oSerializer.Serialize(.oResponse)
  .oSerializer.PropertyNameOverrides=[]
  m.llRetval=.t.
ENDIF 

In my testing, I am getting back a 200 so no errors in the call are being encountered and I fall into the ELSE. The problem I am encountering is surfacing when I attempt to Serialize the response object. I'm stepping through wwJsonSerializer.writeValue. When it gets to the "time" KVP in the screenshot, lcType is N and lvValue = 1600978541021.00000000 and this is what gets executed:

this.cOutput = this.cOutput + TRANSFORM(lvValue)

After I execute the above line of code, here's the relevant part of what is in this.cOutput:

"time": ***********.**********

In Foxpro-speak, I interpret that as a numeric overflow condition. So then I just tried some good old command line stuff. With execution paused at this point, I executed ?lvValue. This is what echoed to the screen:

Then I executed ?TRANSFORM(lvValue) and numeric overflow:

It's interesting to see the position of the decimal point has shifted compared to the actual value. Then I tried this: ?TRANSFORM(1600981020167.00000000) (Note the values are different from earlier screencaps because I'm running and debugging over and over but the precision appears to be the same.)

For my purposes, I don't really care about that time value, The 2 things I want to get more info from are the data and error collections. But the serialization call in wwJsonSerializer.FormatJson throws an error caught by a try...catch exception in my code. This is the message that is in the exception object:

OLE IDispatch exception code 0 from Newtonsoft.Json: Unexpected character encountered while parsing value: *. Path 'time', line 1, position 302...

Of course, I don't get the output into the property I am trying to populate from the response object. At the moment I'm chalking this up to VFP's funky handling of large numbers. I guess I can assign data and errors to individual objects and try serializing those collections instead but that's a bit of a pita. Any thoughts?

I think that might just be TRANSFORM() not being able to handle the number due to the default format string overflowing. If you print the value ? .oResponse.time what do you get? Maybe try explicitly setting the transform string to something that can encompass the value (ie. more significant numbers before the decimal point).

+++ Rick ----

Spot on there as far as default TRANSFORM behavior is concerned.

?m.this.oresponse.time
?TRANSFORM(m.this.oresponse.time)
?TRANSFORM(m.this.oresponse.time,[9999999999999.99999999])

And here's the screen output:

So the next issue is this TRANSFORM call is inside the framework. Line 242 of wwJsonSerializer.prg (FUNCTION WriteValue). I can modify my local copy for now but I'm sure you will want to decide what you want to do with framework stuff. Speaking of, the changes we made together are planned for 7.16, I think? Will you be releasing that any time soon? I've been sitting on 7.15 thinking you might be doing another release.

I was trying to figure out if there's a reliable way to determine if the value to the right of the decimal point is a non-zero value and it just gets strange when testing with small values. The answer seems to be no.

FYI this is the approach I'm taking with my local copy:

CASE INLIST(lcType,"I","N","F")
  IF lctype="I"
    this.cOutput = this.cOutput + TRANSFORM(lvValue)
  ELSE 
    this.cOutput = this.cOutput + TRANSFORM(lvValue,[9999999999999.99999999])
  ENDIF 

Wow that's kind of distressing that FoxPro's default for Transform can't handle the very large numbers. As a matter of fact that never occurred to me nor have I run into this. But I guess the numbers are extremely large. The sad part is this will be a problem in quite a few places in Web Connection that convert numbers to strings for rendering etc.

FoxPro of course can have numbers of any size since numbers are internally represented as floating point numbers and it seems doubly odd that it fails at the level shown for levels of size, and then falls back to floating point notation. Check this out:

So if the number gets sufficiently large it falls back to floating point syntax (which BTW will happen even if you add the explicit template.

To be fair though - those numbers are immensely large - what are you working on that needs numbers in the Quadrillion range? (looks to me 99,000,000,000,000 works, 100,000,000,000,000 is where it starts failing)

Sorry for the slow followup to your last response. I was out on Monday and playing catch-up.

I don't need that time value. It's just what comes back from this API call. I assume it's some odd Java/Hibernate interpretation of a timestamp.

The conversion to scientific notation isn't unexpected for really large numbers, and in this case not what's happening. There are 8 digits of precision to the right of the decimal point and without the TRANSFORM masking that's where it gets into an overflow state. This is why I was proposing adding the additional logic to the 3 numeric datatypes in the writeValue CASE statement. My bit shouldn't have any effect on straight integer values. In an ideal world there would be some magical mathematical formula/function in VFP to parse out the fraction from the value, and then set the mask to the proper number of decimal places, but I'm too mathematically challenged to figure that out. Having said that, keep playing along here and you'll see that VFP throws away precision based on the size of the integer portion of the value.

If you think the EE bit is fun, try this:

**************************************
*      Program: weirdstuff1.prg
*         Date: 09/29/2020 05:23:14 PM
*  VFP Version: Visual FoxPro 09.00.0000.7423 for Windows
*        Notes: 
**************************************

ACTIVATE SCREEN 
CLEAR 
SET DECIMALS TO 8
SET ALTERNATE TO temp.txt 
SET ALTERNATE ON 
SET CONSOLE OFF

funny=1234567890123.12345678
? [SET DECIMALS TO 8]
? [13 plus 8]
? [funny is ]
?? [1234567890123.12345678]
? [TRANSFORM(funny) is ]
?? TRANSFORM(funny)
? [TRANSFORM(funny) with a mask is ]
?? TRANSFORM(funny,[9999999999999.99999999])
funny=1234567890123.10000000
? [funny is ]
?? [1234567890123.10000000]
? [TRANSFORM(funny) is ]
?? TRANSFORM(funny)
? [TRANSFORM(funny) with a mask is ]
?? TRANSFORM(funny,[9999999999999.99999999])
funny=1234567890123.01000000
? [funny is ]
?? [1234567890123.01000000]
? [TRANSFORM(funny) is ]
?? TRANSFORM(funny)
? [TRANSFORM(funny) with a mask is ]
?? TRANSFORM(funny,[9999999999999.99999999])
funny=1234567890123.00100000
? [funny is ]
?? [1234567890123.00100000]
? [TRANSFORM(funny) is ]
?? TRANSFORM(funny)
? [TRANSFORM(funny) with a mask is ]
?? TRANSFORM(funny,[9999999999999.99999999])
funny=1234567890123.00010000
? [funny is ]
?? [1234567890123.00010000]
? [TRANSFORM(funny) is ]
?? TRANSFORM(funny)
? [TRANSFORM(funny) with a mask is ]
?? TRANSFORM(funny,[9999999999999.99999999])
funny=1234567890123.00001000
? [funny is ]
?? [1234567890123.00001000]
? [TRANSFORM(funny) is ]
?? TRANSFORM(funny)
? [TRANSFORM(funny) with a mask is ]
?? TRANSFORM(funny,[9999999999999.99999999])
funny=1234567890123.00000100
? [funny is ]
?? [1234567890123.00000100]
? [TRANSFORM(funny) is ]
?? TRANSFORM(funny)
? [TRANSFORM(funny) with a mask is ]
?? TRANSFORM(funny,[9999999999999.99999999])
funny=1234567890123.00000010
? [funny is ]
?? [1234567890123.00000010]
? [TRANSFORM(funny) is ]
?? TRANSFORM(funny)
? [TRANSFORM(funny) with a mask is ]
?? TRANSFORM(funny,[9999999999999.99999999])
funny=1234567890123.00000001
? [funny is ]
?? [1234567890123.00000001]
? [TRANSFORM(funny) is ]
?? TRANSFORM(funny)
? [TRANSFORM(funny) with a mask is ]
?? TRANSFORM(funny,[9999999999999.99999999])
?
? [12 plus 8]
funny=123456789012.12345678
? [funny is ]
?? [123456789012.12345678]
? [TRANSFORM(funny) is ]
?? TRANSFORM(funny)
? [TRANSFORM(funny) with a mask is ]
?? TRANSFORM(funny,[999999999999.99999999])
funny=123456789012.10000000
? [funny is ]
?? [123456789012.10000000]
? [TRANSFORM(funny) is ]
?? TRANSFORM(funny)
? [TRANSFORM(funny) with a mask is ]
?? TRANSFORM(funny,[999999999999.99999999])
funny=123456789012.01000000
? [funny is ]
?? [123456789012.01000000]
? [TRANSFORM(funny) is ]
?? TRANSFORM(funny)
? [TRANSFORM(funny) with a mask is ]
?? TRANSFORM(funny,[999999999999.99999999])
funny=123456789012.00100000
? [funny is ]
?? [123456789012.00100000]
? [TRANSFORM(funny) is ]
?? TRANSFORM(funny)
? [TRANSFORM(funny) with a mask is ]
?? TRANSFORM(funny,[999999999999.99999999])
funny=123456789012.00010000
? [funny is ]
?? [123456789012.00010000]
? [TRANSFORM(funny) is ]
?? TRANSFORM(funny)
? [TRANSFORM(funny) with a mask is ]
?? TRANSFORM(funny,[999999999999.99999999])
funny=123456789012.00001000
? [funny is ]
?? [123456789012.00001000]
? [TRANSFORM(funny) is ]
?? TRANSFORM(funny)
? [TRANSFORM(funny) with a mask is ]
?? TRANSFORM(funny,[999999999999.99999999])
funny=123456789012.00000100
? [funny is ]
?? [123456789012.00000100]
? [TRANSFORM(funny) is ]
?? TRANSFORM(funny)
? [TRANSFORM(funny) with a mask is ]
?? TRANSFORM(funny,[999999999999.99999999])
funny=123456789012.00000010
? [funny is ]
?? [123456789012.00000010]
? [TRANSFORM(funny) is ]
?? TRANSFORM(funny)
? [TRANSFORM(funny) with a mask is ]
?? TRANSFORM(funny,[999999999999.99999999])
funny=123456789012.00000001
? [funny is ]
?? [123456789012.00000001]
? [TRANSFORM(funny) is ]
?? TRANSFORM(funny)
? [TRANSFORM(funny) with a mask is ]
?? TRANSFORM(funny,[999999999999.99999999])
?
SET ALTERNATE TO
SET ALTERNATE OFF 
MODIFY FILE temp.txt NOWAIT 
SET CONSOLE ON

And here's the output:

SET DECIMALS TO 8
13 plus 8
funny is 1234567890123.12345678
TRANSFORM(funny) is 1234567890123.12400000
TRANSFORM(funny) with a mask is 1234567890123.12400000
funny is 1234567890123.10000000
TRANSFORM(funny) is 1234567890123.10000000
TRANSFORM(funny) with a mask is 1234567890123.10000000
funny is 1234567890123.01000000
TRANSFORM(funny) is 1234567890123.01000000
TRANSFORM(funny) with a mask is 1234567890123.01000000
funny is 1234567890123.00100000
TRANSFORM(funny) is 1234567890123.00100000
TRANSFORM(funny) with a mask is 1234567890123.00100000
funny is 1234567890123.00010000
TRANSFORM(funny) is 1234567890123
TRANSFORM(funny) with a mask is 1234567890123.00000000
funny is 1234567890123.00001000
TRANSFORM(funny) is 1234567890123
TRANSFORM(funny) with a mask is 1234567890123.00000000
funny is 1234567890123.00000100
TRANSFORM(funny) is 1234567890123
TRANSFORM(funny) with a mask is 1234567890123.00000000
funny is 1234567890123.00000010
TRANSFORM(funny) is 1234567890123
TRANSFORM(funny) with a mask is 1234567890123.00000000
funny is 1234567890123.00000001
TRANSFORM(funny) is 1234567890123
TRANSFORM(funny) with a mask is 1234567890123.00000000

12 plus 8
funny is 123456789012.12345678
TRANSFORM(funny) is 123456789012.12350000
TRANSFORM(funny) with a mask is 123456789012.12350000
funny is 123456789012.10000000
TRANSFORM(funny) is 123456789012.10000000
TRANSFORM(funny) with a mask is 123456789012.10000000
funny is 123456789012.01000000
TRANSFORM(funny) is 123456789012.01000000
TRANSFORM(funny) with a mask is 123456789012.01000000
funny is 123456789012.00100000
TRANSFORM(funny) is 123456789012.00100000
TRANSFORM(funny) with a mask is 123456789012.00100000
funny is 123456789012.00010000
TRANSFORM(funny) is 123456789012.00010000
TRANSFORM(funny) with a mask is 123456789012.00010000
funny is 123456789012.00001000
TRANSFORM(funny) is 123456789012
TRANSFORM(funny) with a mask is 123456789012.00000000
funny is 123456789012.00000100
TRANSFORM(funny) is 123456789012
TRANSFORM(funny) with a mask is 123456789012.00000000
funny is 123456789012.00000010
TRANSFORM(funny) is 123456789012
TRANSFORM(funny) with a mask is 123456789012.00000000
funny is 123456789012.00000001
TRANSFORM(funny) is 123456789012
TRANSFORM(funny) with a mask is 123456789012.00000000

With a 13 digit integer, VFP appears to round to 3 digits of precision. With a 12 digit integer it rounds to 4 places. I haven't tried this with 11 or 10 digit integer values but my assumption is it will keep adding precision to the decimal portion of the value.

Since the overflow causes the serialization to fail, how would you suggest handling it in the context of what the framework does? This is obviously an edge case. I can avoid the problem using my writeValue kludge or just avoid serializing the response object as all I'm really interested in with this response is the http return code. But if I do run into a use case where I actually need to serialize the response, you're the final authority on the framework. 😃

The real issue that essentially there's a bug in the FoxPro Transform() function which doesn't support all numbers that are valid in its default format string.

As you say the workaround is hardcoding a format string - which I suppose isn't going to have big overhead although I do think that FoxPro probably optimizes the default string formatting heavily.

The issue in WWWC is that Transform(value) is used extensively throughout the framework to generically render content - it's not just in the JSON serializer, but just about any output function for the HTML features uses that as does the template processor etc.

So this may bite in lots of other places.

I think this should definitely be fixed in the JSON parser because it's isolated and there the value really matters. For others though...

+++ Rick ---

After taking another look, it also seems that the problem is a rounding error - basically a number that rounds up and rolls over to a new digit count fails but works if passed directly.

Check this out:

? TRANSFORM(99999999999999999)   && fails
? TRANSFORM(100000000000000000)   && works  - even tho this has more digits!
? TRANSFORM(99999999999999999,"999999999999999999")   && also works

So it looks to me what's happening here is that the very large number rounds up by one and rolls over to a larger digit count and then fails to render. But if you actually use that same number that it rounds up to it renders just fine in Transform(). It almost looks like FoxPro is pre-calculating how wide the format string needs to be, and when it rounds up it fails because it's a digit short.

Argh. Ugly.

Here's some more craziness. Try this:

? LTRIM(TRANSFORM(99999999999999,"99999999999999999999999999999999999999999.9999") ) 
? LTRIM(TRANSFORM(999999999999999,"99999999999999999999999999999999999999999.9999") ) 
? LTRIM(TRANSFORM(9999999999999999,"99999999999999999999999999999999999999999.9999") ) 
? LTRIM(TRANSFORM(99999999999999988,"99999999999999999999999999999999999999999.9999") ) 
? LTRIM(TRANSFORM(99999999999999900,"99999999999999999999999999999999999999999.9999") ) 

Notice it's rounding up, not just by a single digit from 99, but even from a number like 88 (which is not even close to roll over).

This is seriously messed up FoxPro behavior.

+++ Rick ---

Yeah. I have vague memories of Christof(?) writing an article back in the day describing VFP's troubles with really large/small numbers and precision. I wonder if Chen has done anything with this engine bug? 😃

I don't know how much time you've spent screwing around with this. Were you ever able to get VFP to echo a value with more than 4 digits of precision? OK. I couldn't help myself. Here are the results with an 8x8. (I did the whole series from 11, 01, 9 but will spare you the details.)

8 plus 8
funny is 12345678.12345678
TRANSFORM(funny) is 12345678.12345678
TRANSFORM(funny) with a mask is 12345678.12345678
funny is 12345678.10000000
TRANSFORM(funny) is 12345678.10000000
TRANSFORM(funny) with a mask is 12345678.10000000
funny is 12345678.01000000
TRANSFORM(funny) is 12345678.01000000
TRANSFORM(funny) with a mask is 12345678.01000000
funny is 12345678.00100000
TRANSFORM(funny) is 12345678.00100000
TRANSFORM(funny) with a mask is 12345678.00100000
funny is 12345678.00010000
TRANSFORM(funny) is 12345678.00010000
TRANSFORM(funny) with a mask is 12345678.00010000
funny is 12345678.00001000
TRANSFORM(funny) is 12345678.00001000
TRANSFORM(funny) with a mask is 12345678.00001000
funny is 12345678.00000100
TRANSFORM(funny) is 12345678.00000100
TRANSFORM(funny) with a mask is 12345678.00000100
funny is 12345678.00000010
TRANSFORM(funny) is 12345678.00000010
TRANSFORM(funny) with a mask is  12345678.00000010
funny is 12345678.00000001
TRANSFORM(funny) is 12345678.00000001
TRANSFORM(funny) with a mask is 12345678.00000001

With 8 digits on either side of the decimal point, TRANSFORM behaves with and without the masking and there's no funny rounding things going. As suspected, each digit I removed from the integer side made the rounding error smaller and smaller.

Unfortunately none of this really leaves us with any clean solutions to the problem of very large numbers. Because no matter how you slice it there will be issues (at minimum with a rounding error). Worse, using a generic format string that works with everything wastes a lot of characters making JSON payloads much bigger than they have to be (ie. 1.0000 is a lot more bytes than 1). So I'm not actually so sure I actually want to address this.

Maybe add a some sort fo size filter. IF num > x (some large number) then use the explicit format string. Otherwise go with the default? This would be reasonably efficient still (just a simple value check which Fox is good/fast at). Still not quite sure what the format string should be even in that case:

Something like:

TRANSFORM(lnValue,"999999999999999999999.9999") 

+++ Rick ---

BTW a fellow name John Ryan is doing a session right now on VFPA and apparently Chen has fixed this floating point math problem. Not sure if that applies to how transform works but the precision issue itself has been patched from the looks of it.

He's also apparently fixed the handling of VARCHAR(max) mapping when using current ODBC drivers.

While it appears the actual floating point math precision problem itself is fixed, it doesn't look like the TRANSFORM function itself has been fixed.