Hi Rick,

Have not had to do many changes since you implemented large file handling a couple years ago. But had a problem this morning when user tried to download a large (240mb) file. Checked their logs and they have only had about 5 instances of this in the past year - so not that common.

The code fails when it hits this line in your code:

tcBuffer = lcBuffer

It was also failing at times on my code back when I tried to pull out the data between 2 tags in the string (the string is an xml string with the actual file contents between the tags).

.icData = STREXTRACT(.icResultString,"<data>","</data>",1,5) 

The actual error is #43, out of memory.

Before I went to the work of seeing if I can change the code to write to a file and then extract the data, I wondered if this is even related to the long string problem. I have also not traced through your code to see if outputting to file would solve my problem (I would still have to pick up the file and get the data into another file).

I also tried wrapping my code in TRY/CATCH and then just alerting if the document is too large to be downloaded - the users have other ways they can download if too large. This is what I have done on uploads - I read the file system size of the file and just alert if over 96 mb that they need to upload via a browser. When I tried that though, it choked on your code first - so I would need to alter your code to add a TRY/CATCH which I really did not want to do.

Thanks, Albert

If you're downloading large files you shouldn't be streaming to memory but go directly to file. Anything larger than 16mb is likely going to fail to download to memory or when you try to use it to do anything but write it to disk. By streaming to file you're not pulling the entire download into memory but rather only a small buffer that is immediately written to disk as the download progresses.

Also recent versions of Web Connection let you download larger than 16mb into memory - with limitations.

Don't use HttpGetEx() either - that's just a lot more work than it needs to be and there's no benefit to it unless you need to do something special with the data as it comes in.

You can do everything you need with HttpGet() or Get().

Here's more info on how to stream to file:

Streaming a File to Disk

The entire topic is worth a read, if you're still using really old (pre 4.x) code with HttpGetEx().

+++ Rick ---

Hi Rick,

Problem with this code is that I don't know ahead of time if the file is large or not.

And the other problem is that the data stream is an xml document with some meta data about the file embedded in the "header" portion of the file. I typically parse this first and then if everything okay, I pull out the data between the tags for the actual file.

So I would end up traversing the string at least a couple times:

• once for the download to file (which could be faster so that might help)

• 2nd time use FILETOSTR() to put into a var

• then typically I just look for the "header" stuff in the first 1000 chars so pull those chars into another string and then use STREXTRACT() to pull out the individual meta data

• but it looks like that is my only real option; I can put a TRY/CATCH around my basic call but cannot really put it around yours; like I said, the super large files only seem to occur a few times at best a month so would rather trap it then rewrite the code and test it...but maybe on a slow day I can do that 😃

Albert

If you can extract the "header" from the 1000 first characters, you should use a low-level file function instead of filetostr().

Great idea Tore. That would save a double pass through the entire string. Thanks.

If you think the file could be large, read the thing into file. It's fast and you can check the file size when it it's done (or look at the content-length header) afterwards. It's easy to get a string if you really need it later using FileToStr(). There's so little overhead in reading a file there is no downsize to this IMHO over any alternatives where you selectively decide how to read the file.

As I mentioned - wwhttp supports larger than 16mb files via lUseLargePostBuffer. But if you do this you have the problem of a larger than 16mb string in FoxPro that you can't do much with. Assume you get a >16mb string - what are you going to do with it? Or rather what can you do with it given the string limitations. About the only thing you can do with it is do direct assignments and it may not even work to assign it to a COM object (like an XML parser). It basically becomes an immutable variable that you can't modify in any way and will fail in a myriad of ways due to the FoxPro string limitations.

If this is an XML doc why not run to disk, then load the XML Document from disk instead of memory?

+++ Rick ---