I have recently upgraded the West Wind Web Connection version to the latest version in our Visual FoxPro application so that I could incorporate SFTP into my application for file upload / download.

We have encountered a bug when uploading our clients’ data files using the wwSFTP FtpSendFileEx method.

Using the default nFtpWorkBufferSize of 0x6400, it seems that every 10 or 11 MB, an upload block will be corrupted, causing our larger ZIP file uploads to be broken.

When I increased the nFtpWorkBufferSize to 0x7600 which is approaching the SFTP protocol limit, the safe block size increases to about 15 MB, but still has the problem.

Testing with the FtpSendFile method for the same large files shows that FtpSendFile does not have this bug in it.

The difficulty with using FtpSendFile is that I cannot show a progress bar, and my clients will think my program has locked up and kill the application in the middle of the upload.

Do you have a work-around or bug fix upgrade for this issue?

I have placed my test program code at the end of this post that demonstrates this issue, after which I used the DOS file compare command FC to see that the messed up blocks are about 15 MB apart in my test file.

Here are the interesting parts of the file comparison for a 32,000 KB file:

Comparing files \\SFTP.SERVER.NETWORK.LOCATION\SENDFILETEST.ZZZ and C:\DEV\TRANSFER\SENDFILETEST.ZZZ
00E76225: 0A 25
00E76226: 0B 26
00E76227: 0C 27
00E76228: 0D 28
00E76229: 0E 29
… sequential output until …
00E7D7E4: C9 E4
00E7D7E5: CA E5
00E7D7E6: CB E6
00E7D7E7: CC E7
01CEC44A: 14 4A
01CEC44B: 15 4B
01CEC44C: 16 4C
01CEC44D: 17 4D
… more sequential output …
01CF3A08: D2 08
01CF3A09: D3 09
01CF3A0A: D4 0A
01CF3A0B: D5 0B
01CF3A0C: D6 0C

* Test SFTP large file upload with binary file of all ASCII characters from NUL to CHR(255)
IF NOT FILE("SENDFILETEST.ZZZ")
  lcBlob = ""
  FOR i = 0 TO 255 
    lcBlob = lcBlob + CHR(i)
  ENDFOR 
  lcBigBlob = REPLICATE(lcBlob,256)
  FOR i = 1 TO 500
    STRTOFILE(lcBigBlob,"SENDFILETEST.ZZZ",1)
  ENDFOR
ENDIF 

SET PROCEDURE TO wwFtp Additive
SET PROCEDURE TO wwAPI ADDITIVE
SET PROCEDURE TO wwUtils ADDITIVE 
SET PROCEDURE TO wwSFtp Additive
SET PROCEDURE TO wwdotnetbridge ADDITIVE
loSFTP = NEWOBJECT("WWSFTP","WWSFTP.PRG",NULL)
loSFTP.nFtpPort = 22 && our sftp server uses port 22

* Setting the nFtpWorkBufferSize to 0x7600 instead of the default of 0x6400 increased the minimum safe size from about 10MB to 15MB.
*  Approximately every 15 MB the block uploaded does not match the block it should be uploading, and then it is correct for another 15MB.
loSFTP.nftpworkbuffersize = 0x7600 

loSFTP.cFtpServer = "sftp.server.com"
loSFTP.cUsername = "username"
loSFTP.cPassword = "password"
loProgress = CREATEOBJECT("progresscheck")
BINDEVENT(loSFTP,"OnFtpBufferUpdate",loProgress,"OnFtpBufferUpdate",0)
? "Simple Send:"
? loSFTP.FtpSendFile(loSFTP.cFtpServer,"SENDFILETEST.ZZZ","SENDFILETEST.HIGH.ZZZ",loSFTP.cUsername,loSFTP.cPassword)
? "Low Level Send:"
? loSFTP.FTPConnect()
cCompatible = SET("compatible")
SET COMPATIBLE ON
loSFTP.nCurrentFileSize = VAL(ALLTRIM(STR(FSIZE("SENDFILETEST.ZZZ"))))
SET COMPATIBLE &cCompatible
? loSFTP.nCurrentFileSize 
? loSFTP.ftpsendfileex("SENDFILETEST.ZZZ","SENDFILETEST.LOW.ZZZ")
? loSFTP.nERROR
? loSFTP.CERRORMSG
? loSFTP.FTPClose()

DEFINE CLASS progresscheck as Custom 

  PROCEDURE INIT
    ERASE ("UPLOAD_LOG.TXT")
  ENDPROC

  PROCEDURE OnFtpBufferUpdate
    LPARAMETERS lnbytesdownloaded,lnbufferreads,lccurrentchunk, lnTotalBytes, loFTP
    LOCAL lnPercent AS Number

    IF lnBufferReads > 0   
      IF lnbytesdownloaded > 0 AND lnTotalBytes > 0
            lnPercent = (lnbytesdownloaded/lnTotalBytes)*100
      ENDIF
    ENDIF
    STRTOFILE(TEXTMERGE("[DATETIME()]: lnPercent=[lnPercent]%  lnbytesdownloaded=[lnbytesdownloaded], lnbufferreads=[lnbufferreads],LEN(lccurrentchunk)=[LEN(lccurrentchunk)], lnTotalBytes=[lnTotalBytes] ",;
      .F.,"[","]")+CHR(13)+CHR(10),"UPLOAD_LOG.TXT",1)
  ENDPROC 

ENDDEFINE 

--- Lou Harris.

Lou,

I can't duplicate that behavior running a straight forward test. Maybe you can try this out on your setup with this local server from http://labs.rebex.net/tiny-sftp-server...

************************************************************************
*  UploadBigFile
****************************************
***  Function:
***    Assume:
***      Pass:
***    Return:
************************************************************************
FUNCTION UploadBigFile()

loFtp = CREATEOBJECT("wwSftp")

loFtp.cFtpServer ="127.0.0.1"
loFtp.nFtpPort = 23
loFtp.cUsername = "tester"
loFtp.cPassword = "password"

lcSourceFile = "C:\installs\Distribution CD\Demos\wconnect.exe"  && 36meg file
lcTargetFile = "/SubFolder/wconnect.exe"

BINDEVENT(loFtp,"OnFtpBufferUpdate",this,"BufferUpdate")

loFtp.FtpConnect()

lnResult = loFtp.FtpSendFileEx(lcSourceFile,lcTargetFile)

loFtp.FtpClose()

this.AssertTrue(lnResult == 0,loFtp.cErrorMsg)

ENDFUNC
*   UploadFile

FUNCTION BufferUpdate(lnbytesdownloaded,lnbufferreads,lccurrentchunk, lnTotalBytes, loFtp)

WAIT WINDOW NOWAIT TRANSFORM(lnBytesDownloaded) + " of " + TRANSFORM(lnTotalBytes)

ENDFUNC

After running this upload, the file comes over intact and opens.

FTP in general can be tricky and in some situations servers can be finicky. Any chance you can try to upload the same data to a different server? First try your exact code against the local server and see what you get though.

+++ Rick ---

Rick,

Thank you for your help so far.

I have more information based on the results of uploading to the local sftp server app to which you directed me.

The upload failed in a different part of the file for the local server, at 32532962 bytes into the file.

The corruption seems to be exactly the same variance every time it uploads the same file. (starting at 15163941 bytes to my sftp server, and at 32532962 bytes on the local server)

Is there a specific version of msvcr71.dll that I should be using? Or a specific version of any other supporting files?

We are using:

MSVCR71.dll ver. 7.10.3052.4
gdiplus.dll ver. 5.1.3102.1360
vfp9r.dll ver. 9.0.0.2412 (but my tests in VFP SP2 had exactly the same results)
renci.sshnet.dll ver. 2016.0.0.0
wwdotnetbridge.dll ver. 6.10.0.0
wwipstuff.dll ver. 6.10.0.0

--- Lou Harris.

P.S. This is what I have in my wconnect_override.h file:

**** WCONNECT_OVERRIDE.H
**** CUSTOMIZE AND OVERRIDE SETTINGS INDEPENDENTLY
**** OF THE WC INSTALLATION

#IFDEF DEBUGMODE
	#UNDEF DEBUGMODE
	#DEFINE DEBUGMODE .T.
#ENDIF

#IFDEF INCLUDE_WWSCRIPTLIBRARY_WEBRESOURCE
	#UNDEF INCLUDE_WWSCRIPTLIBRARY_WEBRESOURCE
	#DEFINE INCLUDE_WWSCRIPTLIBRARY_WEBRESOURCE .F.
#ENDIF

Can you try a different large file? Try using the Web Connection download EXE - that's what I used in this case and see if there's a problem with that.

It's possible it has something to do with the file's specific signature on disk. If the file's not private send me a download link via email and I can try it here to see if it fails for me as well.

+++ Rick ---

The files that we are uploading are ZIP files created by the program that then immediately uploads the file to the server.

Every single ZIP file that was over 11 MB was uploading with corruption.

If it is the specific file signature, then why does wwSFTP.FtpSendFile work without this bug to the same server on the same file?

My test file can be created by the following code, it is simply the ASCII characters from CHR(0) through CHR(255) repeated over and over in the file.

* Test SFTP large file upload with binary file of all ASCII characters from NUL to CHR(255)
IF NOT FILE("SENDFILETEST.ZZZ")
  lcBlob = ""
  FOR i = 0 TO 255 
    lcBlob = lcBlob + CHR(i)
  ENDFOR 
  lcBigBlob = REPLICATE(lcBlob,256)
  FOR i = 1 TO 500
    STRTOFILE(lcBigBlob,"SENDFILETEST.ZZZ",1)
  ENDFOR
ENDIF 

--- Lou Harris.

P.S. When I modified the handler for OnFtpBufferUpdate to write the chunks to a file for comparison to the original file, the two files matched, so the problem is somewhere in the upload process but is not effecting the chunk that is passed to OnFtpBufferUpdate.

My changes to the progresscheck class in my test program:

DEFINE CLASS progresscheck as Custom 

  PROCEDURE INIT
    ERASE ("UPLOAD_LOG.TXT")
  ENDPROC

  PROCEDURE OnFtpBufferUpdate
    LPARAMETERS lnbytesdownloaded,lnbufferreads,lccurrentchunk, lnTotalBytes, loFTP
    LOCAL lnPercent AS Number, lnAdjustedSize

    IF lnBufferReads > 0   
      IF lnbytesdownloaded > 0 AND lnTotalBytes > 0
            lnPercent = (lnbytesdownloaded/lnTotalBytes)*100
            IF lnBufferreads = 1
              ERASE ("UPLOAD_CHUNKED_FILE.ZZZ")
            ENDIF 
            lnAdjustedSize = MAX(0,LEN(lccurrentchunk)-1)
            IF lnbytesdownloaded = lnTotalBytes
              lnAdjustedSize = lnAdjustedSize - (lnAdjustedSize * lnBufferReads - lnbytesdownloaded)
            ENDIF 
            STRTOFILE(LEFT(lccurrentchunk,lnAdjustedSize),"UPLOAD_CHUNKED_FILE.ZZZ",1)
      ENDIF
    ENDIF
    STRTOFILE(TEXTMERGE("[DATETIME()]: lnPercent=[lnPercent]%  lnbytesdownloaded=[lnbytesdownloaded], lnbufferreads=[lnbufferreads],LEN(lccurrentchunk)=[LEN(lccurrentchunk)], lnTotalBytes=[lnTotalBytes] ",;
      .F.,"[","]")+CHR(13)+CHR(10),"UPLOAD_LOG.TXT",1)
  ENDPROC 

ENDDEFINE 

After adding a block identifier to each sequential byte in my test file, I can confirm that the corrupted blocks are simply shifted by a number of bytes, in the case of my local server test, 23 bytes, starting on the 23rd byte of block 1078.

0x01f069f9 - 0x01f069e2 = 23
0x01f069e2 % 30207 = 23
* Block number with the shifted bytes:
CEILING(0x01f069e2 / 30207) = 1078

And then, the second time the error occurs, the same number of blocks later, block # 2156, the shift was different, a shift of 18 bytes starting at the 18th byte:

0x03E149B9 - 0x03E149A7 = 18
0x03E149A7 % 30207 = 18
* Block number with the shifted bytes:
CEILING(0x03E149A7 / 30207) = 2156

What are the differences between the 2 SFTP upload methods in the WWIPSTUFF.DLL that would cause one to work correctly and the other to fail?

--- Lou Harris.

And just to confirm that it is not just my test file, note that on the local SFTP server application, the latest WebConnection install executable has file transfer corruption in exactly the same place and way that my test file does:

--- Lou Harris.

Ok so I can verify the problem with corrupted bytes, but oddly only on files that are generated like this. Any binary files I've thrown at it seem to work ok. I tried 20 different zip and exe files and it all works. Not sure why that would make a difference because SSH defaults to binary transfer of files so it shouldn't care in any way about the content of files.

I've set up some tests in .NET to isolate the problem with the SSH library we're using, and I can't duplicate the problem in .NET - calling the same exact code that the FoxPro code is calling using wwDotnetBridge - which is very strange.

In .NET the following code works just fine (no binary differences):

[TestMethod]		
public void UploadLargeFileTestOriginalFile()
{
	var sftp = new Westwind.WebConnection.SftpFtpClient();

	using (var client = sftp.Connect("127.0.0.1", 23, "tester", "password"))
	{
		string file = @"C:\webconnection\Fox\SENDFILETEST.ZZZ";

		Console.WriteLine(file);
		Assert.IsTrue(File.Exists(file),"Invaild Path");
		Assert.IsTrue(sftp.UploadFile(file, "SendFileTest.zzz"), sftp.ErrorMessage);
	}
}

Not sure what's going on but investigating.

+++ Rick ---

Ok so after some experimentation it looks like the Buffersize is the problem.

Removing the BufferSize from wwSFtp makes the uploads work. Changing the BufferSize to anything other than 50000 appears to fail.

Try changing the following in wwSftp.prg and the Connect() method:

* loSftp.BufferSize = this.nFtpWorkBuffersize

and then try your tests again or - if you don't want to change any framework code - change the buffer size to 50,000.

loFtp.nFtpWorkBuffersize = 50000;

Just for reference, I've filed a bug for the SSH.NET library that Web Connection uses.

https://github.com/sshnet/SSH.NET/issues/227

+++ Rick ---

Rick,

Thank you for researching this, and reporting the bug.

My tests showed that it worked correctly with the local SFTP server application, by using your BufferSize fix.

However, when I tested against my testing SFTP site that is a mirror of our production site, nearly everything after byte 0xff86 (65414) in the file is incorrect. Not everything after that point is corrupted, but it is a lot more than with the smaller buffersize tests which initially prompted this thread.

I do not know if it is anything you can fix until the buffersize issue is fixed in the SSH.NET component that you are using in West Wind.

I just wanted to let you know that the BufferSize workaround that you found does not entirely fix the problem.

Thanks,

--- Lou Harris.

Lou might be a good idea if you can chime in on the issue Github. Even better if you could try to run the code in test and verify it fails in the raw .net code as well.

Rick

Rick,

Do you know why the loSFTP.FtpSendFile(loSFTP.cFtpServer,lcFileToSend,"/casepub/SENDFILETEST.2.ZZZ",loSFTP.cUsername,loSFTP.cPassword) call works on my SFTP servers, even though the loSFTP.ftpsendfileex(lcFileToSend,"/casepub/SENDFILETEST.ZZZ") method fails?

Does it call a different send method in the SSH.NET library?

Thanks,

--- Lou.

SendFile is a wrapper that sends everything directly off its buffer stream. I'm guessing the issue is the SSL stream somehow getting off sync with the manual calls. I haven't looked but underneath it all I'd assume SendFile() calls into the same low level methods that SendFileEx is using.

Good idea though - maybe looking at the code I can find a hint on what's different. For now - you might be stuck with using the Sync version 😦

+++ Rick ---

They have a Beta 1 build of the SSH.NET library that they completed in December.

Can you test with that to see if perhaps one of the stream fixes they have done may have fixed this issue?

I obviously cannot do so as your released wwipstuff.dll is linked to the released version and generates an error if I try to substitute the beta dll. (I tried it just to see, and that was the result)

Thanks for all your help. We're using the synchronous version that works right now and making a little "spinner app" to run in a separate thread that will indicate to the user that it's not locked up.

--- Lou.

Yeah the problem is that that's a pretty old build and there hasn't been any progress on that since in the Github repo. So looks stalled... They're also not addressing their issues in the repo which is also a bummer. Surprising since this is a popular and very widely used .NET library.

+++ Rick ---

Lou,

I finally had a chance to take a look at this and sure enough the 2016.1.0-beta1 package fixes the problem you reported.

I've double checked a few of the examples and updated my tests with various combinations of file and buffer sizes and it looks like it's reliably working now.

I've pushed out an experimental package with updated DLLs for Client Tools and Web Connection at:

• https://west-wind.com/files/webconnectionexperimental.zip

Let me know what you find.

+++ Rick ---

Rick,

Thank you for making the experimental build for my testing.

I can confirm that my test connection code works for the remote server with the beta version of the SSH.NET files.

We will have to wait for a new official build of SSH.NET for this to be fixed, but at least we know that my bug will be fixed in the next release of SSH.NET.

I thank you again for researching this issue,

-Lou Harris.

I think you can probably go ahead with the beta build of SSH.NET - it's been out for nearly year now so dev on that project is very slow and infrequent. I've been running it through a bunch of tests and it works well enough - the change footprint is very small in the beta.

IAC, I'll be including this updated build since the issue you brought up is fairly major and having this release at least make this work consistently.

It'll be in the next updates of Web Connection and Client Tools unless SSH.NET puts out a full release before then.

+++ Rick ---