FTP CRC fail + slow performance

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • rasterian
    Visitor
    • Jan 2017
    • 4

    FTP CRC fail + slow performance

    Hello everyone,

    I am testing Beyond Compare to synchronize two Windows 10 machines over the Internet. Running Bitvise SSH server 7.24 on one side and Beyond Compare 4.1.9 on the other, both are the latest versions available. Both applications are running with default settings, just configured user and password on the SSH server and SFTP access details in Beyond Compare (using SFTP (SSH2) protocol, increased number of simultaneous connections to 10).

    Problem #1: CRC comparisons fail on the SSH server

    I connect to the server and browse the remote filesystem, but when I enable "Compare contents" and "CRC comparison" (by clicking on "Rules" in the toolbar), errors are generated in the log window and no comparisons are performed. Binary comparisons and file transfers are working fine. I have examined the logs on both sides and found this (just pasting one entry here to illustrate the problem):

    Beyond Compare log:
    2017-01-21 20:56:41 Unable to determine CRC32 of sftp://[email protected]:XXXX/W-Wares/Backup/Cloud backups/2016-02 Dropbox backup.rar: Decoding packet content failed. See server logs for more info.
    Bitvise log:
    <event seq="2976" time="2017-01-21 21:56:39.318786 +0100" app="BvSshServer 7.24" name="I_SFTP_SERVER_EXTENDED_PACKET_DECODE_ERROR" desc="Error decoding Extended SFTP packet.">
    <session id="1010" service="SSH" remoteAddress="XX.X.XXX.X:XXXXX" virtualAccount="XXXXXXX" windowsAccount="XXXXXXXXXX\BvSsh_VirtualUsers"/>
    <channel type="session" id="1"/>
    <parameters extendedPacketType="check-file-name"/>
    <error type="Exception"
    message="Decoding procedure failed, call stack:
    - DecodeString(): premature end of input data
    - DecodeWStrAsUtf8(): error decoding UTF-8 string
    - SftpPacket.Extended.CheckFileName: failed at decoding file name"/>
    </event>
    Problem #2: Binary transfers are slow

    Binary transfers from the server seem to be running slow in Beyond Compare. I transfered the same file both in Beyond Compare and in Filezilla:

    Filesize: 96.7 Mb
    Beyond Compare: 48 seconds
    Filezilla: 21 seconds

    Filezilla generally exhausts the upload bandwidth available on the server side, but Beyond Compare seems unable to do so.


    Any feedback on what may be triggering the errors/performance issues above would be much appreciated. Beyond Compare does seem to be an awesome tool, and I am hoping that it will be just what I need to keep these two machines in sync.
  • Aaron
    Team Scooter
    • Oct 2007
    • 15941

    #2
    Hello,

    #1. Thanks for those logs. Our developer is investigating and has emailed Bitvise for a bit more information.

    #2. For specific SFTP servers, we're able to detect and use the full throttle. If we are unable to detect the server (which, is admittedly older code), then we use a method of transfer that is slower but offers greater compatibility. This helps us avoid a lot of problems we found with various servers that would corrupt or hang without this transfer type.
    Aaron P Scooter Software

    Comment

    • Zoë
      Team Scooter
      • Oct 2007
      • 2663

      #3
      Bitvise is still looking into the Decoding "Decoding packet content failed" error, but they have told me that their server doesn't support calculating CRC32 hashes. The information provided by the server at login just indicates whether it supports server-side hashes or not, not the specific hashes (CRC32, MD5, SHA256), and BC currently assumes that if it supports it at all it supports CRC32. We don't currently handle any others, and while MD5/SHA* hashes are on the wishlist as a long term goal, none of them are supported widely in a wide enough range of backends to justify the additional development effort yet.

      Short-answer, even once the bug is fixed, you're not going to gain anything by doing CRC calculations against a BitVise SSH server, because BC will still have to download the entire file in order to do so.
      Zoë P Scooter Software

      Comment

      • rasterian
        Visitor
        • Jan 2017
        • 4

        #4
        Hi Aaron, Zoe,

        Thank you very much for your responses and for clarifying the source of the issues. I had installed BitVise specifically for Beyond Compare, but it seems this is not a good match.

        May I ask what SFTP servers would you recommend for Windows that enable full throttle and implement CRC32? I am sure this has been asked before, but I would like to get an up-to-date response to get it right on the second attempt.

        Comment

        • Aaron
          Team Scooter
          • Oct 2007
          • 15941

          #5
          Hello,

          Not offhand, no, we aren't aware of a specific SFTP server that supports CRC32. Zoe mentions that some FTPS servers may, but that these would re-negotiating the SSL connection and would limit the speed.
          Aaron P Scooter Software

          Comment

          • rasterian
            Visitor
            • Jan 2017
            • 4

            #6
            Aaron,

            In this case, dropping CRC32, any SFTP servers recommended for Windows that enable full throttle?
            Alternatively, switching to FTPS, any FTPS servers recommended for Windows that enable full throttle and support CRC32?

            Apologies for inquiring again, but I would really like to know if there is a better alternative to BitVise for the usage scenario I am considering. I was planning to run CRC32 comparisons to catch bitrot, but I guess this could also be done sequentially on both machines using a snapshot. Thanks.

            Comment

            • Aaron
              Team Scooter
              • Oct 2007
              • 15941

              #7
              Hello,

              FTPS + CRC32 similarly wouldn't be able to go full throttle due to how the re-negotiation with SSL.

              We don't have a specific SFTP server we'd recommend, but any popular solutions should work if you aren't trying to use the server based CRC generation or enabling the CRC content scan (which would download the files).
              Aaron P Scooter Software

              Comment

              • rasterian
                Visitor
                • Jan 2017
                • 4

                #8
                Thank you Aaron. If I may offer a suggestion, this could be clarified on the following support page:

                http://www.scootersoftware.com/support.php?zz=kb_crc
                "The behavior of CRC comparisons depends on the FTP server you're connecting to. If the FTP server supports it, the server will generate the CRCs and only transfer the CRC values. If the server doesn't support it, Beyond Compare will have to transfer the entire file and calculate the CRC locally."

                After reading this, I was under the impression that CRC would be supported on modern SFTP server software (which is why I ended up here reporting the issue in the first place). But based on your feedback it seems the feature is not operational (at least over SFTP in Windows) as things currently stand. Not that there is anything incorrect stated there; but it does hint of functionality that does not seem to be widely available.

                Cheers.

                Comment

                • Aaron
                  Team Scooter
                  • Oct 2007
                  • 15941

                  #9
                  The issue is that the feature is supported in the SFTP spec, but the spec is implemented to varying degrees by each server software. And FTP and SFTP are different protocols. We aren't aware of any SFTP that support it fully, so we don't have a test case to perform against. FTP Servers have a specific xCRC flag which, when true, is something we can read and then use for server side hashes. BitVise's server has a flag that it supports server side hashes (which, we read and assumed would include multiple protocols like CRC32, when we see this flag is true), but they don't actually support CRC specifically but only other hashes instead. BC4's current support is limited to only CRC32.
                  Aaron P Scooter Software

                  Comment

                  Working...