Dropbox - poor upload speed and upload problems with large files

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • DavidW
    Visitor
    • Oct 2014
    • 3

    Dropbox - poor upload speed and upload problems with large files

    I'm fortunate enough to have an 80/20 Mbit/s Internet connection with minimal visible contention and a router that barely breaks a sweat at full line speed. BC4's uploads to Dropbox does not make good use of the available upload bandwidth because it appears to be using chunked upload with fixed 4 MByte chunks. The small chunk size means there are excessive protocol overheads, not least as a complete TLS negotiation is required per chunk. If I use another implementation of the Dropbox Core API that uses fixed 32 MByte chunks, I*can upload large files to Dropbox at 40-45 seconds per full chunk, which works out at approximately 50*MBytes per minute.

    The quick fix solution would be for BC4 to allow the use to select alternative fixed chunk sizes. In the longer term, maybe the developers could experiment with an adaptive chunk size algorithm that seeks the sweet spot for chunk size based on the time taken to upload each chunk and the time taken with per chunk overheads.

    If the upload code is revisited, it might be worth adding a feature to throttle uploads to prevent saturation of the available upload bandwidth. This isn't a huge issue for me, as I typically have plenty of upload bandwidth remaining and I make use of my router's sophisticated traffic prioritisation capabilities, but it may well be an issue for others.


    Despite the apparent use of Dropbox's chunked upload API, I'm unable to upload files larger than around 150 MBytes to Dropbox using BC4. I get a mixture of "Connection lost (error code 100354)" and "Connection timeout (error code 96270)" errors.

    I don't know how BC4 generates its chunks for upload, but it might be significant that the Windows 7 x64 machine I'm using for testing has an almost full C: drive, typically with a few hundred MBytes spare. I realise this is horrid from a performance and reliability standpoint. Unfortunately, the machine is cursed with a 128 GB SSD, which is too small for my needs but is not cost-effective to replace considering the limited time remaining before the entire machine is replaced.
  • Aaron
    Team Scooter
    • Oct 2007
    • 16000

    #2
    Thanks for the report. We'll investigate these. With BC 4.0's launch, we've implemented and tested against Dropbox with the provided API, but I'm seeing similar errors if I try to copy 4gb files.
    Aaron P Scooter Software

    Comment

    • DavidW
      Visitor
      • Oct 2014
      • 3

      #3
      Thank you for your reply, Aaron, and for the ongoing work you and the Scooter team put into Beyond Compare. I've had a lot of value out of my BC3 Pro licence and have been seeing additional value from BC4 Pro since I upgraded around half way through the public beta.

      As you have been able to reproduce the large file upload problem, you should have no problem producing and testing a fix in house. If you want confirmation that fix solves that problem in my environment when it is ready, or if there is anything I can do to help test a proposed fix for the upload performance issue, I'm happy to help.


      When the developers are revisiting the Dropbox upload code, maybe they can address a related user interface infelicity. In BC 4.0.0 the progress bar shows the upload of each chunk as instantly complete, giving the impression that the upload surges and stalls repeatedly. This false impression will only worsen if larger chunk sizes are supported. It would seem better for the progress bar to reflect the ongoing progress in uploading the current chunk.

      Comment

      • Chris
        Team Scooter
        • Oct 2007
        • 5538

        #4
        BC 4.0.1 (in testing now) will have better progress reporting for Dropbox and other cloud transfers.
        Chris K Scooter Software

        Comment

        • DavidW
          Visitor
          • Oct 2014
          • 3

          #5
          Unfortunately all the problems I noted in the first post of the thread still exist on 4.0.3. BC4 is still slow at uploading (peak speed of around 2.5 Mbit/s on a 20 Mbit/s upload connection). BC4 still fails to upload files larger than about 150 MBytes, though the symptoms have changed slightly with 4.0.3 where you get a "Connection lost (error code 10058)" or "Connection timeout (error code 96270)" error after ~4 MBytes.

          Testing was performed using the same Windows 7 x64 machine mentioned in the first post.


          Please can the Scooter team give this some attention. Thanks.

          Comment

          • Aaron
            Team Scooter
            • Oct 2007
            • 16000

            #6
            Sorry, we're aware of some of the issues with Dropbox but haven't been able to tackle them yet amongst all of the other bug fixing we've had scheduled out. It's still on our radar and I'll bump the issue a bit.
            Aaron P Scooter Software

            Comment

            Working...