Progress bar out of synch

[Marjolein Katsma]

Cosmetic, but somewhat confusing:

Yesterday I needed to copy a largish set of files to an SFTP server (Folder compare). It went without a hitch.

But the progress bar was confusing on two counts:

• First, it seemed to lag behind, until I realized it was showing "byte percentage" rather than "files percentage" (or at least I assumed that's what I is - is it?);
• But when it was nearly done, the progress bar was completely "full" and the number of files also seemed to be complete - but it still went on to copy several files for a few more minutes - see attached screenshot.

That's actually two buglets:

• the number of files copied indication is not correct (it still needed to do a whole bunch); maybe it shows the number it has started to copy? It should show the number it has finished copying.
• The progress bar should not be full until the copy process is (very nearly) complete: it looks as if the program is rounding up instead of down

[Michael Bulgrien]

Progress reporting... X of X bug

I thought I would add details of Progress Bar issues I reported earlier this year:

Michael Bulgrien wrote:

Something weird is happening with the progress bar when copying multiple files.

First file started with a "Copying 2 of 3" message with 1st file name.

The message eventually changed to "Copying 3 of 3" with 2nd file name.

There never was a "Copying 1 of 3" message, and the 3rd file name never showed up in the progress status messages. However, as soon as the 3 of 3 message closed, the log reported "Successfully copied 3 items" and the 3rd file was present on the target folder.

Scooter wrote (Friday, June 29, 2007 4:41 PM):

There isn't a bug here, but we are going to tweak the behavior to try to make this a little less weird.

When you start a copy of more than one file on local or network drives Cirrus copies two files in parallel. It's cumulative, so if you select 5 files and start a copy and then 5 more and start a second copy (with two progress bars), it will try to copy 4 files at the same time.

So here's what's happening in your case:

- You selected three files to copy, and #2 was significantly larger than #1 and #3.
- Cirrus starts copying #1 and #2 and shows the filename of the first file it's still working on
("Copying 2 of 3: #1").
- #1 finishes and #3 starts, but #2 is still in progress ("Copying 3 of 3: #2").
- Since #3 is smaller than #2, it finishes before #2's copy does, so its filename is never shown.
- When you canceled the copy it was still copying both #2 and #3, which is why you got the
"1 succeeded, 2 failed" message.

To fix the weirdness we're going keep local/network copies from performing more than one copy at a time, and for copies that are done in parallel we'll show the most recently started filename instead of the oldest one that's still running.

Simultaneous copies can be faster in certain circumstances, so we will support them again with an option at some point. The next release will allow multiple connections for FTP transfers, but it will be a hard limit over all of the running copies instead of the soft limit like disk transfers use, and a single operation will be able to use all the connections instead of just two.

Craig

Michael Bulgrien wrote:

Thanks for the explanation.
May I suggest the following:

Even when copying two files simultaneously, check the size of the two files. List the smaller file name first. Only increment the x of x count when a file copy completes. This way Cirrus doesn't mislead a user into thinking file 1 is finished because it is reportedly working on file 2. When a file completes copying (regardless of how many it is working on in the background) increment the x of x count and update the name of the file being copied to the one that has the fewest remaining bytes to copy. This also gives the best chance of displaying all filenames during the copy.

Scooter wrote (Friday, June 29, 2007 5:37 PM):

That sounds pretty reasonable. Tim wants to get some of the other
progress issues ironed out first, but we'll keep these suggestions in mind.

Craig

I haven't followed up on this, so I can't be sure if the Scooter team has done anything more with it yet...

[Aaron]

Transfer Behavior

Hello-

The reason the progress bar appears full is probably because the Refresh that occurs at the end is not included in the progress bar. So the bar must fill, then it Refreshes the changed directories (which can take some time).

For local transfers this should be addressed, since we do one at a time now. The other suggestions we have not had time to address yet.

[Marjolein Katsma]

Hello-

The reason the progress bar appears full is probably because the Refresh that occurs at the end is not included in the progress bar. So the bar must fill, then it Refreshes the changed directories (which can take some time).

That would apply when the transfer is actually done - no problem there. But I'm seeing a full progress bar when it's still copying, and I'm seeing the file names still being copied mentioned one by one in the status bar (behind the already-full progress bar).

For local transfers this should be addressed, since we do one at a time now. The other suggestions we have not had time to address yet.

Obviously there are more issues and weirdnesses here... but I reported this for an local-to-SFTP session.

Point is, I was watching that progress bar closely to see when it's done so I can do a refresh in the browser - and that strategy doesn't work, I actually have to wait till it's gone (refresh/recompare both done) to be sure the transfer is done. That tends to slow me down more than needed (I don't need the recompare result to be able to see a new result in the browser).

Still cosmetic, I know, but it would be great if such weirdnesses could be addressed (as well as the ones Michael reported).

[Zoë]

Marjolein & Michael:

I've modified the progress reporting for the next release so it will only count the number of completed jobs (+1 if there are any active) instead of all of the completed+inprogress jobs. I also modifed the prompt to show "Finalizing" once all the copies are done. It seems to make the progress reporting work like you'd expect.

Michael:

I tried implementing your suggestion of showing the name of the item closest to being finished, but it was disconcerting since filenames could appear and then disappear before they were actually done. Let me know if you still think some more finessing is required after you get the next build.

Marjolein:

The progress bar (unlike the "n of m") is based on the bytes to transfer, plus a small overhead per job to handle command overhead/setting/attributes/etc. AFAIK it should only show 100% once all the jobs are done, though it could get awfully close depending on the sizes of the files, the order they're transferred, and how many directory refreshes are required. If you're seeing incorrect behavior here consistently, can you try to pinpoint what types of transfers are triggering it? Thanks.

[Marjolein Katsma]

Hi Craig,

I've modified the progress reporting for the next release so it will only count the number of completed jobs (+1 if there are any active) instead of all of the completed+inprogress jobs. I also modifed the prompt to show "Finalizing" once all the copies are done. It seems to make the progress reporting work like you'd expect.

I'll be looking forward to that.

The progress bar (unlike the "n of m") is based on the bytes to transfer, plus a small overhead per job to handle command overhead/setting/attributes/etc. AFAIK it should only show 100% once all the jobs are done, though it could get awfully close depending on the sizes of the files, the order they're transferred, and how many directory refreshes are required. If you're seeing incorrect behavior here consistently, can you try to pinpoint what types of transfers are triggering it? Thanks.

I'm doing largish transfers only occasionally at the moment but I'll certainly keep a close watch at what's happening!

[Tom]

I've modified the progress reporting for the next release so it will only count the number of completed jobs (+1 if there are any active) instead of all of the completed+inprogress jobs.

Maybe this'll resolve some oddities I've observed as well. Seems like the bar's not directly sync'd to the actual local filesystem activity. Also, I miss the ability to track progress of individual files instead of just the overall "job" since some of 'em are pretty large. The bar just "leaps" forward whatever percentage when a file's finished, rather than advancing steadily as the file's copied.

Are you guys considering adding a second bar somewhere later to mimic the v2 modal dialog with separate file- and job-progress bars?

[Michael Bulgrien]

I tried implementing your suggestion of showing the name of the item closest to being finished, but it was disconcerting since filenames could appear and then disappear before they were actually done.

I don't understand how implementing my idea could result in filenames disappearing befire they were actually done. My suggestion was to display the name of the file closest to being done, then leave that name in place until it finishes copying. When it has finished, then display the next file closest to being done. That does not meant that one or more files won't finish first, it just means that if the closest to being done files are the ones displayed, the user is more likely to see more of the filenames during the copy.

[Zoë]

Sorry, I missed the part about not changing it until it was done in your initial suggestion. It's a bit disconcerting either way though. Implemented as you suggest, if you had files A, B, and C sorted by name and C was the smallest it would show C instead of A if all three are copying simultaneously.

I did test it the way you suggested (with files large enough that rapidly switching wasn't an issue) and it just felt a little off since it seemed to be transferring files in the "wrong" order.

[Zoë]

The changes I made affect the "n of m" count and which filename is displayed; they didn't affect the progress bar at all. The count is based on the number of countable top-level jobs (eg, individual file copies), whereas the progress bar is based directly on the bytes transferred.

Are you doing local-to-local, local-to-ftp, ftp-to-local, or ftp-to-ftp transfers? Local-to-local and local-to-ftp should track individual files pretty closely. FTP-to-FTP can track slightly off if once server is faster than the other. Ftp-to-local (downloads) can appear off because it downloads it to a cache first and then copies it to the final location.

[Michael Bulgrien]

I did test it the way you suggested (with files large enough that rapidly switching wasn't an issue) and it just felt a little off since it seemed to be transferring files in the "wrong" order.

Would it make sense to display a message similar to this:

Copying 1 & 2 of 6 ...

If file 2 completes first, then the message updates to:

Copying 1 & 3 of 6 ...

This would give the user a better idea of what is happening behind the scenes without the confusion of out-of-order filenames.

[Michael Bulgrien]

Are you doing local-to-local, local-to-ftp, ftp-to-local, or ftp-to-ftp transfers?

FYI - Most of my work is local-to-remote and remote-to-remote (not FTP, but copies between remote machines over a LAN connection).

[Zoë]

Cirrus doesn't currently differentiate between local/remote so hard drives, mapped drives, and unc paths are all considered the same. I just use "local" in that context as shorthand.

Are your remote servers far enough that multiple connections will help?