why refresh so slow(or why refresh?)

[Michael Bulgrien]

Also, are you doing an interactive comparison or using scripts?

I believe gbenson already answered this question:

...I started up my scripts to remote sites I ran into a major problem with it...

[David Tilton]

I am having the same problem as Shen. One difference is that my target of 'Copy Right' is often completely empty. There are zero files for possible conflicts yet BC3 can sometimes refresh for hours. A small set of files refreshes for a shorter amount of time which is understandable but the refresh time seams to grow exponentially. The refresh time on 300 files vs 600 is not double but more than quadruple.

I have found a possible workaround. If I do a full refresh 'CTRL F5', which takes only a few seconds, then immediately issue the 'Copy Right' the files start to copy in only a few seconds. I have tried this workaround with as many as 4 thousand files and so far it has eliminated the long refresh times.

[gbenson]

I have tried both interactive comparisons and scripts. Normally I run about 10 sessions at the same time, but haven't done this during my testing and have been doing 1 at a time.

The CTRL-F5 trick did not work for me, still refreshes no matter what I do. I have tried a number of different settings, comparison rules, and even views, using copy vs sync vs mirror and as soon as I have more than a few thousand files to copy that Refreshing thing goes into la-la land for hours and hours.

On a side note, I let the Sync continue to run that I had posted about the other day. The last time I looked at it, it had been refreshing for a little over 20 hours. The following morning it had completed the Sync with a total time of 30 hours. So refreshing was > 2/3 of the total time.

[rlindell]

I'd like to see more information this as well.

I have the trial version of BC3, and have been using it to help me with moving some large directories from a remote server to a local server. I REALLY like the software, and am considering purchasing the full license, but this refreshing problem needs to be resolved.

In my case, I'm doing a left to right copy of files from a remote server (left) to a local server (right). The servers are connected through a 3mbps MPLS. Much like gbenson was experiencing, I am seeing LOONG periods where it is refreshing - in my case sometimes longer than 24 hours depending on the size of the directory and how many files are in it.

What I have been doing as a workaround is drilling down deep into the directory structure and initiating individual copies for a much smaller subset of files and directories. This seems to work much faster than if I were to attempt a copy of the root parent folder. While this has been getting me by, it is much more labor intensive than I would like. I need to babysit these copies and start new ones when the previous ones finish.

[norrisg]

I must say that this "refreshing" issue is quite tedious, not least because it is inconsistent. Yesterday I was using BC3 to make two very similar directories different.

To be more specific, I was doing this:

1. Making sure that there was nothing I wanted on the right hand side (which was another XP PC) that I wanted on the left before I deleted it from the right
2. Making sure that what I did want to keep on the right was the same as on the left
3. Removing a lot of stuff on the right I didn't need

When totally removing a large directory on the right, BC3 would sometimes, but not always, sit there "refreshing" for minutes before deleting the directory.

I can't see any reason why it would need to refresh anything on either side before deleting an entire directory on one side. Still less do I understand why it sometimes needs to, and sometimes doesn't. There's something really wrong here, and it adversely affects usability of BC3.

I'm running the most recent build, but this refresh behaviour has persisted over many builds, beta and otherwise, and it is very annoying. I really hope Scooter can get to the bottom of it soon.

[DorothyFan1]

Why aren't members of the ScooterTeam replying in this thread? They must *clearly* be aware of this issue. The fact they haven't posted workarounds is very troubling. However, I believe David Tilton's workaround should be mandatory reading for everyone following this issue.

[David Tilton]

Some Additional Details

First a correction, I have done quite a bit of testing today and it does not look like the refresh time grows exponentially after all.

My connection allowed BC3 to refresh the information on about 6 files per second. Using this rate I was able to predict when the refresh would finish to within a second or two as far out as 2hrs. Being able to predict this at least helps me know what to expect. Also, I am still occasionally able to get BC3 to skip this action if I start the copy just after a full refresh. I have not looked into why the full refresh itself seems to be so much faster.

Watching the network traffic using Wireshark I was able to verify that BC3 is not getting file contents during this time but is getting network open information and basic file information for each file. So this does not look like an issue with the 'Quick Test'/'Compare Contents' settings. This also means that the size of the files are not relevant; 300 1k files and 300 100k files take the same refresh time.

I think there are a couple of issues with getting all of the open and detail information before any files are copied.
1. If others are waiting on the files it is often better for me to get something to them immediately while the rest are on the way. For a large set of files an entire work-day can go by with not a single file for others to work on getting copied.

2. Once the copy begins the information can be far to old to be trusted. It seams to me that an extra 1/6 second of copy time just before the file is copied is far superior than working under the assumption that information this old is still valid. For a small set of files, say 300, much is not likely to change in the minute it takes to refresh but for a large set like 43k the information on the first file will be over 2hrs old when BC3 finally starts to copy it.

I like the idea of BC3 getting the information up front so that I can walk away and know the job will not stall 5-minutes-in waiting for me to choose 'overwrite yes/no', etc. However, for a large set of files because of the cost of BC3 getting organized up front and considering #2 above I think I would like an option to control whether the information is gathered up front BC3 style or during the copy like BC2.

[Zoë]

David,

Thank you for the extra information. We'll follow up with this some more, but as an immediate question, have you checked whether BC2 is any faster than BC3? It sounds like they should be about the same speed, and it's just the fact that "refreshing" occurs up front that makes it more noticable. Is that the case?

[gbenson]

I just ran a full scale test over the weekend between 2 sites that have similar bandwidth and identical servers at the other end, both sites use the same Riverbed network accelerator. Using BC2 both sites were within 5% of each other from a time standpoint. Over the weekend (Fired up Friday 4:00pm) I ran a test where 220,000 files had changed on my master server and I ran BC2 to site1, and BC3 to site2. Site1 finished replicating at about 7:00 AM Sunday, and Site2 at that point was 55% of the way through copying. BC2 proved to be faster by about 12 hours. I will check with IT to see if anything strange went on with that site over the weekend, but I am running another smaller update (both using BC2) and their speed is comparable again.

have you checked whether BC2 is any faster than BC3?

[Zoë]

The new release (3.0.8) no longer has the "Refreshing..." step when using the "Folder Sync" session or when using command line scripting. This won't have any effect on those of you using the "Folder Compare" session, even if you're using the "Synchronize" commands, and I hope to come up with a different solution for that soon. Do note that this change in behavior does not re-introduce BC2's style, where it refreshed before each individual file. If something changes externally it won't notice it, which is why the new behavior was only introduced for non-interactive commands where the possibility of the program sitting idle for a long time is a lot less.

[JohnFLand]

More ranting about "Refreshing" --

Using File Compare, I compare 5 local drives, aggregated under one directory as junction points, against a BCSS file of a similar 5 remote drives (also aggregated under one directory as junction points). Total number of files is about 1,167,000. It takes about 40 minutes to do the initial compare (this is on an admittedly slow machine -- essentially a WinXP NAS). I end up with a list of files on the left (local) system that are not on the right (remote BCSS) system. I expand the list and Select All Files, hit Shift Left Arrow to select only the files on the left, and click on Copy To (to copy to a transfer drive).

Now, the screen already shows the exact name of each file to be copied, and I presume that under the hood BC3 has the full path of each file (or can easily generate it), so copying should be able to commence immediately -- just copy each highlighted file of the expanded list on the left. But no -- Refreshing starts that takes LONGER than the initial comparison.

Something is broken.

[Aaron]

We are looking into the different accounts of refreshing issues, but it is important to note that not all users are running into the same issues. Please read over this thread as some users have submitted ideas that have helped them, and others have helped explain what you may be running into.

The BC3 combined refreshing stage + copy is most often faster than BC2. It is not just checking for file location, but any file flags such as Read-Only. BC2 performed this check on each file during the transfer, so the entire transfer not only took longer, but if it hit a conflict on any particular file it would cause a dialog prompt to come up at that time. This would stop the transfer until a user interacted with the program. BC3 front loads the checking, so any user interaction needed is presented as soon as it can be. By front loading it, the overall process also takes a shorter amount of time. BC3's "Check All, then Copy All" is quicker than "Check,Copy,Check,Copy,Check,Copy". If you perform this workflow with BC2, and then BC3, and then BC2 again (multiple tests help negate the benefit of cached files), which is faster for you?

You mention your left side is a local side, and that your right side is a BCSS (Beyond Compare Snapshot). Are you using the Copy To Folder command after selecting your Left Side files? Or are you changing the Right Side to be a new destination, and then copying to it with the Copy To Other Side command?

Let us know if you have any questions.

[JohnFLand]

You mention your left side is a local side, and that your right side is a BCSS (Beyond Compare Snapshot). Are you using the Copy To Folder command after selecting your Left Side files? Or are you changing the Right Side to be a new destination, and then copying to it with the Copy To Other Side command?

I use the Copy to Folder command after selecting left side files that do not exist in, or are newer than, the right side snapshot. The destination disk is then used to update the remote system by physically carrying it to the remote location and copying the new/changed files to the remote system. (The Internet link between the local and remote systems is about 500 MB/hour, and the remote site is 15 minutes away; over the course of a few days, the total changes may be 50-100 GB, so sneakernet is faster and less irritating than the Internet link).

[winlux]

When refreshing takes longer than I have time

My problem is that refreshing takes longer than the time I have. When I am at the office I have a couple of hours to let it copy. Afterwards I need to close my notebook and go working at a customer's office etc. BC2 had a lot of the job done that way and I could do the rest when I was back in the office (of course needing another comparison but that really seems to be much faster than that "Refreshing..."). BC3 always starts over refreshing and therefore it's always refreshing but never actually copying something.

Of course I can copy the few files I really changed by hand. But whenever I got a new notebook (which is the case right now) I have a real problem. I started copying the data from the old machine a week ago and there are still over 100'000 files left (I have a lot of source codes from PHP-projects like vtiger, egroupware, gallery etc. made of millions of tiny files, in several copies customised for different customers).

I can't copy the whole stuff with Windows Explorer the first time because there is for sure one file somewhere in those 40GB of stuff which fails and makes Explorer stop.

The really ugly thing is that "Refreshing..." is completely blocking every other action. I can't copy a single file while that task is running. It seems like I can't even cancel any operation. The program is perfectly responsive but I can only use it as a read-only-filebrowser during that time :-(

Regards
Peter

[Lutz]

Did you try to use a new tab/window/instance (/solo-switch starts a seperate BC)? I sometimes used this to do some work on the same directories when a task takes long time.