Multiple sessions causes all files to be copied?

[gbenson]

I have a server that replicates a set of folders out to many machines. However one in particular seems problematic. About once a week it seems like one of the BC3 processes decides it suddenly needs to copy a HUGE number of files. For instance this morning when I came in there were only 2 proceses running, both were the same script. When I cancelled them this is what the log shows:

3/29/2010 6:48:25 AM >> log 3Dvia-fcipofs01.log
3/29/2010 6:48:25 AM >> option confirm:yes-to-all
3/29/2010 6:48:25 AM >> sync create-empty mirror:lt->rt
3/29/2010 10:31:24 AM Successfully synchronized 3681 items. Failed to synchronize 3825 items. Completed in 3 hours, 42 minutes.
3/29/2010 10:31:24 AM >> User Cancelled
3/29/2010 10:31:24 AM Script completed in 6 hours, 52 minutes
3/29/2010 10:31:31 AM Successfully synchronized 104539 items. Failed to synchronize 335616 items. Completed in 4 days, 5 hours.

So the script that started 4 days ago thought it needed to copy (104539 + 335616) files. However even though 4 days later it still had not copied 335616 of those files, when another BC3 script started up, it saw that only 3681 files needed to be copied. Obviously this just makes no sense.

My .bat file:
"C:\Program Files\Beyond Compare 3\BCompare.exe" @d:\admin\replicationdata\3Dvia-fcipofs01.txt

My script:
load "Seemage-Ref fcipofs01"
log 3Dvia-fcipofs01.log
option confirm:yes-to-all
sync create-empty mirror:lt->rt

I use Windows Server 2003's scheduled tasks to start the process at 3:39AM each day. Although I have the 'stop taks if it runs more than.. hours' set, it only kills the bat file, so if this happens, it starts running multiple BC sessions to the same remote client, which only kills our bandwidth even more.

BC ver 3.1.9 build 11282

[Chris]

Did this happen after daylight savings time (March 14 for the US in 2010)?

It might help to load your saved session interactively to see what kind of differences are indicated by BC. If many of your files have a 1 hour time difference, it is probably a daylight savings time issue that you can fix by changing a setting in your saved session. To ignore DST differences, select "Session > Session Settings". Go to the "Comparison" tab. Check "Ignore daylight saving difference (1 hour)".

[gbenson]

This has been happening at least once every 2-3 weeks for the past year or two with BC2 and BC3 on multiple servers, to multiple remote sites. (I actually still have an email I sent to scooter support on this back on 09/19/2008). The only 'common thread' is it more often than not seems to happen when replicating to a remote site that has a slow connection (China, India). That said, it has happened to some with faster connections and we get calls from IT wondering why my server xferred 20GB across the network during the weekend.

If I run the session interactively it shows the 3825 number as needing to be copied. Daylight savings also isn't the cause, since this script runs daily, and it would have been caught up after a few days (and its been an ongoing occurance for the past year or two).

Although I know nothing about the inner workings of BC, it acts to me like when BC first fires up and is doing a compare, it gets partway through then some sort of network 'hiccup' causes the BC3 session to fail to get the correct information on the remaining files, or thinks the remaining files aren't there, so it just copies them all.

[Aaron]

A network hiccup shouldn't cause the kind of behavior you are seeing. It can sometimes kill a session entirely and fail to reconnect, but it shouldn't proceed to copy everything.

I agree with Chris; I would suggest running the compare interactively a few times. For some reason, the comparison is returning "different" when you think it should not be. But we do not know what difference it is returning.

What comparison criteria is your session set to use? Timestamp and Size? Since it is a mirror command, if either of those values reports back as different, then it will see the file pair as different and require a sync. Is it possible that either the timestamp is off (as Chris suggests) or that the Size is mis-reported (can happen 100% of the time if an ASCII transfer is used to copy text files between a Windows and Linux machine, which changes the line ending characters used and subsequently the size of the files after they are transferred).

The Binary vs. ASCII can be configured in the Tools menu -> FTP Profiles, under the profile's Transfer tab and can be switched to binary to guarantee similar file size, but that should affect text files every time they are transferred.

[gbenson]

Opening interactively never returns more than a few thousand new files to be copied (out of about a million). However it takes about 4 hours to open the session, so we don't do it interactively that often.

If this were related to the timestamp I would expect to see it be consistent, in that if the timestamp on the remote side were off, why would reloading the session cause all the timestamps to suddenly become 'fixed' on the remote end? If I kill a session trying to copy all the files, then it stands to reason that it has not modified the timestamps on the remaining 300,000 files, so any subsequent session would find those remaining files and try and update them. That is not what happens, reopening a session (interactive or scripted) will only copy a couple of files that did actually change.

My comparison settings are below (note its a windows machine to windows machine, both running Server 2003)

Quick tests:
Compare file size
Compare timestamps (2sec tol)

[Aaron]

Is this a Network share or FTP that you are accessing? If FTP, which FTP server software and version are you using?

Would you be able to email us a copy of your support.zip from the Help menu -> Support; Export and a screenshot of the interactive session and logs so we can try to eyeball any differences or oddities. [email protected]

Your BC Script should function the same as when running the same steps with the graphical interface. Are the script and graphical instances being run by the same user account? Normally the best way to troubleshoot these issues is to get the graphical interface working first, since it can provide a bit more feedback (show files and timestamps before a sync occurs, a preview of the actions a sync will attempt to take vs. successfully takes, etc).

[gbenson]

Network Hiccup

I have confirmed it is the network hiccup that is causing this. When BC is listing the remote server's contents, if the network is interrupted or packets lost, the listing gets truncated and all files in the truncated listing are identified as orphans and copied to the remote server. I have been able to reproduce this in the GUI.

[Aaron]

Hello,

Are you connecting to a network share (samba) hosted by a Windows machine? If so, which version of Windows are you using as the remote server?

[gbenson]

Windows Server 2003 to Windows Server 2003. Accessing standard windows shares using \\server\share

Would think you'd be able to duplicate this by just yanking the cable on the remote machine while its comparing the folder contents and immediately plugging it back in. It will want to copy a whole chunk of orphan files to the remote machine that are 'missing' due to the network interruption, even though they already exist on the remote machine.

[Chris]

This sounds like a timing issue. We've had a handful of other customers report this same problem when network connections dropped while BC was building the list of files to be synced.

I've tried repeating this myself by unplugging network cables while running a folder sync, but I haven't been able to repeat it yet.

I'll add making BC's sync behavior on bad network connections to our wish list for a future version of Beyond Compare.

[gbenson]

Not sure if it helps, but there are close to 1 million files in the main folder, so it could have something to do with how poorly windows handles folders with lots of files in them. It also occurs most frequently on slower connections from the US to our China, India, and Japan sites. Not sure if the other customers have either of these issues in common with their data.

[gbenson]

Curious if any progress has been made on this? Our IT dept just instructed me to find an alternative to Beyond Compare because this issue keeps causing massive amounts of data to be transferred.

I just downloaded v3.3.3 because I noted some of the release notes (for v3.3.1) referred to:
"Improved error handling of network errors when loading folder listings." but the problem appears to be just as bad, or even a bit worse, in v3.3.3.

If you need to webex in or anything to diagnose this further please let me know.

[Aaron]

Hello,

We may be able to get your a debug build to help figure out what is going on in the background. Would you be able to email into [email protected] with a reference to this forum post so we can get your email? Or is the email registered with this forum account a good email to contact/send a debug build with?