Copy then compare?

[kydar]

I have some large folders to copy and compare. The initial copy usually takes several hours (we're talking hundreds of gigabytes) so I like to let it run overnight. I would like to compare the folders after the copy, to ensure that an accurate copy was made. Problem is, I don't want to get up at 2am to see if the copying is done and then start the comparison. Can it be done in a single step?

[Chris]

If you turn on binary comparison in session settings, BC will compare the contents of files. Note that this will compare all files in the folders, not just the files that are copied if you are only uploading a subset of files in the folders.

To turn on binary comparison in the Folder Compare:
Click the Rules toolbar button (referee icon).
Go to the Comparison tab.
Check Compare Contents.
Select Binary comparison and click OK.

An option to run a binary comparison only on copied files is on the feature request list for a future version of Beyond Compare.

Beyond Compare also supports command-line scripting if you'd like to automate multiple operations. See the Using Beyond Compare > Scripts topic in the help file for details.

[kydar]

Thank you for the quick response! Since I am copying entire folders, your suggestion should do just what I need.

[kydar]

So I tried Chris' suggestion, but it didn't work out very well. What would happen is, A file would be copied, and when it was done, the icon between the source and destination files would change to an hourglass, presumably meaning they were being compared. However at the same time, it appeared that it was trying to copy the next file in the sequence. So the drive was trying to read and write at the same time, and this brought the whole process down to a standstill. It went far smoother letting the copy job complete, then starting the compare job. The only problem with that is, as I said, it requires human intervention in the absence of a script. And by the way, I took a look at the scripting reference, and I don't think I could do that. I used to write some .BAT files in the old DOS days, but BC's script language is not very intuitive to me. Maybe if I offered $5 for someone to write a script for me??

Edit: so I am trying the exact same thing now except in Linux instead of Windows, and it's running pretty well...no appreciable slowdown when doing a copy/compare. So it would seem the Windows is the bottleneck, not BC. Still, I'd like to be able to write a script to do the copy first, then the compare. Anyone want $5? I'll paypal it to you.

[Chris]

Here's an example script to copy orphan (only on one side) and newer files left to right, then run a binary comparison and list different files in an HTML file.

load c:\folder1 d:\folder1
sync update:left->right
criteria binary
expand all
folder-report layout:side-by-side options:display-mismatches output-to:c:\report.html output-options:html-color

To run the script, save the above file as a text file, then run it using the command line:
"c:\program files\beyond compare 4\bcompare.exe" "@c:\script.txt"

The @ character makes Beyond Compare run a file as a script instead of loading it for interactive comparison.

[kydar]

How would I run from a command line and pass parameters to the Linux version of BC? Something like:
sh /usr/bin/bcompare @/home/me/script.txt
?

[Chris]

On Linux, bcompare should be in the path, so you can run:
bcompare @/home/me/script.txt

Command line arguments other than the script path on Linux use dashes instead of /.

Run a script silently on Windows:
bcompare /silent @script.txt

Run a script silently on Linux:
bcompare -silent @script.txt

Note that Beyond Compare on Linux requires an X-Window session (GUI) to run, even for a script. This means a Beyond Compare script will not run if you only have an SSH terminal to the Linux system and they will not run in a cron job.

[kydar]

Tried the sample script. It fails thusly:

>>load /media/me/sourcedir smb://drobo.local/public/subdir1/subdir2
Unable to load smb://drobo.local/public/subdir1/subdir2: Unexpected output:
Load comparison: /media/me/sourcedir <->
Fatal Scripting Error: Unable to load base folder

[Aaron]

Outside of script, can you launch the graphical interface, then copy and paste the text of the base folder paths into the Left and Right side of the Folder Compare, then load the comparison?

I assume /media/me/sourcedir loads correctly, but smb:// is failing? Can you load directly to the smb:// location using your file Explorer with the same text syntax? Which version of Linux are you running?

[kydar]

Hi Aaron,

Yes, I can paste the smb:// link into the BC GUI and it loads just fine. I can also paste it into Nemo's address bar and it loads. And yes, /media/me/sourcedir loads fine as well (even from the script). Using Linux Mint 19.3.

While it would be nice to automate the process with a script that completed the copy process, then initiated the compare process, I found the workaround that Chris gave me (checking compare binary contents under Rules) to be adequate. As I said, under Windows, trying to copy and compare at the same time brought the system down almost to a halt. Presumably due to the fact that when file1 is copied from the source to the destination, then file2 starts to copy while file1 is read back for comparison. In other words, the destination disk is being asked to write file2 and read file1 at the same time. Windows can't seem to handle that. But under Linux, it seems to work ok.

Great product you guys make, it was one of the best software investments I ever made.

[Aaron]

Hello,

What was your order of operations for using the smb:// command in the interface? On the first connection, smb:// will try to create the bind. Was that done in BC4 or the file explorer? After the bind is created, future calls would then be expected to work. If you reboot and try the BC4 interface first, does this work? If you can connect that way, then immediately attempt the script call from the terminal, does this work? Generally, if the bind can work from BC4's interface, it should be able to work from a script call; I'd like to try and determine which combination of calls and order cause it to fail in scripting but works in the interface.

For the Windows feedback, copying and comparing at the same time is intensive, but generally works just fine. BC4 is a bit more demanding than the average file explorer, since it is loading all subitems in all subfolders (and, in this case, also performing a temp download and binary scan). Sometimes, this can stress hardware or drivers that are not used to that level of workload; are there any driver or firmware updates available for NAS device?

[kydar]

Ok so I just tried something. I rebooted the machine, which disconnected the SMB share. As soon as I rebooted, I opened BC GUI and started a new folder compare. On the address line, I manually typed in smb://drobo.local/public/subdir1/subdir2 and pressed enter. A popup asked if I wanted to connect to the drobo with a username/password or anonymously. I selected anonymous. Another popup said that the subdir did not exist (which is correct because I had not created it yet) and would I like to create it? I said yes, then a popup gave the same error as before:

Unable to load smb://drobo.local/public/subdir1/subdir2: Unexpected output:

So then I opened Nemo and put the same address in the address bar. The popup asked if I wanted to log in with a username or anonymous, and I selected anonymous. Without further prompting, subdir2 was then created. Then I closed Nemo and BC GUI, and ran the script from a terminal window. And whatcha know, it started running.

So it seems that, for whatever reason, one must connect to the smb share thru Nemo, not BC GUI, before running the script. BC, although it offers the same popups asking to create the subdir, is unable to do so. Once it is created though, the script runs.

So I am happy. If you would like me to do any further testing for debugging purposes, I am willing to do so, but as far as I'm concerned, the problem is solved. Tell Chris to PM me his email address and I'll paypal him the $5 as promised.

[kydar]

A couple other quirks I discovered about using the script that Chris gave me versus updating side-to-side from BC GUI:

1. Empty subdirs are not copied. I discovered this by looking at report.html which the script generated. It looks like there is an empty.folders selection criteria in the script language reference that could change this behavior, but it's not a big deal to me anyway.

2. File modified times, but not folder modified times, are copied by the script, whereas both are copied by the GUI. (Folder modified times are set to "now" when using the script). This is easily fixable by just doing a touch at the end of the process.

[Aaron]

Hello,

Interesting. In my experience, anonymous access normally blocks folder creation, so maybe there's some difference in how BC4 is logging in 'anonymously'. Does the dialog for user/pass look identical (a system dialog) or are they different (an internal bc4 dialog)?

1. add create-empty as an option to the sync script line:
sync create-empty update:left->right

2. If you use the GUI, then close the gui and re-open/compare, are the Folder Timestamps still preserved? My expectation would be for the folder timestamps to be updated, as after copying the folder as part of the sync, copying a file into it almost always updates the folder timestamp. This is usually the behavior when using BC4 or other applications for a regular folder copy. A touch can then re-update the folder Last Modified timestamp (but not Created or Accessed).
Sometimes with a NAS device, it reports a cached, incorrect timestamp back, and it takes a Full Refresh or reload to catch that the timestamp is actually changed. I would suggest verifying with BC4 reloaded and Nemo to double check.