Speed

[markm9999]

Hi,

Background:
I'm running Centos 6.3/BC 3.3.5 trying to compare 2 large (>6TB) file collections mostly (.ISO, .WAV, .MP3), because I'm moving to a new & larger array. Copying files, I was ranging from 75-100MB/sec (smaller files) to about 175-250MB/sec (large files).

Problem:
Attempting to compare folders using either 'Compare Contents->Binary' or '=? Compare Contents...' options, BC reports it will take >6000 minutes!

When comparing a single file (8GB ISO) using these same methods, it takes about 5 min to complete. But if I use "Quick Compare (HEX)" it takes about 50 secs. This is also the approx the time it takes for the linux 'CMP' command to complete.

I don't know what is causing this slow-down (GUI updates?), but something is not quite right. I can definitely see that the drive lights are only 'blipping' as opposed to almost full on for the 2 quicker methods.

To the Scooter Team: what accounts for such a huge difference?

To All: I'm a noob when it comes to bash scripting, but I'd like to put together a script to compare this collection and output to a file a list of path/files that have differences.

EDIT: I'm also fine with using a combination bash/BC scripting, if the comparison can be done just as quickly.

Thanks!
Mark

[Aaron]

Hello,

In the Tools menu -> Options dialog, Startup, for your option "When starting with file comparison, show quick compare dialog", which content compare scan does the quick compare use?

How many files are in your Folder Compare? There is a degree of graphical interface usage, and if the file count is greater than our max test case you may run into performance issues.
http://www.scootersoftware.com/suppo...z=kb_maxfilev3

Do you see improvement if you increase the binary buffer size? This can be done in the Tweaks dialog (Ctrl+Shift+T), File Views tab, Buffer size for binary compare.

[markm9999]

Aaron,

Sorry for the late reply, holidays and all...

I had forgotten about the 'Tweaks' dialog!

Interactive 'compare binary' (64535B/1M/32M/64M/256M/512M buffer sizes):
compared 20 items. All size completed in approx ~4 minutes! (stopwatch)

In other news however, I did whip together a script and here is the reult of the same folder compare of 20 items/3.96GB size:

Script:
Successfully compared 20 items. Completed in 20.5 seconds.

Clearly, there is something going on. Increasing the buffer size appeared to only change the amount of 'refreshes' of the currently 'comparing (size) of (file size) as I would see it jump by the amount of buffer capacity.

Edit: During script processing, I see what I think is a jump of about 128M on each refresh, however this refresh occurs approx every 1 second, where as interactively the larger the buffer the slower the refreshes

Edit2: Script processed 3.38TB worth of 830 large ISO's in 4 hours, 20 minutes


Mark

[Aaron]

Hello Mark,

I would initially suspect that the script might not be running the same type of comparison on the same set of files. This could be caused by filters or session settings. Would you like to email your current BCSupport.zip (Help menu -> Support; Export) with a saved session of the long folder compare, and a copy of your Script.txt file? We could analyze them. Our email is [email protected] and please include a link to this forum thread for our reference.

[markm9999]

Aaron,

Posting code here so others can see. Support package emailed to address provided.

    
    
#    %1 = Filter (*.*)              (manditory parameter)
#    %2 = report name               (manditory parameter)
#    %3 = left folder to compare    (manditory parameter)
#    %4 = right folder to compare   (manditory parameter)



# -------------------------------------------------------------------
# start the log
# -------------------------------------------------------------------
log verbose "bclog-%2.txt"


# -------------------------------------------------------------------
# use "criteria" in conjunction with "expand all".
# This is equivalent to the "Folder Compare-Session Settings" dialog.
# If not set, it's as if the 
# "Folder Compare->Handling->Automatically Scan Sub-folders in background" is turned off,
# which is what you want if you *DON'T* want "expand all" to start the comparison
# -------------------------------------------------------------------
#criteria binary
filter "%1"

# -------------------------------------------------------------------
# load the folders to compare
# -------------------------------------------------------------------
load "%3"  "%4"


# -------------------------------------------------------------------
# without "criteria" being set, "expand all" just expands all the
# sub-folders, but no comparison is performed on folders/files so it's fast!
# -------------------------------------------------------------------
expand all


# -------------------------------------------------------------------
# select what we want to compare
# -------------------------------------------------------------------
select all


# -------------------------------------------------------------------
# now perform the compare, so we get a progress bar instead of a long
# pause in script processing with no feedback as to what is going on
# -------------------------------------------------------------------
compare binary


# -------------------------------------------------------------------
# now let's report on what happened
# -------------------------------------------------------------------
folder-report  layout:summary   options:display-mismatches,column-attributes,column-timestamp,column-size  title:"(%2) folder compare summary"  output-to:"bc-report-(%2) folder compare summary.txt"  
folder-report  layout:summary   options:display-mismatches,column-attributes,column-timestamp,column-size  title:"(%2) folder compare summary"  output-to:"bc-report-(%2) folder compare summary.html"   output-options:html-color

[Aaron]

Thanks for the email. I've replied with a couple of ideas. A pair of log files (one from script and one from the graphical interface; both set to verbose) might help.