Multiple location large mirror sync

[Gasgem]

Hey great product.

I've started using your product to backup files to a remote Gigabyte network attached storage device.

But I find it is taking a long time to complete the sync
This sync involves about 100k of files ranging for 1kb to 200gb.

I'm wondering if a snapshot would improve the sync speed. Timestamp was my inital idea as it should be the quickest. But it takes a long time doing the this even if few files have changed.

This is the current script:


# Set comparison criteria
log normal "R:\SYNC_LOGS\SYNC_ADEN_D.txt"
criteria timestamp size
# Load the base folders
load "\\aden\D$" "R:\ADEN\D"
#Make the target identical to the source
#includes deleting files that only exist on the target side
sync mirror:lt->rt
option confirm:yes-to-all

any help would be great,

thanks

[Chris]

Re: Multiple location large mirror sync

Using a snapshot probably won't help unless you have a very slow link to the remote location.

To check this, try loading an interactive comparison of \\aden\D$ and R:\ADEN\D and see how long it takes before the Cancel button grays out. The time until the Cancel button disables should give you an indicator if it is the comparison that is taking a long time, or if the time taken is due to files being copied.

Also, you'll want to place "option confirm:yes-to-all" at the start of your script. The command only affects script operations that occur after it.

If you want to try using a snapshot, here's an example script that compares the current state to a snapshot, copies the differences, then updates the snapshot:

option confirm:yes-to-all
criteria timestamp size
load c:\source c:\source.bcss
expand all
select lt.newer.files lt.diff.files lt.orphan.files
copyto lt c:\target
snapshot path:c:\source output:c:\source.bcss


You'll need to create the source.bcss snapshot before you run the script the first time.

Re: Multiple location large mirror sync

Dear

I refer to the above topic posted earlier that we have the same problem for sync files from a distant remote site.

It takes a very long time (around 1.5 days) for file comparison but in fact only a few MB data are being transferred from remote server to the local server.

I wonder if there is script to complete the following process:
- Create a snapshot file in remote directories
- Copy the snapshot file to the local server
- BC2 compare the snapshot and the target folder locally
- Copy the changes files to the local server
- Such job will be scheduled daily as increment backup

This way we utilize the bandwidth for copying the changes files only.

Regards

[Aaron]

Re: Multiple location large mirror sync

Hello,

Would you be able to post the script you are currently using, or email it to support@scootersoftware.com ?
Do you require a CRC content comparison, or would Timestamp/Filesize be enough? Which criteria are you currently using?

If you require a Content Comparison:
There are a couple of strategies that could be employed here. In general, one would have you take a snapshot each day on the remote machine, detect the changes for the previous day by running a script remotely that compares the day old snapshot to the remote folder. Then zip up those changes and send them over to the Local machine. Once on the local machine, run a script that then syncs the zip's contents to the local folder.

Alternatively, if you have an FTP server that supports xCRC, you could also use the FTP to calculate the CRC content comparison.

Re: Multiple location large mirror sync

Dear Aaron

We are on the evaluation stage to look for another backup tool to replace our existing one that with the problem I mentioned before. BC2 seems can do what we expect with its snapshot feature.

We do not require a content comparison. We just want to centralize and maintain an up-to-date copy of the remote data from different remote sites in local server.

My concern is all these jobs have to be run automatically and on a scheduled timetable.

How the script will be? Please help.

Thanks

[Aaron]

That would consist of creating a Windows scheduled task that calls:
"bc2.exe @c:\path\scriptFile.txt"
to run automatically on a scheduled timetable.

The script itself is a series of steps that represent steps you can take in the application. You will want to plan out those steps via the GUI first, and then create script that recreates those steps. There is Script Reference and Samples in the Help file under the Command Line and Scripting section.

One such example would be:
log normal "C:\My Documents\Webpage\Synclog.txt"
filter "*.html;*.htm;*.png;*.jpg"
criteria timestamp:2sec
load "ftp://user@ftp.some-isp.com/webroot" "C:\My Documents\Webpage"
sync update left->right

You can write a bat file that calls a series of scripts in a row to perform multiple syncs one after the other:
bc2.exe @script1
bc2.exe @script2

And, if you want to get a bit fancier, you can use parameters:
bc2.exe @scriptFile folder1 folder2
bc2.exe @scriptFile folder3 folder4
and then refer to the folders in the script as %1 and %2. This would allow you to have a single script file that is then called multiple times.