Folder Compare : Queue sync jobs option

[LesterB]

Dear Scooter Software Team,

Good day!

I am using folder compare on a *huge* drive (currently contains 11Tb of files).

Instead of mirroring the parent folder, I find that breaking the whole task to a few folders at a time seems to result in collectively much faster sync-times.
e.g.
-- mirroring parent folder indicated 5,000+ minutes (3.5++ days)
-- mirroring the subfolders around 10 at a time took around a day (including the time between "sync jobs" when I'm away from the screen (need to sleep/eat too).

==============
Projects A (parent folder)
-- project x1
..
-- project x200++

Projects B (parent folder)
-- project y1
..
-- project y150++
==============

I cannot use scripts because the subfolders change (renamed, additions/removals).

Is there a way to QUEUE the scanning+sync jobs, instead of them being run simultaneously (also slows down the sync process -- physical drive heads are reading data all over the place).

What I'd like to do is basically:
1. Highlight folders x1-x10, right-click > mirror-to-right > sync
2. Highlight folders x11-x20, right-click > mirror-to-right > sync

In the current build, job#2 will run concurrently with job#1. Is there a way for me to queue up job#2 instead, so that it will run only after job#1 is finished?

Thanks in advance, and looking forward to hearing from your team soon.

Regards,
Lester

[Aaron]

Hello,

BC3 does not currently support queued operations. If this were over an FTP, you could limit the number of connections and then start multiple processes, but we do not have this level of control for standard transfers. It is something on our wishlist; I can add your use case notes to our entry on the subject.

If you were to use script, you can use general file name filtering to show only folders that start with the letter a, b, and c for example, and then only sync those. Since filtering is a series of steps, you can have a single script parse through the alphabet (alphanumeric + all non-alphanumeric to capture symbols like _), broken into parts, to perform the sync in multiple parts. If you would like to pursue this strategy in greater detail, just let us know and we can help with the script specifics.

[LesterB]

Dear Aaron,

Thanks for your reply. We're not copying files over FTP, it's just within the same computer (File Server).

I have used BC scripting and continue to do so on our other site offices.

However, scripting in this case is a bit ineffective. Here's an example:
1. We have around 10 folders in the root of the drive
2. One folder could have something between 50-200 projects (i.e. subfolder). This can change as projects move between "phases". Below is an simplified version of the process:

e.g. This week:
---- Phase1_Projects\
----------- Project A009
----------- Project A008
----------- Project A007
----------- Project A006
---- Phase2_Projects\
----------- Project A005
----------- Project A004
----------- Project A003
---- Phase3_Projects\
----------- Project A002
----------- Project A001

Next week:
---- Phase1_Projects\
----------- Project A010
----------- Project A009
----------- Project A008
---- Phase2_Projects\
----------- Project A007
----------- Project A006
----------- Project A005
----------- Project A004
---- Phase3_Projects\
----------- Project A003
----------- Project A002
----------- Project A001

3. If I manually (or via script) try to sync "Phase1_Projects", BC already hangs, due to the sheer volume of filenames it needs to read through. It is in this pre-sync phase where BC seems to struggle with.

4. However, if I try to sync the folders inside "Phase1_Projects" one at a time, BC seems to be able to accommodate the task.

5. If I try to select each folder, and run a sync on each, they all run simultaneously. Which makes the cumulative time of the sync operations longer.

So the best option we have right now is to run a sync on each subfolder (or even sub-subfolders), but do it manually one at a time. Which is unfortunate, because we can't do this at night.

What would be great is if I select a folder, trigger a sync task, but having ticked/indicated that I wanted it queued, BC would then simply create a script on the fly based on the selected folders/filters, and add it to a list/queue of script-based sync tasks. A queue-manager process would then run the scripts as needed (sequentially).

Then of course, these are just my thoughts, and maybe I'm just oversimplifying the solution. I am hoping your team can see the reason/need for task queuing feature.

Another alternative fix/solution is the option to limit the reading of files/folders on a per-subfolder basis.

Basically, if I can tell BC to:
1. Please sync the "Phase3_Projects", process each subfolder one at a time
2. However, instead of reading the files/folders list in advance the whole of "Phase3_Projects", please treat each subfolder as a mini-sync task, i.e.

1.a. Read the folders/files list of "Project A003".
1.b. Sync folder "Project A003"

2.a. Read the folders/files list of "Project A002".
2.b. Sync folder "Project A002"

3.a. Read the folders/files list of "Project A001".
3.b. Sync folder "Project A001"

This would bypass the issue of the pre-sync operation hanging due to large numbers of files.

Thanks for taking the time to read through my post.

LesterB

[RodolfoGiovanninetti]

Just my two cents.
You might want to run a batch file that starts B.C.3 many times, each one with a different path.
For example, in Your example, You might run
start /wait bcomp "Phase1_Projects\Project A009" and other parameters.
start /wait bcomp "Phase1_Projects\Project A008" and other parameters.
start /wait bcomp "Phase1_Projects\Project A007" and other parameters.
start /wait bcomp "Phase1_Projects\Project A006" and other parameters.
start /wait bcomp "Phase2_Projects\Project A005" and other parameters.
and so on.
You said that the directory structure changes, so You would have to change Your batch file, but that might be done during the day, whereas the batch file could be run at the evening.
You might want to try to automate the creation of the batch file.
For example, "dir /S /Ad /B" shows one line per directory.
There might be too many lines.
I use TakeCommand, at www.jpsoft.com. They have also a free version, the Light Edition.
With it, You can run a command such as "dir /S2 /A:d /B" and it shows lines going down only to the second level.

Regards

Rodolfo Giovanninetti

[Aaron]

Hello,

How large are these folders? Our current max test cases are here:
http://www.scootersoftware.com/suppo...z=kb_maxfilev3

I would recommend something similar to Rodolfo's idea. If you can pass the folder name on the command line, this can load a specific session or be passed as a parameter to BCScripting:

Bcompare.exe "@c:\bcscript.txt" "c:\Phase1\Project9" "d:\backup\Phase1\Project9"

Where the bcscript.txt would control the sync:
load "%1" "%2"
sync update:left->right

And a bat file or other scripting solution could call multiple script calls sequentially.

Adding the option to queue operations is something on our wishlist, but is not likely a feature we will be able to implement soon.