Synchronising very large files

[GrahameGrieve]

I'm trying to use BeyondCompare to compare my local dev copy of a very large set of files and the set on the web server. There's upwards of 1000000 directories, 120GB of content

BeyondCompare compares them- that works fine. But it will take a month or more for BC to work through all the directories and it keeps losing contact as I switch files. Also, it has died twice with no notice on me, a day or say into the compare. Running out of memory?

Is there any way that I can break this task up to not compare sub folders I have already compared? Is there any other tricks I can use to help make this manageable> is there any way that I can pause the scan when I switch networks/locations?

[Aaron]

Hello,

That directory size and quantity is not too great, but if it is a particularly slow network connection it could have the issues you describe.

I would recommend using either subfolders as the base folders, to break up the comparison as several smaller comparisons. You can also use File Name Filters to limit what is built, such as three sessions, each uses one of the following in the toolbar.
[A-Ka-k]*\
-./*.*;[L-Sl-s]*\
-./*.*;[^A-Sa-s]*\

The first is A through K folders and base level files,
the second is L through S folders and no base files
and the third is Not A through S (everything else) and no base files. You can tweak the letter ranges or add additional steps, as long as all items are logically included in one of the filters.

There isn't a way to pause a comparison load to resume later.

[GrahameGrieve]

Thanks. I'm just trying to understand this. The File Name Filters... you refer to Rules... Name Filters... which has file & folder include and exclude... I'm not sure about the interplay between the file and folder filters...? Where do the filters you provided go?

[Aaron]

Hello,

The filters above could be directly pasted in the Folder Compare's main interface Toolbar, Filters toolbar ("*.*" area). In the Session Settings dialog, Name Filters tab, you can place the Include Folder filters without the trailing \
[L-Sl-s]*
and in the Exclude File section, when needed
.\*.*

The Session Settings dialog and the Filters toolbar populate each other, so adding to one will then add the syntax needed into the other.

The above 3 are 3 separate filters that logically divide the visible subfolders into 3 parts: A-K, L-S, and T-Z+everything else. You can customize the ranges if you need to cut the comparison into more parts, such as A-C, D-G, etc, as long as the last filter is then "Not A-G". The -.\*.* is added so the items not in base folders are only included once.

The goal is to set filters so only a subset of the folders need to be scanned for any specific task. The above example breaks the comparison into 3 parts, so each part size is more manageable and quicker to scan.

[GrahameGrieve]

So it seems as though the filter applies to sub-directories as well. So I can review A-K, but only the A-K directories...

[Aaron]

Hello,

Could you include the text snippet from the Filters toolbar section? Depending on how it is defined, you may have a filter that will specifically include the top level and include all the subfolders, or on that excludes the folder pattern in any subfolder. You should use Include filtering on the folder name to find the top level (and any subfolders inside)