simple question from technical layman

[tom f.]

hello everybody,

i would like to politely ask a very simple question about the use of bc, since
i am by no means a software specialist in this context and not a native english speaker:


i have backuped many data in the last 20 years on quite some harddrives.
the backups were pretty random from a systematic point of view and some harddrives even contain copies of entire other harddrives etc...

so over all the time i have gigs of double, tripple or maybe even quadruple data on the drives and on single drives themselves, and all i would like to do is the following:

starting the cleansing with a copy of hardrive 1 on a very big new drive and then systematically compare all other drives - only adding to the new big drive what is not already there.

can i do this without being an expert with bc ?
is this kind of operation risky?
...in the end i have to compare and merge about 7 harddrives.

thanks a lot in advance for any kind of information.

[Aaron]

Hello Tom,

Because BC3 does not support an Undo command, there is a bit of risk when performing any specific action. We do try to offer as full a preview as possible; just be aware of the selection of items before you commit a Copy, Delete, etc.

BC3's comparison will align files by folder structure and file name by default. Since you have a variety of folders that have moved around, these would not automatically align, but you could step through and set them individually as base folders, and reconcile the differences within. Once a folder has been thoroughly compared and you've reviewed the differences, you could delete the copy on the small hdd.
We do have a View mode that ignores the folder structure, aligning all files in any subfolder based only on file name. However, duplicate file names are not handled in this instance, and you would need to manually review any that occur.

Does this help point you in a possible workflow direction? Let us know if you have any questions.

[JimG58]

Hi Tom,

I believe I had a similar need some time ago and found a program that worked perfectly for me. It is called CCLEANER and has a "File Finder" options under "Tools" which finds duplicate files across many drives. It has many options which allows you to define what makes a duplicate (content, date, name, size) plus lots of Ignore, Include and Exclude items. It will give you a list at the end that you can check which ones you want to delete / remove.

I had no problems with it finding duplicate files and making it easy to tag them for removal. But, I do not recall it having an undo - so be careful.

With both Beyond Compare and CCleaner, I have a much better organized system.

Good Luck,
Jim

[mmag]

Hello all,
Seems that this situation is going to happen more and more, as data storage is growing, and we use different USB-Local-Cloud-virtual drives, with more and more data, right?

My situation is almost exactly as the one Tom had: dozen drives, folder structure duplicated with modifications, several years of accumulated changes on files and subfolders, zip and archived files…
I was wondering if BC (now in version 4.3.3, 2020 folks!) might generate somehow of report, pair-to-pair comparison, and let later use Excel’s feature called “consolidate”, to calculate the “optimal merge strategy”. Some food for thought:

1. Cost (or effort) of a pair-drive merging operation, depends of both size of files, but much more on number of files.
2. Using BC’s synchronize option might lead to duplicates, on modified structures, for the same files. We should look for the newest to keep, but BC does not reach that point.

Question: if I have E, F, G, H drives, decreasingly ordered on number of files. So E has more files and size that the others. If I do merge E into F and G into H… Would it be less costing that merging sequentially F into G, and G into H? And duplicate cleaning

I guess this forum might be a good starting point to develop an optimal strategy for merging in the era of the big data.

PD: Some data for your info: +10 drives, going from 172k files to 3k files; average drive size 600Gb.

[Aaron]

Hello,

Duplicate searching has been a long-term wishlist item, and finding and resolving between multiple drives (2+) would be nice to tackle. BC4 supports alignment first on file name, and then compares the matching pairs, but does not support this level of scanning for content first (exact or similar) and then finding matches.

[mmag]

Well... Thank you Aaron for your kind reply.
My main problem is not to find duplicates. My main problem is to find the best sequence of comparisons and merging operations, minimizing the number of duplicates that a "simple synchr" operation would lead.
For example: it might not be a good idea to synch 172k files and find that due a different structure, you have duplicated a high percentage, and those files would be later need to be deleted.

I am trying to get "data" on number of files, matches, orphans, etc, between drives, to dump into Excel, and get a sequence for an optimization of minim duplicates generation.

Hope this makes more sense and is better explained.
Thank you!

[Aaron]

If the scenario has only moved files, and no duplicate file names or any renames, you could use the View menu -> Ignore Folder Structure. This would help align any files that have moved to a new folder. However, this still aligns by file name first, so duplicate file names in any subfolder would cause issues. Also, if you are copying from side to side and an item is unaligned, it will copy into the root of the base folder (since structure is ignored). This view option might help with the Excel file generation, however, reducing the amount of manual work you need to do.

It sounds like you would want to scan for files based on content (to handle any renames or moved files). BC4 does not support this style of alignment/scanning (it currently only supports aligning by folder and file name, optionally removing the folder structure), but it is something on our customer wishlist.

[mmag]

Hello Aaron , again... Hope you are all fine and great now in fall 2021.
I was wondering if would make sense to have an "alternative sync" feature that would not copy a file if
(a) there is a file with the same name in the other side,
AND
(b) regardless of the file structure.

That would help to copy files that would be duplicated in the destination drive.

How does this sound? Would it be hard to do?
Thank you very much! Regards, Manuel

[Aaron]

Hello,

I think we have something close to that.

The Folder Compare session has Ignore Folder Structure (View menu) is a mode that removes all subfolder structure and lists the files in a single bucket. Any file name match will align to a match on the other side (regardless of location). This assumes no duplicates (several ReadMe.txt files on the same side) as the alignment still has to occur before the comparison, so a pair will need to be picked first before they can be scanned. Also if you have an odd number (3 readme's on the left and 2 on the right) one of them will be orphan, as the alignment still sets things up in pairs.

You can then set the display filters to show Only Orphans (remove anything that did align), and you'd be left with items that didn't pair up, but review to make sure it isn't other cases as listed above. You could also show Differences and find pairs that did align with differences and Orphans, removing only the equal files from view.

Once the comparison is set, you can select the files and act on them (copy to the other side, etc), but please carefully review the comparison first. There is no folder level UNDO command, so any actions (copy, rename, delete, overwrite, etc) can not be undone. I'd suggest testing on test folders/files first while learning how the comparison is built.