Flat folder to folder comparison with several files of same name, size and date.
Hello world!
I have been a Beyond Compare 3 addict for the past six months. It all started out as a need for a software application designed specifically to compare files. You see I had this heavily loaded collection of all sorts of data that have accumulated over the past five years or so. Great part of it comes from the Web a.k.a. Internet a.k.a. The Net. As such, there are plenty of duplicates of both Audio files and other file types or several unwanted versions of same files and information. But it also contains virtually all my old backup copies of my personal and important information which is not replicable. In the past I used to make backups on CD and later on DVD discs. I don't know the exact figures, because I forgot to note it down when I began with this project. But this massive collection is on the Gigabyte scale, that's for sure. We are talking about hundreds of Gigabytes here, I would guess somewhere between 300 to 400 GB at least.
Since the collection as whole contains both useful and useless data, deleting it all together and starting all over was out of the question. So I really had to dig in to it, clean up and sort things out to get even close of an overview of it. I am usually an orderly person, and this will for sorting these things out kind of shows it doesn't it? I am also a collector so I like to keep old good things mostly because of the memories associated to them, and this is kind of reflected on my computer use as well. As such person I have worked out a pretty damn good system for organizing my digital assets (data). But sometimes even I do mistakes and other times I just neglect to keep up with the organization process. Apply time to that, and the deep and complex directory/folder structure I usually use and you end up with a pretty messy, nested folder structure with no end to it and with no logic or overview of it. In the end you don't go through those folders, you just search for the files you need.
This is where a comparison software could come in handy. I was about to dig in to this data mess, and while at it I wanted to get rid of the duplicates for good. Except for saving me disk space, that would also save me hundreds of hours of organizing files that I already have organized once and have a record of. While no comparison software could do this whole work for me automatically, and even if there was such software I would fear to use it on sensitive data. However, it would be of great help to use a comparison software to at least partially automate this long, never ending process. (I have already messed up some of the less important data solely by using Beyond Compare.)
Now it was official. I started planning on how I would perform this trick the best way practically and in the shortest time possible. I also started to hunt for a comparison software. One of my main requirements was that this software would need to have a GUI and be able to compare not only files one to another but to compare folders and whole sets of folders to other folders. So using traditional command line based Diff tools was not an option. Wiki articles File Comparison and Comparison of file Comparison Tools were of great help here. That's how I found out about Beyond Compare. Other softwares that stack to my mind are Araxis Merge, ExamDiff Pro, Kompare, WinDiff, and WinMerge. Some of these are proprietary like Beyond Compare, and some are freeware or OSS. I am a fan of OSS but I prefer what best confirms to my needs. So in the end I chose to go with Beyond Compare. But prior to that I have also tested some of the above mentioned, and I have read some serious reviews on them and on Beyond Compare 3. I don't see Beyond Compare 3 as the winner among them, but it sure did stood the test and it has become one of my favorite tools now.
I am still working on this project, and I am using Beyond Compare 3 (BC3). I estimate that I have done about two thirds (2/3) of the work, leaving 1/3 remaining. I have read most of the stuff written in the Help file for BC3, and by experimenting with the procedures described there I have come to understand it (so please no RTFM here, lol). So I am pretty good with BC3 now. But I just can't figure out how to do the following.
There is a folder called Pictures which is located in the parent folder temp at %UserProfile%\Desktop\. It contains 1798 files and 147 sub-folders, totaling 203 MB. The files are mostly pictures, with very few exceptions. I would like to figure out which one of these files I already have in my main pictures folder called Pictures (Windows special folder) at %UserProfile%. I would like to delete those that I allready have from the %UserProfile%\Desktop\temp\Pictures location, and keep those that I don't have and then move them to the main Pictures folder. How do I do it?
I know this sounds simple, I did similar things many times before, but the problem is that the main Pictures folder at %UserProfile% contains thousands of files that have the same file name and extension but are located in different sub-folders. This seem to give BC3 headaches. When I do a folder to folder comparison here, these are the settings I usually use.
As left folder I chose the %UserProfile%\Desktop\temp\Pictures and for the right folder I chose %UserProfile%\Pictures.
The results?
No results! No equal files found!
Actually, BC3 did help me find at least half of the equal files using these same settings, but only up to some point. I think that when BC3 started finding more then two files of same name, extension, size and date it kind of hit a dead end. That's my interpretation of BC3's behavior. Two files equal is fine, it shows them. But if more then two files in a row are equal it kind of doesn't know what to do so it gives you nothing. It seems as if BC3 is relying too much on the name of the files.
I am using Windows Vista as my primary OS, so I opened up two instances of Windows Explorer where in one of them I have chosen to show the new advanced search features, and in the other I just showed the Pictures folder I want to compare. File by file I took the file name, size, and date and entered it into the advanced search options. This way I was able to find out which files are duplicates so I could delete them. This proves it that there are for sure dublicates of files in these two folders, but this takes so much time to find them this way one by one. Why can't Beyond Compare 3 help me here?
I have made few modifications to the settings in BC3 but it didn't help me find any duplicates. What do I do to make this right so that BC3 can help me find these files?
I love BC3, but on this point it has disappointed me. I would really appreciate if anyone could help me figure out how to do this comparison in BC3, in case I am missing something. If this is something that BC3 lacks in functionality, I expect it to be included in the future.
Thanks in advance!
I have been a Beyond Compare 3 addict for the past six months. It all started out as a need for a software application designed specifically to compare files. You see I had this heavily loaded collection of all sorts of data that have accumulated over the past five years or so. Great part of it comes from the Web a.k.a. Internet a.k.a. The Net. As such, there are plenty of duplicates of both Audio files and other file types or several unwanted versions of same files and information. But it also contains virtually all my old backup copies of my personal and important information which is not replicable. In the past I used to make backups on CD and later on DVD discs. I don't know the exact figures, because I forgot to note it down when I began with this project. But this massive collection is on the Gigabyte scale, that's for sure. We are talking about hundreds of Gigabytes here, I would guess somewhere between 300 to 400 GB at least.
Since the collection as whole contains both useful and useless data, deleting it all together and starting all over was out of the question. So I really had to dig in to it, clean up and sort things out to get even close of an overview of it. I am usually an orderly person, and this will for sorting these things out kind of shows it doesn't it? I am also a collector so I like to keep old good things mostly because of the memories associated to them, and this is kind of reflected on my computer use as well. As such person I have worked out a pretty damn good system for organizing my digital assets (data). But sometimes even I do mistakes and other times I just neglect to keep up with the organization process. Apply time to that, and the deep and complex directory/folder structure I usually use and you end up with a pretty messy, nested folder structure with no end to it and with no logic or overview of it. In the end you don't go through those folders, you just search for the files you need.
This is where a comparison software could come in handy. I was about to dig in to this data mess, and while at it I wanted to get rid of the duplicates for good. Except for saving me disk space, that would also save me hundreds of hours of organizing files that I already have organized once and have a record of. While no comparison software could do this whole work for me automatically, and even if there was such software I would fear to use it on sensitive data. However, it would be of great help to use a comparison software to at least partially automate this long, never ending process. (I have already messed up some of the less important data solely by using Beyond Compare.)
Now it was official. I started planning on how I would perform this trick the best way practically and in the shortest time possible. I also started to hunt for a comparison software. One of my main requirements was that this software would need to have a GUI and be able to compare not only files one to another but to compare folders and whole sets of folders to other folders. So using traditional command line based Diff tools was not an option. Wiki articles File Comparison and Comparison of file Comparison Tools were of great help here. That's how I found out about Beyond Compare. Other softwares that stack to my mind are Araxis Merge, ExamDiff Pro, Kompare, WinDiff, and WinMerge. Some of these are proprietary like Beyond Compare, and some are freeware or OSS. I am a fan of OSS but I prefer what best confirms to my needs. So in the end I chose to go with Beyond Compare. But prior to that I have also tested some of the above mentioned, and I have read some serious reviews on them and on Beyond Compare 3. I don't see Beyond Compare 3 as the winner among them, but it sure did stood the test and it has become one of my favorite tools now.
I am still working on this project, and I am using Beyond Compare 3 (BC3). I estimate that I have done about two thirds (2/3) of the work, leaving 1/3 remaining. I have read most of the stuff written in the Help file for BC3, and by experimenting with the procedures described there I have come to understand it (so please no RTFM here, lol). So I am pretty good with BC3 now. But I just can't figure out how to do the following.
There is a folder called Pictures which is located in the parent folder temp at %UserProfile%\Desktop\. It contains 1798 files and 147 sub-folders, totaling 203 MB. The files are mostly pictures, with very few exceptions. I would like to figure out which one of these files I already have in my main pictures folder called Pictures (Windows special folder) at %UserProfile%. I would like to delete those that I allready have from the %UserProfile%\Desktop\temp\Pictures location, and keep those that I don't have and then move them to the main Pictures folder. How do I do it?
I know this sounds simple, I did similar things many times before, but the problem is that the main Pictures folder at %UserProfile% contains thousands of files that have the same file name and extension but are located in different sub-folders. This seem to give BC3 headaches. When I do a folder to folder comparison here, these are the settings I usually use.
- Show same
- Ignore folder structure (flatten)
- Compare file size: yes
- Compare timestamps: yes
- 0 second tolerance
- Ignore daylight saving difference (1 hour): no
- Ignore timezone difference: no
- Compare file attributes: archive, system, hidden, read-only
- Compare contents: CRC comparison
- Override quick test results: yes
As left folder I chose the %UserProfile%\Desktop\temp\Pictures and for the right folder I chose %UserProfile%\Pictures.
The results?
No results! No equal files found!
Actually, BC3 did help me find at least half of the equal files using these same settings, but only up to some point. I think that when BC3 started finding more then two files of same name, extension, size and date it kind of hit a dead end. That's my interpretation of BC3's behavior. Two files equal is fine, it shows them. But if more then two files in a row are equal it kind of doesn't know what to do so it gives you nothing. It seems as if BC3 is relying too much on the name of the files.
I am using Windows Vista as my primary OS, so I opened up two instances of Windows Explorer where in one of them I have chosen to show the new advanced search features, and in the other I just showed the Pictures folder I want to compare. File by file I took the file name, size, and date and entered it into the advanced search options. This way I was able to find out which files are duplicates so I could delete them. This proves it that there are for sure dublicates of files in these two folders, but this takes so much time to find them this way one by one. Why can't Beyond Compare 3 help me here?
I have made few modifications to the settings in BC3 but it didn't help me find any duplicates. What do I do to make this right so that BC3 can help me find these files?
I love BC3, but on this point it has disappointed me. I would really appreciate if anyone could help me figure out how to do this comparison in BC3, in case I am missing something. If this is something that BC3 lacks in functionality, I expect it to be included in the future.
Thanks in advance!
I just skim-read what caught my eye... Beyond Compare matches/aligns files by filename regardless of how you've configured your compare settings. So a flattened folder view will not identify duplicate files if they have different names. When I am looking for duplicate files, I sort by CRC and manually look for consecutive files with identical CRC values. I don't know if it is possible to generate a report that contains CRC values... I haven't tried it...
That not only equals but it also exceeds the 1782 I originally had in the Pictures folder I wanted to compare to my main Pictures folder. This means it found duplicates within my main Pictures folder as well. Duplicate File Hunter (DFH) seems to be a great tool for finding duplicates, while BC3 is not essentially a duplicate file hunter. BC3 sort of requires you to know the locations of the files you want to compare, in order to figure out if they are exactly the same or not, i.e. if they are duplicates. DFH on the other hand does not have the extensive control and options found in BC3.
Comment