MD5 checksums

[daxliniere]

Hello forum!
I'm a long-time BC2 user and I just heard about MD5 checksums and why they are so great.

It is an established fact that MD5 is far superior over CRC-32 checks for ensuring that what you copy is ABSOLUTELY IDENTICAL*, which in my line of work is essential.

Thereare several programs that can create and check a batch of MD5 checksums, but none of these have the flexibility and speed when it comes to copying the way BeyondCompare does.

My request: Please implement MD5 checksums as an additional comparison method in BC3 (then I will VERY happily upgrade!)


Since MD5 is an open standard, there's no problem of copyright infringement and I understand it's easy to implement because there are quite a few very fast libraries (DLLs) floating around the net.

So, please let me know how you feel about this.

Thanks so much,
Dax Liniere.


*Yes, Binary Comparison is another way to guarantee that files are absolutely identical, but as we know it's very S-L-O-W

[Zoë]

Are you uploading to an FTP server that supports the XMD5 command (most don't)? If not, there's absolutely no difference between a CRC or MD5 calculation and a binary comparison. In both cases it has to read the entire contents of both files to compare them.

[daxliniere]

Hi Craig,
Thanks for your quick reply.
I'm production manager for a radio station and my line of work involves lots of audio file copying. This is the sole reason I recommended BC2 to be purchased.
We don't do any FTP transfers or merging of multiple versions of documents, I just love the interface and options like "orphans only" or "mismatched", etc.

So how does that influence your answer

[Zoë]

You really won't gain anything whatsoever from us adding MD5 comparisons right now. If we had a program that ran on the remote computer, calculated the MD5s there, and just transferred those, it would be faster. As it is it, to calculate the MD5s right now, we have to download the entire file, which means it will take the same amount of time as a binary comparison. Adding support for remote calculations is on the wishlist, but it's a lot more involved than just adding the MD5 code.

[daxliniere]

Sorry, I should have been more specific - We only use BC2 to copy files from internal to external HDDs and back again, hence, there is no remote computer.

[Zoë]

In that case there won't be any difference in speed between an MD5 check and a binary comparison.

[daxliniere]

So then why is CRC32 faster than binary?

[Michael Bulgrien]

CRC32 is not faster than binary unless you are comparing files to an FTP server that provides precalculated checksums to Beyond Compare. If used on a local network, the entire file on both sides of the compare must be read in order to calculate and compare their checksums. Binary is actually faster than CRC32 for comparisons on a local network because the binary compare stops reading the file pair as soon as it finds a difference.

[daxliniere]

aaaaHA! I see. Thankyou for that information.

So does that mean the only reliable way to copy files is to use binary comparison??

[Michael Bulgrien]

I'm not sure what you mean by "the only reliable way to copy files". A checksum is not used to copy files... it is used to determine if the content of the files is the same. When Craig said:

If not, there's absolutely no difference between a CRC or MD5 calculation and a binary comparison. In both cases it has to read the entire contents of both files to compare them.

He was responding to your comment:

Binary Comparison is another way to guarantee that files are absolutely identical, but as we know it's very S-L-O-W

Since the entire file must be read to calculate the checksum, neither CRC32 nor MD5 checksums will speed up comparisons on your system. I still use CRC in BC3 at times just as a visual indication showing whether a file pair is identical or different. It doesn't speed up the compare, but after copying files from one disk to another, seeing the identical checksum on both sides of the compare verifies that the copy was successful...else the checksum would be different.