Removed "Bypass disk cache during binary comparisons" option.

[Dave_L]

In the "Notable Changes" for Beyond Compare 4.2 Beta at http://www.scootersoftware.com/download.php?zz=beta42, is listed:

"Removed "Bypass disk cache during binary comparisons" option. Due to changes in hardware, operating systems, and drivers, it no longer works as originally intended."

Could you explain further? Even if this function is not fully supported by the host operating system, is there no advantage whatsoever in providing the setting? (My O/S is currently Ubuntu 12.04.5)

[Zoë]

Hi Dave,

I'm happy to go into more details; I figured we'd have at least one person ask.

The short answer is, No, I don't think there's any advantage to keeping the setting, and I actually think it's harmful to have it, because it gave people a false sense of security.

The long answer:

The setting was originally added to support verifying copies to floppy disks. The code hasn't really been touched since then, and it isn't possible to make it work for that purpose on other drive types.

As background, on Windows it was implemented using the FILE_FLAG_NO_BUFFERING flag, on Linux it opened the file using O_DIRECT, and on macOS it called fcntl with F_NOCACHE. The one thing all of those have in common is that they're largely intended as an optimization for applications that do their own caching, not as a way to verify things.

On Windows:

1) Microsoft's official documentation on the flags we're using say that it isn't supported by the CD/DVD driver nor on network drives and does not affect caching done by the hard drive itself. That last has become a more significant as HDD onboard memory caches have grown, since SSD and hybrid drives have multiple layers of cache, and since SSDs have complex logic to handle bad sectors.

https://support.microsoft.com/en-us/kb/99794

2) Windows does not have a way to force writing a file directly to disk and then reading it back without accessing the disk cache. It used to for some cases, but bypassing the disk cache isn't supported by reads anymore, and doing it on writes requires passing a flag when opening the file, which we can't do since we're using Windows' CopyFile API. Drivers and/or disks could and did ignore the flags even when they were supported. (OSR is a consulting company for file system drivers; a Microsoft employee also chimed in).

https://www.osronline.com/ShowThread.cfm?link=264166

3) XCOPY's /V (verify) command just checks that the the source and destination are the same size. In MS-DOS it used to be able to issue a "Write then verify" command instead of a normal "Write" one, which would tell the driver to write the data and then immediately read it back and compare the results, but the Windows driver model no longer supports that.

https://blogs.msdn.microsoft.com/old...21-00/?p=92911

I'm more immersed in Windows developer culture than Linux/macOS, so it was easier for me to find (or already know about) the justifications for removing it on Windows, but the situation would be similar on the other OSes. The flag was advisory at best and was silently ignored if the file system didn't support it. In those "not supported" cases, BC was still much slower if the flag was on because we were using smaller read buffers. Since it was intended for already slow floppy drives, we didn't put effort into optimizing the reads, and since it was "working", there wasn't much incentive to improve it.

If we wanted to keep it, we could redo everything to do optimized reads and theoretically improve I/O for large comparisons, but IMO, that's a new feature request, not a bug fix, and one that would have to compete with all of the other items in the wishlist.

[Dave_L]

Zoë, thanks for the detailed answer.

[florian]

Until now I was relying on the 'bypass disk cache' option to ensure that a newly connected USB thumb drive is actually read and not data from any cache.
What would now be the correct way to test multiple USB thumb drives (same hardware, same volume label etc.) which are expected to contain identical content against reference data from the hard drive?

[Aaron]

Hello,

Are you seeing different behavior with the option disabled/removed? As Zoe mentioned above, the initial use of the option was for floppy disks, and it was not intended or supported by Microsoft for newer types of access. Even with the option enabled, it is possible to get cached results back.

[uidd]

I'm on v3 so it's theoretical for me, but I think it wouldn't hurt to leave this option there even if it's not a cure-all, and even if it's slow in its current implementation.

I don't understand the reasoning, on Windows at least. There are always multiple layers on the way to the hardware that can decide to ignore flags, but the main cache people want to avoid is the OS cache, and Windows' own caching behavior is fully specced. FILE_FLAG_NO_BUFFERING avoids the OS' eager and potentially big cache.

CopyFile() indeed limits what you can do, but CopyFileEx() has COPY_FILE_NO_BUFFERING (but not on XP). I don't know if that also implies FILE_FLAG_WRITE_THROUGH, but as far as I understand the main extra thing that adds is preventing metadata write caching.

[Aaron]

As Zoe mentioned above, the combination of modern Windows OS, drivers, and hardware, have made using the option unreliable, which lead to a false sense of security since, if it is running slower, it feels like it is helping. While it's theoretically supported in some combinations, there were a growing number of real world examples where users thought they were bypassing cache but weren't, and we couldn't indicate or guarantee the option as described. Adding improved verification methods is something on our wishlist, but it would be a larger overhaul project.

[uidd]

I think adding a warning and a brief explanation is a better interim solution, since FILE_FLAG_NO_BUFFERING and COPY_FILE_NO_BUFFERING should at least be honored by Windows' own caching (else it's a bug to report to Microsoft), and that by itself provides benefit. But all right.

Thanks!

[Aaron]

BC3 can tell Windows to use the flag, but detecting if the results are valid is part of the enhancement Zoe mentions. The current flag assumes validity, but that flag is both old, and only one of several caches that will be encountered. While we cannot guarantee the effectiveness of using the flag, we can reasonably assume the user is likely still encounter at least a level of caching. Additionally, setting the flag greatly increases the comparison time, which gives a strong impression that it is working, one or more caches is still in use.

[uidd]

Thanks. I wasn't trying to retread the thread, sorry if that's how it appeared.

[Dave_L]

I just noticed that my thread was "stickied", and wanted to thank you for this great honor.

[Aaron]

It's no problem. The desire to keep the option is part of why it was removed. Removing the option wasn't an easy choice because it gave the impression it was working, but given the current behavior and changes in modern OS caching, it was better to remove it in the meantime.