No announcement yet.

Removed "Bypass disk cache during binary comparisons" option.

This is a sticky topic.
  • Filter
  • Time
  • Show
Clear All
new posts

  • Removed "Bypass disk cache during binary comparisons" option.

    In the "Notable Changes" for Beyond Compare 4.2 Beta at, is listed:

    "Removed "Bypass disk cache during binary comparisons" option. Due to changes in hardware, operating systems, and drivers, it no longer works as originally intended."

    Could you explain further? Even if this function is not fully supported by the host operating system, is there no advantage whatsoever in providing the setting? (My O/S is currently Ubuntu 12.04.5)
    Last edited by Dave_L; 28-Feb-2017, 07:24 PM.

  • #2
    Hi Dave,

    I'm happy to go into more details; I figured we'd have at least one person ask.

    The short answer is, No, I don't think there's any advantage to keeping the setting, and I actually think it's harmful to have it, because it gave people a false sense of security.

    The long answer:

    The setting was originally added to support verifying copies to floppy disks. The code hasn't really been touched since then, and it isn't possible to make it work for that purpose on other drive types.

    As background, on Windows it was implemented using the FILE_FLAG_NO_BUFFERING flag, on Linux it opened the file using O_DIRECT, and on macOS it called fcntl with F_NOCACHE. The one thing all of those have in common is that they're largely intended as an optimization for applications that do their own caching, not as a way to verify things.

    On Windows:

    1) Microsoft's official documentation on the flags we're using say that it isn't supported by the CD/DVD driver nor on network drives and does not affect caching done by the hard drive itself. That last has become a more significant as HDD onboard memory caches have grown, since SSD and hybrid drives have multiple layers of cache, and since SSDs have complex logic to handle bad sectors.

    2) Windows does not have a way to force writing a file directly to disk and then reading it back without accessing the disk cache. It used to for some cases, but bypassing the disk cache isn't supported by reads anymore, and doing it on writes requires passing a flag when opening the file, which we can't do since we're using Windows' CopyFile API. Drivers and/or disks could and did ignore the flags even when they were supported. (OSR is a consulting company for file system drivers; a Microsoft employee also chimed in).

    3) XCOPY's /V (verify) command just checks that the the source and destination are the same size. In MS-DOS it used to be able to issue a "Write then verify" command instead of a normal "Write" one, which would tell the driver to write the data and then immediately read it back and compare the results, but the Windows driver model no longer supports that.

    I'm more immersed in Windows developer culture than Linux/macOS, so it was easier for me to find (or already know about) the justifications for removing it on Windows, but the situation would be similar on the other OSes. The flag was advisory at best and was silently ignored if the file system didn't support it. In those "not supported" cases, BC was still much slower if the flag was on because we were using smaller read buffers. Since it was intended for already slow floppy drives, we didn't put effort into optimizing the reads, and since it was "working", there wasn't much incentive to improve it.

    If we wanted to keep it, we could redo everything to do optimized reads and theoretically improve I/O for large comparisons, but IMO, that's a new feature request, not a bug fix, and one that would have to compete with all of the other items in the wishlist.
    ZoŽ P Scooter Software


    • #3
      ZoŽ, thanks for the detailed answer.


      • #4
        Until now I was relying on the 'bypass disk cache' option to ensure that a newly connected USB thumb drive is actually read and not data from any cache.
        What would now be the correct way to test multiple USB thumb drives (same hardware, same volume label etc.) which are expected to contain identical content against reference data from the hard drive?


        • #5

          Are you seeing different behavior with the option disabled/removed? As Zoe mentioned above, the initial use of the option was for floppy disks, and it was not intended or supported by Microsoft for newer types of access. Even with the option enabled, it is possible to get cached results back.
          Aaron P Scooter Software