Exclude files older than... different behaviour

[ThomasB]

Hello, I have a problem, hope there is a solution.

I use Folder Compare (with binary comparison) to verify the integrity of files which I have backed up with another program (SyncBackSE). I compare the backup folder with the source folder, just after doing the backup.
In order not to binary check/compare all files in the folders, which would take a long time (hours), I use the option "exclude files which are older than" in Folder Compare. I thus binary-check just the recent files (a few days before), which should include all my documents which I have recently created or modified (I do backups daily).
The problem arises if I copy an old file to the source folder, then do a backup and then do a binary comparison. Since the file can be older than the days set in the "exclude files which are older than", the file gets excluded and thus doesn't get compared.

Is there a solution for this? I would prefer not to change the date and time of a file, during the backup (maybe SyncBack has such option, but I'm not sure).

I have noticed that Folder Compare checks just the "last modified" date, with the "exclude files which are older than" option. I have also noticed that if a file gets copied (with the standard Windows/Explorer copy), the "last modified" date remains the same, but the "creation" date gets updated to the current date and time.
Would my problem be solved, if there would be an option inside Folder Compare, to check BOTH "last modified" and "creation" date, for the "exclude files which are older than" function?
If yes, is there a hope that you will implement it in a future update?

Thanks.

[Aaron]

Hello,

Filtering and comparing Creation Date is on our Customer Wishlist, but is not currently scheduled development. Does your backup software happen to use or set/unset the Archive bit for files it has transferred, or something else you could filter on?
http://en.wikipedia.org/wiki/Archive_bit

[ThomasB]

SyncBackSE v5 has the following option...

"Reset the archive file attribute on files once they have been copied": When enabled the archive attribute on a file, both on the source/left and destination/right, will be cleared once the file has been copied. Enabling this option will slightly decrease performance. This option is not available when doing Fast Backups. Note that when doing a backup to CD/DVD, the archive attribute is cleared before the file is burned to disk. This means the archive attribute could be cleared even if the file failed to be saved to CD/DVD.

But I'm not sure that this strategy would work in conjunction with Folder Compare, since I do multiple backups of a same source folder, to several different hard disks.
Usually I work like this (I do usually two separate backups daily):
1) Do the backup to an USB hard disk;
2) Binary-compare the source and destination folder (with Folder Compare) after the backup;
3) Do the backup to a second USB hard disk;
4) Binary-compare the source and destination folder (with Folder Compare) after the second backup;
...I have got more USB hard disks, where I do additional backups (now-and-then). And one USB hard disk contains two different backup folders.
Actually a single backup session (on a single hard disk) includes different/separate folders; some get checked with date exclusion set, others with not set.

What do you think? Do you see a solution?
I'm not sure if there's something else which I could filter on.

Currently, if there is no other solution, I think I will do a binary comparison of all files, once a month or more (per hard disk), to verify those files which Beyond Compare didn't "catch". They should be a minority in my case.
Anyway I think it will be very rare that a file will get corrupted during copy on an hard disk. I didn't yet find one! But I prefer to do verifications, just in case.
Actually SyncBack has a function for verifying files afer copy, but there's an issue with that: if the copied file is still in the write cache when it gets verified, then SyncBack would read from the cache, and won't detect a file corruption on disk. As far as I know this is an issue common with other programs too.

[Aaron]

Hello,

Sorry, multiple backups and different locations probably complicate things beyond the point where using the archive bit would be reliable. You'll likely need to perform a full binary compare on everything to catch the older time stamp-ed pairs of files.

As for the caching, yes, that can happen. We have an option to tell windows to not use the cache, but in some instances (such as from a CD-ROM drive), Windows can sometimes still use the cache anyway. Folder Compare -> Session menu -> Session Settings, Handling tab, "Bypass disk cache during binary comparisons."

[ThomasB]

Hello,

How long does data remain in the write cache, usually? A few seconds? Several seconds? Minutes?
How long should I wait before doing a comparison (just after a backup), to be sure that data doesn't get read from the write cache, but gets actually read from the hard disk? (in case I'll decide not to enable the "Bypass disk cache during binary comparisons" option).
I do backups both to USB hard disks, and to an old internal hard disk (but would be interested also about newer SATA disks, for the future). I'm using Windows XP Professional SP3, but am interested to know that in general.
Thanks.

[Aaron]

I do not think it is timing related, but would be under Windows control; possibly until the cache is filled and needed for something else? The cases where it isn't cleared using our option are limited; the CD-Rom issue (when the next CD has the same serial number as the first, even when ejected) is the only instance I know of.

[ThomasB]

Please let me know if I understood correctly...
By using the option "Bypass disk cache during binary comparisons" there shouldn't be those issues with write cache and verification of correct copy (backup); the option "Bypass disk cache during binary comparisons" ensures that data is always read directly from disk, and not from the write cache, except in rare cases (with CD-Roms).

Another question: are there some disadvantages when enabling this option?

Thanks.

[Aaron]

Hello,

Yes, and the disadvantage is the constant use of the disk and the performance hit of never using the cache (which is normally used in day-to-day operations).

[ThomasB]

Hello,

When enabling that option, will it bypass disk cache just for Beyond Compare's operations (inside a single Folder Compare session), or will it affect all Windows disk operations (of any process/program)? (I'm interested about any version of Windows, though I'm currently using XP Professional.)

Is it also possible to have different settings of this option, for each session inside a workspace, without problems?

Thanks.

[ThomasB]

I have got an idea to avoid enabling the "Bypass disk cache..." option.

Just after doing my backup, before doing the comparison/verification with Folder Compare, I could copy a dummy file to the backup hard disk, in order to fill the disk cache completely. This way the other files, copied just previously, should be read directly from the disk, because they are no longer in the cache.

Do you think it is a good idea, it should work?

If yes, how large do you suggest the dummy file to be, in order to fill the disk cache completely? I think different hard disks can have different cache sizes (and I'm not sure if also Windows has its own cache); I would like to have a suggestion that will surely work in all cases. I think I will likely exaggerate the size, just to be sure.

Another question: if I will do this operation, do you think I should delete each time the dummy file from the backup hard disk, before copying it again after each backup, or is it OK to just overwrite it each time? I mean: if I just overwrite the file, the disk cache will be filled with the copied file, all the same? (I believe yes, but would like your confirmation.)

(Please answer also to the previous message.)

Thanks.

[Aaron]

The bypass setting is in our Session Settings dialog, which only pertains to the current session (tab). It would only affect binary comparison operations within that tab. You can set it as a default for future sessions, or save that specific session (like a bookmark) for future use.

[Aaron]

I am not certain of the technical details on how to try and force or trick Windows into clearing the cache. I think the creation and effort to maintain the dummy files may put more strain on your harddrive than the benefit of maybe catching cases where Windows did not properly bypass the cache.

As with any backup, it is always important to have at least 2 copies of your data. That way, if you find a problem with either side down the road, it is likely the other copy would be fine. It would probably be more beneficial to create 3 layers of backup (or add an off-site backup) rather than trying to catch 100.00% of every transfer, if you are extra cautious with this data.