Discrepancy between folders in BC3 and windows properties

[jammer]

Hi,
I use BC3 often to copy large amounts of data and also to compare versions of folders. These can often be many TBs of data and hundreds of thousands of files, many of which are tiny, obscure, hidden, system, etc. Often there are files with long file names, which are buried deep in nested folders, and the total location/name are too long for windows to manage. In those cases I can skip over the offending files in windows, and then come back and do a second pass with BC3 to make up the difference, or sometimes I just use BC3 to do the entire copy from the beginning.

I often run into a situation where BC3 says two folders are the same, but if I right click on both folders in windows and get properties, the file count and file size are listed as different in windows. Sometimes it might be only a few files, or not much file size, sometimes it can be a lot.

I'm working on a folder now which is a good example. It's from a version control software which creates its own internal structure which is completely "insane" in terms of very lengthy, non-sensical, and deeply nested files and folders. It has many tens of thousands of tiny files that are only a few K. I did the copy using BC3, which says both folders are the same, but if I right click on the source folder, windows says it has around 8000 more files, and around 11GB more data than the destination folder.

I need to resolve if I should trust BC3, or windows, and it would help also to understand more about where the discrepancy comes from. I've seen this on many different machines (all windows), many different OS, many different file types, so its not a one time thing, or specific to my current computer, or this specific copy, etc.

Any help much appreciated.

[Aaron]

Hello,

If you enable Suppress Filters in BC3, are any additional files revealed or file counts changed? You may have a filter set that is hiding certain files; we hide protected operating system files by default. You can disable this filter first in the Session Settings, Other Filters tab, and disable Exclude protected operating system files. Would you have any other filters configured?

Session Settings can be updated for a specific view or as global defaults for any future comparisons.

[jammer]

Hi Aaron,
thx for the response. I am using BC3 in default config with no filters, aside from the one you mentioned above with the default setting on protected OS files. I tried unchecking that option and comparing the two folders, BC3 still shows them as same, even though windows show the top level folder properties as having significant amount of difference in file count and file size.

A question:
is there somewhere in BC3 that will give me a total file count and total file size, at the top level, for the data being compared? I would like to check the BC3 count against the windows count.

I believe what is happening is that windows skips any files where the destination file name path is too long (260 characters). BC3 does not seem concerned with this issue, and will happily copy over these "problematic" files, and I believe, makes the source and destination the same. However, the "suspect" files which have been "forced" into destination locations which exceed windows character limit are now outcasts within windows, and when you right click on the parent directories to compare file and folder size and count, in both the source and destination, it ignores those files. That is just a theory. I suppose I could keep renaming top level folders shorter and shorter and see if ultimately the total file counts in windows properties ever converge to exactly the same on the source and destination.

My concern is that I am copying data forward, over and over, with many files that exceed the windows path limit, and I'm trying to understand both how BC3 is handling this situation, and also windows, and what the implications are, etc. I don't know how it might affect application handling of those files, which windows does not seem to think exist in the same way that regular files do (within the path limit).

I could have a situation where even my source folder has been copied forward a few times, and has files which exceed the path limit, and this issue is then compounded with every copy instance going forward as the folders may be nested more and more deeply, as any copy is typically going to be extending the total path length by at least a few, if not many, characters.

I appreciate that how windows deals with this problem is outside of your jurisdiction, but any info or thoughts you have on how BC3 handles files which exceed the path limit is greatly appreciated.
thx
Ben

[Aaron]

Thanks for the clarification and your own troubleshooting. Usually, when we see this complaint it is due to filtering and "less" files are showing up after transferring in BC compared to Explorer's count. But you are correct: BC4 is capable of copying beyond the Windows path limit, which could result in the opposite being true.

You can get a full count of what BC4 sees by issuing an Expand All command (Toolbar or Edit menu) to build all folders and subfolders, then use the Session menu -> Folder Compare Info to see a file count for each status. You'll have to do a little math to add together a couple of the numbers. You could also perform a Select All after an Expand all, and look at the bottom status bar below each pane/side, to see the Selected count.

The Expand All is important in both steps, as otherwise you might have unbuilt top level orphans, or the reporting methods will only display information for exposed files.

[jammer]

With further testing it seems like the difference in windows folder properties are indeed due to exceeding the max path limit. If I make sure the source and destination folders are the same number of characters, then windows properties will show both folders with the exact same file count, folder count, and total size.
However any variation, even a single character length, and those values will start to diverge. The greater the difference in character length, the greater the difference in file and folder count, and total volume.
Of course this only applies to folders where some of the contents exceed the max path limit.
Thx for the pointers on getting file counts in BC3, that is helpful.
Seems like it's relatively clear how BC3 is handling these files, but will take more research to understand the implications in windows and other applications.

[Aaron]

Correct. BC3 handles these lengths since we also interact with FTP servers and other remote connections which may exceed path lengths as well, but it is generally recommended to use BC to help resolve these lengths so that the native OS can also support/see the paths.