Suggestion: Queued transfers

gik25,

Thanks for the suggestions.

Beyond Compare 3 does allow you to queue file operations, but they run in parallel. An option to queue file operations that run in sequentially is on our wish list for a future version.

BC 3.2.4 uses CopyFileEx to copy files, so performance should be similar to Windows Explorer when copying files. I don't think we have any plans to change the copy method in the near future, but we'll keep it in mind.

CopyFileEx uses Buffered I/O

CopyFileEx is a Buffered I/O mode (Microsofts default file transfer mode whether using Windows Explorer drag and drop, the batch Copy command, XCopy, or Robocopy.) Buffered I/O speeds up access to files that get accessed frequently. It buffers hard disk reads and writes in the file system cache to speed up future reads and writes to the same files. Copying a large file using Buffered I/O, however, effectively flushes the file system cache with the contents of the large file. This affects performance in two ways:

1) Once the file cache fills up, the constant flush and re-cache of data from the file being copied adds overhead to the system.

2) Other programs that rely on the file system cache to improve performance (such as SQL server) also suffer, because the file system cache is not available when it is constantly being flushed/overwritten by the large file copy.

For huge files, Microsoft now recommends using Unbuffered I/O (a raw file copy). The result for huge files is a faster copy as well as a smaller impact on overall system performance.

In Windows 7 and Windows Server 2008 R2, Microsoft added a new /J switch to the XCopy command to enable copies using Unbuffered I/O. If you seriously want to prevent system hangs with large file transfers, then you seriously need to consider unbuffered I/O.

Here's a quote from the TechNet documentation on Xcopy

I am raising this issue once again. I work in an environment that manages terabytes of data. I often need to copy cross-network copies of SQL server database backups that are hundreds of gigabytes in size. The problem is that huge file copies using buffered IO in a SQL server environment can actually bring down a production system! The buffered file copies steal system resources away from SQL server and cause a large number of memory hard faults/sec. SQL server freezes up and the OS becomes unresponsive. Sometimes cycling SQL Server is necessary, sometimes a system reboot!

I would love to use BC3 to perform my folder syncs and huge file copies... but since BC3 uses the same buffered IO as Windows explorer, it can lock up the production environments I am working with and impact the analysis being done by our clients on these servers! Not good!

Ideally, BC3 should have an option to use unbuffered IO for large file copies. Until this is natively supported by BC3, I need to know how to setup an "Open With" configuration that will call the Microsoft ESEUTIL executable to perform unbuffered file copies. The command line I need to have defined is:

eseutil /Y "\\share\source_path\filename.ext" /D"\\share\dest_path\filename.ext"

The solution should allow me to right click on a file or set of files on the either side of the compare and perform an ESEUTIL copy to the other side. Please help!

Thanks, Michael. I'll add your notes to our entry on the subject. This would be part of a larger task, so it isn't likely we'll be able to implement this soon, but it is something we are looking into.

So you're saying that it is not possible to integrate a 3rd party utility to perform the copies via the "Open With" feature? i.e. there is no way to pass both filenames of an aligned pair of files to the command line in an "Open With" operation?

Oh, sorry. I was referring to 'implementing our own copy routine' to help handle this.

An Open With can pass in both the left and right (first or second) by appending a 1 or 2 to the end of the variable:
%F1 %F2

If you click the Help button '?' it should take you directly to the Open With help article with more details on the syntax. How does this work for you?

I'm having difficulty understanding how to define the "Open With". What I need to specify as parameters are:

The full path and filename of the current side.
The path of the opposite side followed by the filename only of the current side.

I don't see a way to guarantee that the path variable will always refer to the opposite side. If appending 1 and 2 refer to the left and right sides instead of the current and opposite sides, then I definitely don't want to use them. I tried:

eseutil.exe /Y "%f" /D"%f"

But that didn't work. Although I do not want to define two separate open with commands to support unbuffered copies in both directions (left to right and right to left), I tried it anyway:

eseutil.exe /Y "%f1" /D"%p2\%n1"

That didn't work either. Note that the eseutil utility only works on files, but I am trying to copy from one side to the other where the file may not exist on the other side. I can't figure it out.

Hello,

Our Open With only supports the command as F1 left and F2 right (assuming selection on both sides). I'll add your notes to detect which side was selected to our Customer Wishlist.

If I'm reading your command correctly, you could create two commands (similar to our Copy to Right and Copy to Left commands). You would then just need to pick the direction as part of the custom open with command, correct?

Michael, the short answer is that no, this isn't possible right now. Open With doesn't have a way to pass a filename/path for a non-existent file. If the space across from a selected file is blank the %f2 paths will always be empty.

Would you please consider fast-tracking this as an enhancement. It is bad enough that BC3 cannot perform its own unbuffered copies... but not being able to define an "Open With" to do it with a 3rd party utility!?!

It shouldn't be that difficult to pass the path to the other side...especially when the path exists! In my example, I am always copying to an existing folder. Please... if I specify the path variable (which doesn't include a filename anyway) I expect it to return the path of the aligned folder whether or not an aligned file exists.

Hi Michael,

We appreciate the feedback and an unbuffered i/o copy is definitely something we are looking into, but we've got a lot on our plate right now, so it probably won't be a quick fix (for unbuffered i/o copy or the open with enhancement).

Has any progress been made on this request? The ability to leverage alternative copy methods from the BC UI would be extremely helpful. Eventual inclusion of alternate methods natively in BC could come later as time and business needs dictate. In the meantime we really need a way to optimize copying methods from the BC UI.

Please take action on these (and related) requests. For my part I woudl like to be able to call TeraCopy for when I do the kinds of copying that greatly benefit from how it operates. Leveraging the buffered IO methods would also be useful but I would never make it the default. It often has unpredicatable performance results (slower) as often as when it is faster. BC itself just needs flexibility.

Adding alternate copy methods is still on our wish list for a future release, but it doesn't have a release date set right now.

Once again, the request for an alternate copy method is secondary. All we really need is the ability to pass the path of the opposite side of an orphan file in an Open With definition! Please, this would be a relatively small effort from a developer perspective adding great potential for your user base.