Page 1 of 3 123 LastLast
Results 1 to 10 of 29
  1. #1
    Join Date
    Apr 2011
    Posts
    1

    Default Suggestion: Queued transfers

    I started using beyond compare by more than 6 years, I was looking for something better and I couldn't find yet.

    I'm also a developer, so let me give a suggestion to this really usefull tool: there is a feature lacking to this software. When I use BC to sync a lot amount of data (i.e. backup, but not only) I start to transfer files and I cannot queue the next data tansfer after the first one has finished. Some software (i.e. teracopy) allow to enqueue file transfer, and this means a really good gap, since hard drives are THE bottleneck in any computer.

    Another think is that data transfer between two USB are slow. 10 MB/s on my PC. If the software would copy all data to the equivalent of .NET Framework MemoryStream, the speed would raise to 30 MB/s.

  2. #2
    Join Date
    Oct 2007
    Location
    Madison, WI
    Posts
    4,610

    Default

    gik25,

    Thanks for the suggestions.

    Beyond Compare 3 does allow you to queue file operations, but they run in parallel. An option to queue file operations that run in sequentially is on our wish list for a future version.

    BC 3.2.4 uses CopyFileEx to copy files, so performance should be similar to Windows Explorer when copying files. I don't think we have any plans to change the copy method in the near future, but we'll keep it in mind.
    Chris K Scooter Software

  3. #3
    Join Date
    Oct 2007
    Location
    Pennsylvania
    Posts
    1,772

    Default CopyFileEx uses Buffered I/O

    Quote Originally Posted by Chris View Post
    BC 3.2.4 uses CopyFileEx to copy files...I don't think we have any plans to change the copy method in the near future...
    CopyFileEx is a Buffered I/O mode (Microsofts default file transfer mode whether using Windows Explorer drag and drop, the batch Copy command, XCopy, or Robocopy.) Buffered I/O speeds up access to files that get accessed frequently. It buffers hard disk reads and writes in the file system cache to speed up future reads and writes to the same files. Copying a large file using Buffered I/O, however, effectively flushes the file system cache with the contents of the large file. This affects performance in two ways:

    1) Once the file cache fills up, the constant flush and re-cache of data from the file being copied adds overhead to the system.

    2) Other programs that rely on the file system cache to improve performance (such as SQL server) also suffer, because the file system cache is not available when it is constantly being flushed/overwritten by the large file copy.

    For huge files, Microsoft now recommends using Unbuffered I/O (a raw file copy). The result for huge files is a faster copy as well as a smaller impact on overall system performance.

    In Windows 7 and Windows Server 2008 R2, Microsoft added a new /J switch to the XCopy command to enable copies using Unbuffered I/O. If you seriously want to prevent system hangs with large file transfers, then you seriously need to consider unbuffered I/O.

    Here's a quote from the TechNet documentation on Xcopy

    Quote Originally Posted by Microsoft TechNet
    /j
    Copies files without buffering. Recommended for very large files. This parameter was added introduced in Windows Server® 2008 R2.
    BC v4.0.7 build 19761
    ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯

  4. #4
    Join Date
    Oct 2007
    Location
    Pennsylvania
    Posts
    1,772

    Default

    I am raising this issue once again. I work in an environment that manages terabytes of data. I often need to copy cross-network copies of SQL server database backups that are hundreds of gigabytes in size. The problem is that huge file copies using buffered IO in a SQL server environment can actually bring down a production system! The buffered file copies steal system resources away from SQL server and cause a large number of memory hard faults/sec. SQL server freezes up and the OS becomes unresponsive. Sometimes cycling SQL Server is necessary, sometimes a system reboot!

    I would love to use BC3 to perform my folder syncs and huge file copies... but since BC3 uses the same buffered IO as Windows explorer, it can lock up the production environments I am working with and impact the analysis being done by our clients on these servers! Not good!

    Ideally, BC3 should have an option to use unbuffered IO for large file copies. Until this is natively supported by BC3, I need to know how to setup an "Open With" configuration that will call the Microsoft ESEUTIL executable to perform unbuffered file copies. The command line I need to have defined is:

    eseutil /Y "\\share\source_path\filename.ext" /D"\\share\dest_path\filename.ext"

    The solution should allow me to right click on a file or set of files on the either side of the compare and perform an ESEUTIL copy to the other side. Please help!
    BC v4.0.7 build 19761
    ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯

  5. #5
    Join Date
    Oct 2007
    Location
    Madison, WI
    Posts
    11,623

    Default

    Thanks, Michael. I'll add your notes to our entry on the subject. This would be part of a larger task, so it isn't likely we'll be able to implement this soon, but it is something we are looking into.
    Aaron P Scooter Software

  6. #6
    Join Date
    Oct 2007
    Location
    Pennsylvania
    Posts
    1,772

    Default

    So you're saying that it is not possible to integrate a 3rd party utility to perform the copies via the "Open With" feature? i.e. there is no way to pass both filenames of an aligned pair of files to the command line in an "Open With" operation?
    BC v4.0.7 build 19761
    ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯

  7. #7
    Join Date
    Oct 2007
    Location
    Madison, WI
    Posts
    11,623

    Default

    Oh, sorry. I was referring to 'implementing our own copy routine' to help handle this.

    An Open With can pass in both the left and right (first or second) by appending a 1 or 2 to the end of the variable:
    %F1 %F2

    If you click the Help button '?' it should take you directly to the Open With help article with more details on the syntax. How does this work for you?
    Aaron P Scooter Software

  8. #8
    Join Date
    Oct 2007
    Location
    Pennsylvania
    Posts
    1,772

    Default

    I'm having difficulty understanding how to define the "Open With". What I need to specify as parameters are:

    The full path and filename of the current side.
    The path of the opposite side followed by the filename only of the current side.

    I don't see a way to guarantee that the path variable will always refer to the opposite side. If appending 1 and 2 refer to the left and right sides instead of the current and opposite sides, then I definitely don't want to use them. I tried:

    eseutil.exe /Y "%f" /D"%f"

    But that didn't work. Although I do not want to define two separate open with commands to support unbuffered copies in both directions (left to right and right to left), I tried it anyway:

    eseutil.exe /Y "%f1" /D"%p2\%n1"

    That didn't work either. Note that the eseutil utility only works on files, but I am trying to copy from one side to the other where the file may not exist on the other side. I can't figure it out.
    BC v4.0.7 build 19761
    ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯

  9. #9
    Join Date
    Oct 2007
    Location
    Madison, WI
    Posts
    11,623

    Default

    Hello,

    Our Open With only supports the command as F1 left and F2 right (assuming selection on both sides). I'll add your notes to detect which side was selected to our Customer Wishlist.

    If I'm reading your command correctly, you could create two commands (similar to our Copy to Right and Copy to Left commands). You would then just need to pick the direction as part of the custom open with command, correct?
    Aaron P Scooter Software

  10. #10
    Join Date
    Oct 2007
    Location
    Madison, WI
    Posts
    2,507

    Default

    Michael, the short answer is that no, this isn't possible right now. Open With doesn't have a way to pass a filename/path for a non-existent file. If the space across from a selected file is blank the %f2 paths will always be empty.
    Zoë P Scooter Software

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •