Parallel processing to reduce elapsed time

Collapse
X
Collapse
 
  • Page of 1
  • Time
  • Show
Clear All
new posts
Previous template Next
  • maiera
    Visitor
    • Jun 2004
    • 3
    #1

    Parallel processing to reduce elapsed time

    Hi,
    I am using BC2 to synchronize a large number (10000+) of small files (few KB) to a SMB mounted resource that is far away as far as the network delay is concerned (ping times around 150ms).

    As you can imagine, this takes several hours, and is a great annoyance.

    The network and file system experts owning that resource tell me that parallel processing would shorten the elapsed time significantly, i.e. the throughput is not limited by network or file system bandwidth, but rather by the response time of the many file creates.

    I think this situation is not unusual, and I like BC2 enough to suggest a parallel processing option. This should be configurable, in order to have some control over the network utilization and the CPU usage, e.g. by defining the degree of parallelism, or the max CPU percentage. A good model for this has been implemented in OfflineExplorer.

    Please let me know what the chances are to get this into BC2.
    Tags: None
    • maiera
      Visitor
      • Jun 2004
      • 3
      #2
      Parallel processing to reduce elapsed time

      BTW, any other ideas on how to reduce the elapsed time for my situation, are highly welcome, of course!

        Comment

        • Chris
          Team Scooter
          • Oct 2007
          • 5538
          #3
          Re: Parallel processing to reduce elapsed time

          Hi Andy,

          I'll add parallel processing to our wish list. It sounds like a good feature to have.

          As a workaround, BC does allow you to run multiple instances, so you can try running several instances of BC on different parts of the files you're updating. You can probably break it up by path or using filters.

          You could even automate this with a script and a batch file. If the script takes the path as a command line parameter, you could run the script multiple times from a batch file and make sure each call doesn't wait for the previous one to finish.

          Please let me know if you need help with the script/batch file.
          Chris K Scooter Software

            Comment

            • krick
              Visitor
              • Sep 2006
              • 6
              #4
              Re: Parallel processing to reduce elapsed time

              BTW, any other ideas on how to reduce the elapsed time for my situation, are highly welcome, of course!

              If your process doesn't require user intervention, you might want to try synchronizing with a command line utility like XXCopy instead... http://www.xxcopy.com/

                Comment

                Previous template Next
                Working...