Nightly Beyond Compare Schedule

[Guest]

We're using a backup program to backup our local data and I have set the data to split into 512MB chunks. On our remote server, I have a session within BC which has a comparision between the remote server (set as local) and the local server (set as FTP) where the data is backed-up.

On the remote server, I have set-up a nightly schedule which runs a BC script to load the session and mirror it. It begins at 00:00 and is set to end after 8 hours if the comparision hasn't finished (to conserve our bandwidth for when we need it).

The problem we're having is that the first 512MB block will transfer, and then the connection just times out. BC is set to reconnect on disconnect, so it does this then starts transferring the same file again! Of course, Windows then ends the schedule and the incomplete file is left in the Windows "temp" folder. This is despite the fact that the file has already downloaded successfully. As a result, we are ending up with no files being backed-up.

I have set-up a verbose log file, and here are some snippets:

...
09/06/2006 00:00:34 Sent> RETR [file1].zip
09/06/2006 00:00:34 Recv> 150 Opening BINARY mode data connection for [file1].zip (537651427 Bytes).
09/06/2006 04:51:35 Stat> 537651427 bytes transferred. (30.0 KB/s) (04:51:01)
09/06/2006 04:53:35 Stat> Timeout (120s).
09/06/2006 04:53:35 LocalHelp: 3
09/06/2006 04:53:35 Stat> Client closed the connection.
09/06/2006 04:53:35 Stat> Resolving host name "[host]"
09/06/2006 04:53:35 Stat> Connecting to [IP] Port: 21
...
09/06/2006 04:53:35 Sent> RETR [file1].zip
09/06/2006 04:53:35 Recv> 150 Opening BINARY mode data connection for [file1].zip (537651427 Bytes).
[END OF LOG]

Thanks in advance...

[Chris]

Re: Nightly Beyond Compare Schedule

I discussed this with our FTP developer.

This is a bug, but we're still working to track it down. When Beyond Compare downloads files using FTP, it saves them to a temp file. Once the download is complete, the temp file is moved to the correct location. When BC hits this bug, it finished downloading the file, but somehow doesn't receive notification from the FTP server that the transfer is done.

This tends to happen more often with larger files, so it might help to reduce the size of the data chunks. It might also help to toggle Mode Z compresson on or off if your FTP server supports it (it is set in the FTP section of Tools|Options).

[Guest]

Re: Nightly Beyond Compare Schedule

I have now turned on MODE Z compression; although the zip files are already quite well compressed.

Do you have a suggestion on a good chunk size I could try? I have set-up a temporary filter to prevent files sized over 100MB from transferring, meaning only the end chunks and some incrementals will upload. I will see the result of this tonight.

Thanks.

[Guest]

Re: Nightly Beyond Compare Schedule

OK, last night it got stuck on a file of size 67613527 bytes, so 64.5MB. It just went around and around, constantly re-downloading the file until the schedule was ended at 08:00.

Prior to this, it managed to download one file successfully first time, which was of a size 657735 bytes (642KB).

[Chris]

Re: Nightly Beyond Compare Schedule

Hello Atelier,

I'm not sure what a good chunk size to try is. The last user that I remember reporting this issue was having problems with files around 100 MB in size.

We're trying to get this sorted out, but we haven't been able to get the problem to show up on any of our test systems yet.

[MarcelFortin]

Re: Nightly Beyond Compare Schedule

Hi Atelier, out of curiosity sake, what is the name and version of the ftp software you use? I currently use Serv-U ftp software from Rhino Software and encounter timeout issues quite frequently. Just curious...

[Guest]

Re: Nightly Beyond Compare Schedule

Hi MarcelFortin,

The FTP server we use is indeed Rhinosoft Serv-U, version 6.0.

Hopefully this will help the developers in narrowing down the problem.

I think it would be useful if anyone else encountering the same problem, with or without Serv-U as the FTP server, could post here.

[Chris]

Re: Nightly Beyond Compare Schedule

Marcel and Atelier,

Thank you for posting your FTP server type. We still haven't exactly tracked down this bug, but if we can narrow it down to specific FTP servers it will help.

For anyone else experiencing this bug, please let us know what server software you're using.

[Guest]

Re: Nightly Beyond Compare Schedule

Hi. I believe i am seeing this using v2.41 and IIS5. Script is still running, but it has downloaded the same file a number of times. The file is 400mb