Mirroring script deletes file with no reason

[myriaxis]

Hi,
I'm new to the forum.
After reading on this forum, I created a simple mirroring script to mirror a set of shared folders from a Windows Server 2012 R2 Essentials to a remote PC connected via VPN.
On the remote PC, I automatically run this script every night using Task Scheduler:
log verbose append:"C:\Users\***\Desktop\Mirror Log.txt"
load "Mirror"
option confirm:yes-to-all
sync mirror:left->right
The left side is the selected shared folders on the Windows Server 2012 R2 Essentials
The right side is the mirror location on the remote PC.

I've been using this script with BC4.1.3 for the past two weeks without a problem.
Yesterday, the script completely erased one of the shared folders on the mirror side without reason (there were no change in that shared folder).
I don't know if it's related, but I upgraded to BC4.1.4 before the script ran yesterday.
Is there something to be done to the script after upgrading?
It's also possible the vpn link went down during the comparison process.
Is it possible BC4 decided to delete a folder because it couldn't see it due to the vpn not connected?

Please help me explain this behavior.

[Aaron]

Yes, the Sync Mirror will delete files on the right if they do not exist on the left. How would a failure to load appear on your VPN? Could this give us a folder structure with folders missing? Or would the folder appear but have an error, as if Access was Denied? This would depend a bit on the remote device and how it presents the error.

The difference between a Sync Mirror and Update:
Sync Update will copy any Newer or Orphan files from Source to Destination.
Sync Mirror will copy any Newer, Older, Different and Orphan Files from Source to Destination. Will also delete any Orphan files in the Destination.

[myriaxis]

Thanks for the reply.

I need Sync Mirror because I want an exact copy of the shared folders on the remote PC.

While preparing my response to your reply, I checked the script log again and found this (I missed it in the first pass):
2016-03-07 04:53:18 Unable to load \\Q\Shared Folders\Software: The network path was not found

This is the folder that was completely removed from the mirror.
Should I understand that BC considered this folder as "removed" because it couldn't find it due to a network error?
If this is the case, how can I prevent BC from acting on this folder, or all folders when this kind of error is found?

Thanks

[Aaron]

Hello, BC 4.1.4 and earlier have a bug which might trigger this behavior if the error occurs mid-load as a timing error. Usually, if a folder fails to load it would be marked as an error, it wouldn't result in deleting files in the destination. We have a fix included with the next release which will help, but may not solve for all scenarios. Are you able to reproduce this error?

[myriaxis]

Because of the quantity of data (over 4 TB to compare), it's kind of impractical to reproduce intentionally.

So far, I've seen it only once during the comparison load. I've seen it more often at the beginning of the comparison load. When that happens, BC simply stops. The comparison load takes about 3 hours (over 4 TB to compare). Therefore, loosing the vpn in the middle of the process is bound to happen again. Is there a way, in the script or BC settings, to indicate what to do when the network path is not found?

[Aaron]

The intended behavior is to log the error in script and not perform sync actions on the folder with the error, so we wouldn't delete on the other side. BC 4 has detection code to try and find various issues that can pop up, and there is expanded detection code included in the next release (4.1.5). Let us know if this occurs again, or you find a reliable means of reproducing the error after the initial load.

[myriaxis]

As I mentioned earlier, I can't afford to loose 0.5 TB when the problem occurs. It's too much data to transfer via VPN. I'd rather wait for your next release. On the other hand, if you have a beta version with this problem fixed, I could give it a try if you want.

[Aaron]

Do you still have access to the full log when this issue occurred? We may need to see this issue from the start to better know if/how we've tackled it. You can post here or you can email us at [email protected] with a link back to this forum thread if you need to submit it privately.

[myriaxis]

Hi, the original log is pretty long. I edited the log to remove repetitive actions like successful copy and successful delete.
You'll see three different behaviors.
March 3rd = unable to load comparison from the start resulting in a script failure.
March 5th = successful completion of the synchronize process.
March 7th = unwanted delete of a shared folder because BC couldn't load it, but was able to access the other shared folders.

2016-03-02 02:00:02 >> log verbose append:"C:\Users\*\Desktop\Mirror Log.txt"
2016-03-02 02:00:02 >> load "Mirror"
2016-03-02 02:00:09 Unable to load \\Q\Shared Folders\: The network path was not found
2016-03-02 02:00:09 Load comparison: <-> E:\
2016-03-02 02:00:09 Fatal Scripting Error: Unable to load base folder
2016-03-02 02:00:09 Script completed in 7,46 seconds

2016-03-05 02:00:02 >> log verbose append:"C:\Users\*\Desktop\Mirror Log.txt"
2016-03-05 02:00:02 >> load "Mirror"
2016-03-05 04:40:45 Load comparison: \\Q\Shared Folders <-> E:\
2016-03-05 04:40:46 >> option confirm:yes-to-all
2016-03-05 04:40:46 >> sync mirror:left->right
2016-03-05 04:47:13 Copied \\Q\Shared Folders\Pictures\a.pdf to E:\Pictures
2016-03-05 04:47:15 Copied \\Q\Shared Folders\Pictures\b.pdf to E:\Pictures
2016-03-05 04:47:16 Copied \\Q\Shared Folders\Pictures\c.jpg to E:\Pictures
2016-03-05 04:47:16 Copied \\Q\Shared Folders\Pictures\d.pdf to E:\Pictures
2016-03-05 04:47:17 Copied \\Q\Shared Folders\Pictures\e.jpg to E:\Pictures
2016-03-05 04:47:17 more files copied
2016-03-05 04:49:08 Deleted E:\Pictures\f.bmp
2016-03-05 04:49:08 Deleted E:\Pictures\g.jpg
2016-03-05 04:49:08 Deleted E:\Pictures\h.jpg
2016-03-05 04:49:08 Deleted E:\Pictures\i.jpg
2016-03-05 04:49:08 Deleted E:\Pictures\j.jpg
2016-03-05 04:49:08 more files deleted
2016-03-05 04:49:10 Copied \\Q\Shared Folders\Pictures\k.jpg to E:\Pictures
2016-03-05 04:49:10 Copied \\Q\Shared Folders\Pictures\l.jpg to E:\Pictures
2016-03-05 04:49:10 more files copied
2016-03-05 06:35:09 Copied \\Q\Shared Folders\Videos\m.mp4 to E:\Videos
2016-03-05 06:35:09 Successfully synchronized 622 items. Completed in 1 hours, 54 minutes.
2016-03-05 06:35:10 Script completed in 4 hours, 35 minutes

2016-03-07 02:00:00 >> log verbose append:"C:\Users\*\Desktop\Mirror Log.txt"
2016-03-07 02:00:01 >> load "Mirror"
2016-03-07 04:53:18 Unable to load \\Q\Shared Folders\Software: The network path was not found
2016-03-07 06:07:22 Load comparison: \\Q\Shared Folders <-> E:\
2016-03-07 06:07:31 >> option confirm:yes-to-all
2016-03-07 06:07:36 >> sync mirror:left->right
2016-03-07 07:47:24 Deleted E:\Software\a.ini
2016-03-07 07:47:28 Deleted E:\Software\b.txt
2016-03-07 07:47:29 all E:\Software\ files and folders deleted
2016-03-07 08:40:55 Copied \\Q\Shared Folders\Games\a.exe to E:\Games
2016-03-07 08:48:44 more files copied
2016-03-07 10:48:01 Successfully synchronized 243 641 items. Completed in 4 hours, 40 minutes.
2016-03-07 10:48:06 Script completed in 8 hours, 48 minutes

[Aaron]

Hello,

Thanks. The log is helpful, and this should be fixed in the next release of BC4.

[Chris]

Beyond Compare 4.1.5 is now available with a fix for this bug.

[myriaxis]

Hi, I installed version 4.1.5 last week. I think the fix is working. The problem occurred this weekend and this is the log:

2016-03-27 02:00:33 >> log verbose append:"C:\Users\***\Desktop\Mirror Log.txt"
2016-03-27 02:00:34 >> load "Mirror"
2016-03-27 03:26:27 Unable to load \\Q\Shared Folders\Videos: The network path was not found
2016-03-27 03:28:14 Load comparison: \\Q\Shared Folders <-> E:\
2016-03-27 03:28:14 >> option confirm:yes-to-all
2016-03-27 03:28:14 >> sync mirror:left->right
2016-03-27 03:28:56 Unable to load \\Q\Shared Folders\Videos: The network path was not found
2016-03-27 03:32:00 Script completed in 1 hours, 31 minutes

From this log, can you confirm the problem is fixed?

[Aaron]

The fix submitted helps prevent the deletes from happening if the targets fail to load. The load error "The network path was not found" is an issue with the network locations. What error are you running into now? Is it that some directories fail to load and the comparison continues after excluding them (expected), or that the Mirror is still deleting these folders in the destination (error)?

Note: if a base folder fails to load, this is a fatal error and the entire script would exit.

[myriaxis]

Hi. I'm sorry about my last reply. It wasn't very clear.

As I mentioned earlier, I use Beyond Compare to synchronize a set of shared folders from a Windows Server Essentials to a remote PC thru a VPN across the Internet. The VPN is not stable. It disconnects and reconnects randomly. When that happens during the synchronization process, I get The network path was not found errors for a folder or the entire set.

The excerpt from my log shows that on March 27th one folder was unable to load during the comparison and BC 4.1.5 excluded that folder from the synchronization process. As a result, the corresponding destination folder wasn't synchronized nor deleted on the remote PC. This is the expected behavior. The fix provided by version 4.1.5 is working well.

Thank you very much.