Multiple Base Folders

[davepacey]

Is there a way to have multiple base folders ? if not are there any plans to do this ?
At the moment, I have one session setup for each base folder, and run them one at a time, I dare say it is possible with scripts, but I have not yet explored that avenue at all.

[Pete]

I would also really like this feature in the GUI.

[armsys]

I've never encountered the needs for comparing more than 2 base folders. Definitely, it can be easily achieved by a simple script especially if you've already save the session defintion. Just look up the Script Reference in the online help.

[Craig]

It's not possible right now, but we do have plans to do it later.

[davepacey]

Good, that will be very welcome Craig. I would certainly make good use of it.

[jdmarch]

Dave, could you give an example of how you would use this?

Thanks!

I would greatly appreciate this feature too.

[Pete]

Dave, could you give an example of how you would use this?


While you're waiting for Dave to give you an example, I'll give you one. :)

I have two main folder structures that I backup everyday. The root folders are C:\Data and N:\. (Drive N: is a BestCrypt encrypted disk.)

I backup these two folder structures like this:

C:\Data --> D:\Data
N:\ --> O:\

Everyday, I have to load the session for the C: drive, visually compare the differences, and then sync. When that's done, I have to do the same thing for the N: drive.

Sometimes it can take a long time to sync the files on my C: drive, so I have to check my computer periodically to know when it finishes so that I can then start the compare & sync process for the N: drive.

Ideally, BC would allow me to visually compare the folders on the C:\Data path and on the N:\ path at the same time and then begin the sync operation for both of them at the same time. This way, I could compare everything before bedtime, for example, and then start the sync process for everything at once. The next morning, the sync process would be completed.

This cannot be done with a script because I need to visually compare the contents of each path before I sync them.

On the one hand, I understand that this could add more complexity to the GUI and the underlying algorithms. On the other hand, however, it seems like there "must" be an elegant way to add this functionality.

I think that it would "wreck" the current, easy-to-understand GUI if they changed it to allow us to specify multiple source & target paths in the same session.

However, I think that there could be ways to achieve this by letting us compare & sync multiple BC sessions at once. In my case, during the compare operation, the files in C:\Data would be listed, immediately followed by the files in N:\. When I start the sync operation, then C:\Data would be synced followed by N:\ without any interaction on my part.

Some possible ideas to achieve this:

1) Have a new BC session "group-type session" that merely contains a list of real sessions that you want to treat as a single session.

2) Or, add a new dropdown box to a session's properties that lets you specify a second session to join with this session. In the second session, you could specify a third session to join with the second session, etc.

[davepacey]

I store all my music (mp3) files on drive D: I store all my other data on my C: drive in several different folders, I do not use the default My Documents folder for reasons I shall keep to myself. I wish to just set up a single session, and be able to add my folder off drive D and the several folders on drive C: and have them backed up to my DVD drive F: in one go. I would rather not use scripts, as I prefer the GUI interface of BC.
At the moment I have to run each of 7 sessions seperately, which is rather painful. I previously used Synchroniser Pro, which is rather basic in functionality, but you can add as many folders as you wish, and it works as I like it.

I know I can use multiple base folders on one drive if I select the route of C: and then exclude all folders not wanted for backup, but this, again, is painful.

[Pete]

DaveP wrote:
I know I can use multiple base folders on one drive if I select the route of C: and then exclude all folders not wanted for backup, but this, again, is painful.


Not only is it painful, but it's impossible for people who use more than the C: drive (such as C: and N: in my case and C: and D: in your case). Now if BC let us choose "My Computer" as the Left base folder (which it doesn't), that would at least make it possible to include multiple Left base drives and folders (by including or excluding the appropriate drives and folders).

However, no matter how we select the Left base drives and folders, there are two potential show-stoppers if they try to cram everything into a single BC session:

1) What about the Right base folders? In DaveP's case, he's only using a single target folder (F:\), so that is the simplest possible case. Other people such as myself, on the other hand, use multiple Left folders AND multiple Right folders. How would we specify this using a single BC session? Would we separate the folders with a semi-colon like this:

(Left) C:\Data;N:\
(Right) D:\Data;O:\

and then the order that we specify the folders in the left and right fields would be important so that C:\Data is synced with D:\Data and N:\ is synced with O:\. While this is certainly possible, it doesn't seem like a very elegant solution for BC's GUI.

2) The other thing to think about is FTP sites. There are probably people besides myself who synchronize multiple FTP sites to/from hard drives. It could be very difficult and/or messy to specify all of the relevant information (userids, passwords, etc.) in a single session.

It's for all of these reasons that I suggested the multiple-session approach instead of the single-session approach. However, it's obvious to me now that there are pros and cons to either method.

[jdmarch]

I think that both Dave and Pete can get pretty close to what you want with BC's current capabilities:

1. BC already permits multiple sessions to run simultaneously. (The setting is in Tools / Options / Startup / Allow Multiple Instances.)

If it seems too laborious to start up 2 or 7 different sessions manually, write a little batch file, which would look something like this:
BC2 "My session 1"
BC2 "My session 2"
etc.
Make a shortcut to your batchfile, and click it whenever you want to begin your backup procedure.

2. To Dave: it is actually very simple to exclude the majority of folders in C:\ from a session. In theory, you
just select the 7 folders that you want (clicking on each while holding down the Ctrl key), then chose Edit / Invert Selection, then Actions / Exclude, then save your session. Voila, you have your desired selection of 7 folders in C:, in one session.

BUT BEFORE YOU TRY THIS: right now there is a BUG in BC (at least in build 215) such that the exact sequence of commands which I just listed, does not work, because it actually excludes all files and folders on C: The same thing happens if you Edit / Select ALL, then deselect your desired 7 folders, then Exclude.

I'm sure that this bug will be fixed soon. In the meantime, you can select all files and folders by selecting the first folder, then pressing Shift+End. Then hold the control key and click your 7 folders to deselect them. Then Exclude. Then save your session. Still short and sweet and gets you where you need to go.

[jdmarch]

I would argue against the feature as requested. However I do support (though not urgently) a similar request: for a "virtual folder", which could be defined as containing any arbitrary (and user-editable) list of files and/or folders. I think this would do what you want, and also cover the more general case of someone who had copied an arbitrary group of files from around their hard disk, into a single folder or archive or removable disk, and wanted to compare all of these files to the copies, together.

[Pete]

Jonathan wrote:

...a "virtual folder", which could be defined as containing any arbitrary (and user-editable) list of files and/or folders.


I'm not clear on where in the BC GUI you would specify a "virtual folder". Could only the source base folder be "virtual" or both the source and target base folders? Are you saying that there would be a single target base folder, or multiple target base folders that match with each source base folder? If multiple target folders, how would you specify them in your "virtual folders"? What about FTP sites?

I think that your "virtual folders" are based on DaveP's idea (you say files & folders, he said folders), but that your definition of "virtual folders" so far doesn't really seem to address the more general cases that I mentioned.

In fact, I would say that there are really two feature requests here. I think that your idea about "virtual folders" is unique enough to be a separate feature request that deserves its own topic in the forums. By posting in the current topic, it seems like you're trying to kill Dave's and my specific feature request in favor of your own. Ideally, though, I think that both requests are valid but different and there could easily be a future version of BC that somehow incorporates all of these good ideas without prematurely eliminating some people's ideas.

Jonathan wrote:

I think this would do what you want, and also cover the more general case of someone who had copied an arbitrary group of files from around their hard disk, into a single folder or archive or removable disk, and wanted to compare all of these files to the copies, together.


I don't understand what you mean by this.

[davepacey]

I think I would rather have the ability to select multiple folders, in one session, and as Craig has already said this is on the wish list, I will leave it at that. If you want anything different, perhaps you should add it as a separate thread.

[Craig]

Don't worry guys, both DaveP's and Jonathan's requests are on the wishlist. Whether they're the same feature or not will really depend on how the final design turns out. :D

[jdmarch]

Pete and Dave, since I'm directly addressing the situations which you raised, I do think this is on topic, so I'm going to stay in this thread, and hope that you'll consider this alternate view.

I haven't seen any proposal for multiple base folders that would meet Pete's described needs. As he says, it's hard to devise a simple interface which interactively, in a single session, would synch both:
C:\Data --> D:\Data
N:\ --> O:\

Rather, as he suggested, this would seem to call for multiple sessions, which BC already supports. He suggested new features to quickly load multiple sessions at once, but this is already easily done with a batch file. Is it worth it, for something that's already easily doable, to add a new complication to BC? I think not, though I'm not opposed if enough users would find it helpful. But this is a totally separate issue from multiple base folders.

Dave's situation would be partially solved by the ability to include multiple subfolders, without the nuisance of explicitly excluding all the files in the base folder. I would also appreciate this feature and am glad it's on the wish list.

I don't know how easy it would be for Tim & Craig to permit My Computer to be the base folder, which would be necessary, as Pete points out, to allow folders from multiple drives to be included simultaneously, without contorting the interface. I'd be curious to hear Craig's thoughts on this.

In contrast, I think that virtual folders (VF) would give you exactly what you are looking for, plus a lot more, with almost no change in the existing interface.

To answer Pete's questions: A VF would behave much like an archive, except it would not actually contain any file data. Here's how Dave's backup job would be handled with VFs:

First time only, create the session:

1. In the left panel, create and open a new (ordinary) folder C:\VirtualFolders (the name and location are arbitrary.)

2. Use the new command "File / Create virtual folder" to create a new VF named "MyBackupData" within C:\VirtualFolders. (Behind the scenes, this would actually create a new empty text file MyBackupData.BCVF, which BC would treat as a folder, just as it can treat archives as folders. But it would display as an empty folder in BC.)

3. Make this new VF be your base folder in the left panel.

4. In the right panel, open drive C:.

5. Select the 7 folders on C: which you routinely back up.

6. "Copy" the 7 selected folders from right (actual) to left (VF). (Behind the scenes, this would not actually copy any files, but would just put the names and paths of your 7 backup folders into the BCVF text file).

7. In the right panel, open drive D:, and repeat steps 5 and 6 to "copy" your music files and/or folders into the VF.

8. In the right panel, open your DVD drive.

9. If desired, in the left panel, "Expand all", to see all the files which are in those 7+ folders in your VF.

10. Save this configuration as a new BC session named "data backup".

Doing the backup:

Once this session has been created, then every time you want to backup, just:

1. Open this saved session. You will see the current contents of your desired 7+ folders on C: and D: on the left, and the contents of the DVD on the right.

2. Compare and copy to your heart's content. Short and sweet. (The comparing and copying, of course, use the contents of the the physical files to which the BCVF entries were pointing; we might call these the "VF original" files.)

Notes:
* All BC features like filters would operate normally in VFs, so your backup could be as selective as you desire.

* Since the VF in this example contains only folder names, it will always show the current contents of those folders. If you wanted particular files to be in a VF, you could copy those files instead of folders.

* One would not be permitted to "Move" a file to a VF, because doing so eliminates the VF original file on which the VF depends.

* Deleting a file in a VF would just remove the listing in the BCVF file, and would not affect the VF original file in any way.

* If you copied from one VF to another, you would actually just be copying names and paths between BCVF files, but this would be a convenient way to create several slightly different backup scenarios.

[[Parenthetically, for anyone that is interested in grimy details: There are a number of implementation issues that would need to be resolved. In each case, there is a tradeoff between extra power and simplicity. For example:
* Whether/how to support FTP references (as Pete mentioned.)
* Whether one would be allowed to rename a file in a VF (if this were supported, then each item in the BCVF listing would need to have both its VF name and its original name, which would initially be the same, but the VF name could be renamed.)
* Whether to allow recursive VFs (copying a VF into a VF).
* Similarly, whether one could create virtual subfolders within a VF. (One way to implement this might be for the VF name, as well as the original (VF original) name, to include a path.)
* It would be useful to be able to put a root directory into a VF, but there's no obvious way to do this with the current BC GUI.
]]

[davepacey]

Jonathan,
I appreciate very much your idea and input into this problem, but I am very much a person who likes the simple approach and solution to tasks, never being one to waste time and energy, I am always seeking the quickest way to perform a given task, the route that uses the least effort, with minimal manpower to get the same result, more often at lower cost (comes with the job). Which is why I like the way Sychroniser Pro works, all you have to do is click on each folder you wish to include. I may try your method, But I like to see what is happening, and to be in total control: I will more likely wait till Craig has given us this option. My method of exclude works well enough for the time being on drive C:\

[Pete]

Jonathan,

Thank you for taking the time to elaborate.

[Craig]

BUT BEFORE YOU TRY THIS: right now there is a BUG in BC (at least in build 215) such that the exact sequence of commands which I just listed, does not work, because it actually excludes all files and folders on C: The same thing happens if you Edit / Select ALL, then deselect your desired 7 folders, then Exclude.



You know, bugs get fixed a lot faster if they're emailed to support and not buried in a long forum thread. :D In this case there are actually two issues, one bug and one not-bug that could be seen as a bug. The bug is that Exclude doesn't take orphan status into account, so if a file/folder isn't selected, but the empty side opposite it is, it will be excluded. That will be fixed in the next build. The not-bug is related to selection handling. If you have two base folders loaded, only select one side, and do an invert selection (or select all and unselecting), one side of that item is still selected so it will still be excluded. To get around that, both sides need to be unselected.

Of course, I'd still recommend using Include filters for cases like this. I haven't done timing tests, but I don't think 1000 exclude filters is going to be faster to check than 7 include ones...



First: This is all tentative, undecided, and with no promises as to when or in what version these ideas will be implemented.

Second: We appreciate the fact that lots of people are happy with the current interface, but we can and will change it if we think it will make the program easier to use. See the 1.9-to-2.0 changes if you need proof. ;)

Now that the preliminaries are out of the way...

On the most basic level we're probably going to allow custom alignments of disparate directories within a pair of "virtual base folders". Each real directory would appear as a subdirectory within that, and there wouldn't be any files in the virtual root. Using you're examples, C:\Data would be a subfolder with D:\Data across from it, and N:\ would be a second subfolder with O:\ across from it. Operations within those subdirectories would work exactly like they do now. FTP sites would be treated the same way, so they would work transparently too.

I really don't see any advantage to treating the virtual folders as archives. The alignments and any associated criteria would just be part of a session. Setting it up would occur in a separate window. We wouldn't try to force copy/move/delete into doing things they aren't designed for. I haven't worked out the shortcut/link support yet, but I haven't ruled out creating a directory full of shortcuts and synching that. That should have a similar effect to the .BCVF suggestion.

Extending things a bit, files could appear in the root too, but only as part of a custom alignment. Copying a orphan file into the virtual space wouldn't be allowed, but overwriting a paired file would be. For example, you could just have your Outlook .pst appear as a single entry in the list without worrying about complex filtering.

Another possibility we may add is a file operation queue. In that form, you would load each session separately, queue up the resulting synch operations, and then start the entire job once you've compared everything.

We do want to support synching shell objects (Desktop, My Computer, Network Neighborhood, etc), but I think that it will be separate from the above. Right now the roots of the drives are assumed to be the top-most points, and I think it's going to take some work to change that. There's also the issue of speed: the shell is/can be much slower than local disk access. When we support it, it will probably need to switch dynamically between shell and disk functions depending on where the synch is taking place, which may complicate the implementation.

[jdmarch]

Sounds good, Craig. I'll be patiently eager to see how it works out. A few brief points.

1. My interest in having the VF list in a text file is that sometimes it's a lot easier to create/edit a long list of file paths in an editor than in a point/click/drag/drop GUI. If you could make each VF list cut/copy/pastable (as, e.g. the filter lists are now) that would be handy. That would also makes it easier to create such lists in special-purpose batch files, for eventual use as BC VFs. (I'm not a big fan of binary configuration files, though I do understand that it makes it a lot easier to avoid support problems due to user error.)

2. I'm not clear from your message, but it would definitely be useful to be able to copy multiple folders from one virtual side into the same target location. In other words, not insist on 1-to-1 matching of folders on the 2 sides.

3. Yes, it would definitely be useful, one way or another, to be able to specify only a particular collection of files which are routinely copied to another folder.

Thanks!

[uidd]

Has this ever been implemented?

[Chris]

Multiple base folders is not in the current version Beyond Compare 3.1.7. It is still on our todo list for a future release.

[uidd]

A cross-posting of sorts: thanks. :)

[Lutz]

FYI: Since WinXP I use Mount Points to collect multiple base folders under one structure i. e.:

C:\AllVolumes
|
+- internal
| |
| +- 1 System (C)
| |
| +- 2 Data (D)
|
+- external
| |
| +- 3 16GB Stick
| |
| +- 4 1000GB HD
...

Create a structure like this with empty folders 1 to 4 and then create mount points using windows disk management for the volumes.

Finally just load C:\AllVolumes as BC3 base folder.

Greetings Lutz