Home Directory Comparisons Between Systems?

[SSnow]

I have just switched from using Vista SP1 and MS Office 2007 on my two notebooks to using Ubuntu 8.04 and OpenOffice (with help from Thunderbird, Sunbird, and Tomboy Notes to replace the functions I was using in Outlook).

I had long used (since Windows 2000) Beyond Compare to synchronize my primary user account between systems at least once each day. I made interim backups of only files changed since the most recent synchronization to a flash drive. I managed this such that complete and sudden failure of one system would only result in the downtime necessary to copy the recently changed files from the flash drive to the remaining good computer.

I'm new to Linux. As such, I'm not quite sure where all the data is. Looks like practically all of it is in /home, much of it being in hidden folders. Is it feasible for me to synchronize these two systems in Ubuntu along more or less the same lines in which I was doing it in Windows?

I'm guessing I need to set up a secure ftp server or nfs or even Samba as a first step. After that it's a matter of figuring which locations under /home to include, and which to exclude.

Would anyone feel like helping me get started on this? I'm just bright enough that a gentle push might be enough to get me started, as long as its down the right path. I presume I would have to edit some configurations / folder names on the backup system so that the profile (for instance) for Thunderbird was identical on the two systems.

I know this question might be more fittingly posted on the Ubuntu forums, but I'm thinking that will raise "software religion" issues with some, and arguments from others about methodology. I'm thinking anyone who uses Beyond Compare under Linux is probably pretty familiar with the information I need.

Would someone please throw this old dog a bone?

[Dave_L]

I'm using Mepis, which, like Ubuntu, is based on Debian.

I use BC to partially synchronize some directories between a desktop and laptop, but I don't do a full synch. A full synch of /home would *probably* work, but I would make a backup before trying it.

One caveat is that the user IDs must be the same on both systems, since BC preserves the user ID when copying files.

I use sshfs to access the remote system. sshfs needs to be installed on the client; it's probably in the Ubuntu repositories if it's not already installed. The server requires the SSH daemon, which (in my distro) is part of the package openssh-server.

There are also configuration files outside /home, such as in /etc, that you may want to synch. Those would be owned by root.

At work, we run an hourly task to mirror a production system onto a spare computer. I use rsync for this, rather than BC.

[SSnow]

Thank you for your response, Dave_L.

I guess I should roll up my sleeves and learn about rsync. I've heard of it before, but I'm truly new to this OS. I have the advantage of having wandered about more-or-less aimlesslessly for many years in VMS, SCO, AIX, Windows, CP/M, etc. I didn't learn a lot, but I can usually get used to a new environment pretty quickly.

Perhaps I should use BC3 for doing file content comparisons and stick with rsync for the inter-system stuff.

Again, many thanks for your observations. I'll start researching sshfs and rsync.

[SSnow]

Resolution with BC3 was dead easy.

Wow! As it turns out I just installed BC3. I had to gksudo bcompare from a terminal window in order to not have BC3 being hassled about what it did and did not have permission to see / alter in the home directories on the two systems. I did that so that I could set up the exclusions easily. I decided against doing a full synchronization of /home on the two computers, electing instead to sync all regular non-hidden data folders and a few hidden folders (like profiles for Thunderbird, Sunbird, Firefox and some dotted-name folders which contain configuration information for things like BC and Zim).

I just used the sftp functionality (being careful to have it use binary copy). Works like a charm. As usual, just couldn't be easier, or more flexible and powerful. This is some piece of software on any platform.

[paulonolan]

I would like to be able to sync home my home directory on an Ubuntu laptop to my NAS (a Netgear, formerly Infrant, ReadyNAS NV). On the NAS it's possible to create a share belonging to sharename (and group membership nogroup) and to rsync a home directory in such a way that permissions are copied (I use grsync which works nicely). For greater security user mode security is also an option, as is domain security, but if it's mostly for sharing Windows files the share mode works fine also for NFS.

I have found that I can connect via sftp to a Documents folder and similar (ordinary, non home) folders on a backup share on the NAS using BC 3.07, however logging in to a home folder with sftp://root@readynasbox//share/home/username starts off well but but runs off the rails as soon as some directory entries are read, and eventually starts looping. Trouble begins with

Unable to load lost+found: Invalid argument
Unable to load chatscripts: Invalid argument
Unable to load .adobe: Invalid argument
etc.
etc.
(loops)

No idea why. Here's what the directory looks like with ls -la

total 264
drwxr-xr-x 46 1000 1000 4096 Sep 23 15:09 .
drwxr-xr-x 3 1000 1000 4096 Sep 9 04:12 ..
drwx------ 3 1000 1000 4096 Sep 8 03:27 .adobe
-rw------- 1 1000 1000 2704 Sep 22 16:17 .bash_history
-rw-r--r-- 1 1000 1000 220 Sep 9 04:12 .bash_logout
-rw-r--r-- 1 1000 1000 2940 Sep 9 04:12 .bashrc
drwxr-xr-x 2 1000 1000 4096 Sep 22 15:27 .beyondcompare
drwxr-xr-x 3 1000 1000 4096 Sep 8 03:20 .cache
drwxr-xr-x 7 1000 1000 4096 Sep 8 18:28 .config
drwxr-xr-x 2 1000 1000 4096 Sep 22 09:58 Desktop
-rw------- 1 1000 1000 28 Sep 13 07:54 .dmrc
drwxr-xr-x 2 1000 1000 4096 Sep 22 11:08 Documents
-rw------- 1 1000 1000 16 Sep 8 03:20 .esd_auth
drwxr-xr-x 3 1000 1000 4096 Sep 8 10:42 .evolution
drwxr-xr-x 2 1000 1000 4096 Sep 8 03:20 .fontconfig
drwx------ 5 1000 1000 4096 Sep 13 07:54 .gconf
drwx------ 2 1000 1000 4096 Sep 22 20:36 .gconfd
-rw-r----- 1 1000 1000 0 Sep 22 20:28 .gksu.lock
drwx------ 12 1000 1000 4096 Sep 22 10:17 .gnome2
drwx------ 2 1000 1000 4096 Sep 8 03:20 .gnome2_private
drwx------ 2 1000 1000 4096 Sep 13 07:54 .gnupg
drwx------ 2 1000 1000 4096 Sep 22 20:33 .grsync
drwxr-xr-x 2 1000 1000 4096 Sep 8 03:20 .gstreamer-0.10
-rw-r--r-- 1 1000 1000 104 Sep 22 09:58 .gtk-bookmarks
drwx------ 2 1000 1000 4096 Sep 8 03:20 .gvfs
-rw------- 1 1000 1000 628 Sep 22 17:37 .ICEauthority
drwxr-xr-x 2 1000 1000 4096 Sep 8 10:54 .icons
drwx------ 4 1000 1000 4096 Sep 22 15:33 .kde
drwxr-xr-x 3 1000 1000 4096 Sep 8 03:20 .local
-rw-r--r-- 1 root root 0 Sep 23 15:09 ls.txt
drwx------ 3 1000 1000 4096 Sep 8 03:27 .macromedia
drwxr-xr-x 3 1000 1000 4096 Sep 22 15:33 .mcop
-rw------- 1 1000 1000 31 Sep 22 15:33 .mcoprc
drwx------ 3 1000 1000 4096 Sep 8 03:20 .metacity
drwx------ 5 1000 1000 4096 Sep 8 18:31 .mozilla
drwxr-xr-x 2 1000 1000 4096 Sep 8 03:20 Music
drwxr-xr-x 2 1000 1000 4096 Sep 11 18:12 MyLifeOrganized
drwxr-xr-x 3 1000 1000 4096 Sep 13 03:10 .nautilus
drwx------ 3 1000 1000 4096 Sep 22 14:40 .openoffice.org2
drwxr-xr-x 2 1000 1000 4096 Sep 8 03:20 Pictures
-rw-r--r-- 1 1000 1000 586 Sep 9 04:12 .profile
drwxr-xr-x 2 1000 1000 4096 Sep 8 03:20 Public
drwxr-xr-x 2 1000 1000 4096 Sep 8 10:43 .pulse
-rw------- 1 1000 1000 256 Sep 8 03:20 .pulse-cookie
drwxr-xr-x 2 1000 1000 4096 Sep 22 15:33 .qt
-rw------- 1 1000 1000 1413 Sep 22 14:40 .recently-used
-rw-r--r-- 1 1000 1000 7406 Sep 22 14:40 .recently-used.xbel
drwx------ 2 1000 1000 4096 Sep 9 04:42 .scim
drwxr-xr-x 5 1000 1000 4096 Sep 22 10:14 Software
drwx------ 2 1000 1000 4096 Sep 22 00:15 .ssh
-rw-r--r-- 1 1000 1000 0 Sep 8 03:32 .sudo_as_admin_successful
drwxr-xr-x 2 1000 1000 4096 Sep 8 03:20 Templates
drwxr-xr-x 2 1000 1000 4096 Sep 8 10:54 .themes
drwx------ 3 1000 1000 4096 Sep 8 10:54 .thumbnails
drwxr-xr-x 4 1000 1000 4096 Sep 8 18:31 .tomboy
-rw-r--r-- 1 1000 1000 4518 Sep 13 07:55 .tomboy.log
drwx------ 2 1000 1000 4096 Sep 22 15:28 .unison
drwxr-xr-x 2 1000 1000 4096 Sep 8 03:32 .update-manager-core
drwx------ 2 1000 1000 4096 Sep 9 04:40 .update-notifier
drwxr-xr-x 2 1000 1000 4096 Sep 8 03:20 Videos
drwxr-xr-x 2 1000 1000 4096 Sep 13 07:55 .wapi
drwxr-xr-x 4 1000 1000 4096 Sep 12 14:35 .wine
-rw------- 1 1000 1000 116 Sep 13 07:54 .Xauthority
-rw-r--r-- 1 1000 1000 12330 Sep 22 22:37 .xsession-errors

I would be very happy indeed to be able to run BC over SSH!

I realize that this is not as reliable solution as running sbackup, say, which also backs up other important non-home directories, but this could make a very convenient adjunct.

[Zoë]

Hi Paul,

In the FTP Profiles dialog, on the "Listing" tab, change the "Link Resolution" option from "Fast" to "Simple". "Fast" assumes that any shortcuts that have an extension are files and any that don't are folders. "Simple" asks the server for every link's target's type.

[paulonolan]

Craig,

Thanks but... same result.

Paul

[Zoë]

Paul,

I don't see either lost+found or chatscripts in the directory listing you posted. Do any of the subdirectories work? Is it possible it's a permissions issue? Have you tried changing a directory's mode to 777?

[paulonolan]

Craig,

Yes. I tried to upload a BcTraceLog.txt file but it's too long. I'll email it.

Paul

[paulonolan]

Some improvement as of 3.0.9 (build 9222), but no cigar yet.

Comparing 220Gb on

sftp://root@readynas//c and sftp://root@readynas//USB/USB_HDD_7/c

(my ReadyNAS share folder and a copy on a USB drive attached to it)

I no longer get the immediate looping but I do eventually get

Unable to load socket-eeepc: No such file
Unable to load socket-eeepc: No such file
Unable to load tmp-eeepc: No such file
Unable to load tmp-eeepc: No such file
Unable to load lock: No such file
Unable to load lock: No such file
(so far... i.e., after 20+ mins)

and, indeed, no such files ever existed that I know of. (I do have folder called EEEpc1000)

uh oh, now a flood

dnsdomainname
egrep
false
frgrep
gunzip
nisdomainname
pidof
sh
uncompress
usleep
vi
ypdomainname
zcat
zcmp
console
core
MAKEDEV
mouse
ram
sr0 ... sr9
vcsa
localtime
and lots more, all the way to vgsplit (two lines for each, as above)

then

unable to load c, lots of "Unable to load exe: No such file", then more linux terms: rsmtp, runq, sendmail, the Desktop. At which point (30 mins) I think we may have a loop.

It looks better, but comparing two remote directories is sloooow (running on my Asus 1000 over a wireless-N link)

[Jacques]

drwxr-xr-x 4 1000 1000 4096 Sep 12 14:35 .wine

I would try excluding this directory, inside .wine/dosdevices there is likely a symlink pointing to '/'.

[paulonolan]

OK, thanks. I'll try it and report back. Is this something that could or should have been picked up by BC? Either generically or specifically --Wine is not an uncommon thing.

Apropos of earlier discussion of rysnc, which I'm using to backup my ReadyNAS at the moment, I'd be interested to see a table comparing basic functionality of a few things like Unison, Rsync etc. with BC. I realize there's little to touch BC for any visual comparisons.

I'm still a relatively new linux user (not many hours, though I have mastered the rsync trailing slash learning curve<g>).

Some things I use, all on Windows, are good at knowing when a file has been renamed (e.g., superflexible). My guess is that rsync has no clue. However, its ability to copy only the parts of files that have changed is outstanding.

[Zoë]

BC's symlink (and possibly hardlink) support needs to be improved, so yes, this should be handled, but isn't currently.

In all honesty, if you're doing non-interactive syncs, rsync is probably better for mirrors, and Unison is probably better for updates, than BC is right now. Since both started as Unix utilities and have been out on that platform for longer, they're both likely to have any quirks with symlinks/hardlinks/permissions handled already. I'm happy with what we've released so far, and I definitely want to make BC competitive in that area, but I think our primary strength currently is in the GUI.

[Zoë]

Oh, and the reason that rsync/unison won't have an issue with this is that they should both have an option to copy symlinks as symlinks. BC currently doesn't, so it recursively follows them.