Folder Sync SFTP Mac/Linux Issue

[EdThorne]

I was using BC4 to sync files from my mac to a Linux box (RHEL 6.4) when I noticed something odd was happening. BC4 when copying my shell script (*.sh) was changing the size of the file.

When I compared the files in the sync view they reported being identical with the same line endings but differing byte sizes. So I did a hex compare thinking that's a byte for byte comparison and should show me where those extra bytes were. It didn't. They were still identical.

So I tar'd up the bash script on the linux machine and then compared the file in the tar file with the file in the source directory from my mac. This time I see the same byte difference in size. However I can now also see that the file on the Linux machine is reporting PC line endings.

To recreate this issue simply create a bash script on the Mac and folder sync it to a Linux machine. You should see it get larger when it arrives on the Linux machine. An undesired side effect in all this is that the script doesn't execute on the Linux machine due to the line ending changes.

[Aaron]

Thanks for the report. I've reproduced something similar to this and we'll look into it.

[Zoë]

This isn't a bug in BC, it's a "feature" of vsftpd in that it treats ASCII transfers as binary ones. ASCII transfers are supposed to send the files using Windows line endings (among other things), so when BC uploads it it does the conversion and the FTP server is supposed to convert it again when it receives the data. vsftpd doesn't do that, so you're seeing the file as it's sent over the wire.

The workaround is to change your default FTP profile in BC to always use binary transfers (FTP Profiles -> Transfer -> Transfer Type). We'll consider changing the default in BC4.

[EdThorne]

I assume you're referring to the SSH_FXF_TEXT_MODE flag in the specification. I'm also assuming that vsftpd is the server process handling the request. In my case OpenSSH is servicing the request.

What I am discovering is that the SSH_FXF_TEXT_MODE flag was added into Draft 11 of the SFTP specification. This is SFTP protocol version 6 according to section 5 which describes initialization. In this section it indicates the client sends it's version and the server sends back the lower of it's version or the client's version. The server's response dictates what version of the protocol is used.

I suspect in the case of OpenSSH that the SFTP protocol version in use is version 3. I'm basing this on their FAQ and the link to draft 2 of the specification contained therein. This means a client connecting to OpenSSH shouldn't use SSH_FXF_TEXT_MODE because it wasn't added until version 6. Thus all transfers with protocol version less than 6 are always binary. I suspect BC4 isn't honoring the server's reported SFTP protocol version.

[Zoë]

Ah, sorry, I didn't notice that this was an SFTP server and not a plain FTP one.

We do respect the SFTP protocol version, but on servers that don't support that flag we do the conversion on the client side and transfer it as a binary file. It's only supposed to do the conversion if the line ending styles for the client and server don't match, so there must be a bug in our detection of the server software. The workaround is still to disable ASCII transfers.

Edit: Found it: the library we're using for SSH support assumes that if it's compiled for OS X it's using the old Mac carriage-return only line terminator locally. Easy fix, it'll be in the next build.