BC changing LF to CR+LF when copying files

[DaveA]

I have a problem with BC changing end of line sequences from LF to CR+LF when copying files from my local machine to my target machine via sftp.

The setup:
Left hand pane /home/developer/vlu/scripts
Right hand pane [email protected] vlu/scripts

Files on the left contain changes that I want to copy to the right.
I click on the files that I want to transfer, right-click and select 'Copy to Right'.
The files are copied to the right, and now show no differences, (however the file on the right is larger than the left).
Inspection of the copied files on the target machine shows that all LF sequences have been changed to CR+LF sequences.

Regards,
Dave

[Aaron]

Hello,

If you are copying to or from an FTP, the default transfer mode is Auto, which will convert the line endings to the destination OS line ending. You can change this setting in the Tools menu -> Profiles, select your profile, Transfer tab, and switch to Binary to always transfer without conversion.

[DaveA]

Hi Aaron,

OK, forcing the transfer type to binary fixes the problem - thank you.

There may be a problem with BC's ASCII transfer mode, as your explanation suggests that I shouldn't get line ending conversion when transferring between Linux machines. In this case, BC is running on Linux Mint 17.1 and merging files/folders between the home folder on the same machine, and an sftp connected Linux embedded PC running Ubuntu 12.04.4 LTS - so there should be no conversion.

Regarding the profile transfer settings: BC help says "The FTP server will automatically make any adjustments to line endings needed in ASCII mode", so this may suggest a problem with the FTP server on my embedded target rather than a BC problem.
To test this, I have edited one of the bash scripts on the embedded targed via sftp using the text editor on the Linux Mint PC, without any unusual end of line conversions; so this would seem to point the finger at BC.

Regards,
Dave

[Aaron]

Do you have access to another FTP client to test this connection? Such as Filezilla, which also has the Transfer menu, which can be set to Auto or ASCII to test the transfer. If this works, please email us a copy of the Filezilla log and a copy of the BC4 log with debugging enabled (in the Options dialog, Tweaks section, Log section). Our email is [email protected] and please include a link back to this forum thread for our reference.

[DaveA]

A quick thread update for anyone else having similar problems with changed line endings.

I sent BC and Filezilla logs to Scooter as requested above, and have received the following response:

Hello Dave,

Given the information in the ReadMe file, you should update your profile to switch from AUTO to Binary. Auto will convert certain line endings of specified extensions, which you do not want. To prevent this, always transfer using Binary. you can update the specific profile in the Profiles dialog, or also update the Default profile.

Any already transferred files will need to be re-transferred or converted (the Text Compare can convert line endings).

-Aaron Polans, Scooter Software

I have set Tools/Profiles/Transfer tab/Transfer type = Binary for all profiles, and have not had any problems with changed line endings since.

[Aaron]

Right, good to hear.

Translating the line endings is purposeful behavior for an ASCII transfer. The methodology is that you are on your Windows machine, editing a website that runs on a Linux FTP. When you transfer files up to the Linux FTP, an ASCII transfer changes the line endings to Linux, so that any Linux apps and the OS can read and handle the files. This is necessary in some cases for the website to then run correctly, otherwise they generate errors. With more modern solutions, this scenario is becoming less necessary; also if the Linux machine is just storage it generally isn't needed.

[dr_barnowl]

I'm getting this with a pure Linux session :

* Left side is on SFTP to a Linux server
* Right side is SFTP to the same machine
* Both files are correctly recognised as having UNIX endings
* My BCompare session is on my Linux desktop

BUT

* Saving either file writes the whole file with CRLF endings

That's really counter intuitive, given every platform in the mix is UNIX, the protocols being used for transfer are typically UNIX, and the files are UNIX.

Changed the default profile as suggested and it's OK again.

Honestly, I'd rather the default was not to mess with line endings. ASCII / Binary modes are a relic of the days when the 8th bit made a difference in transmit time. Most sensible editors OTHER than Windows Notepad can cope with LF endings. The only other tools I've used in the last 15 years that I noticed caring about them are VB6 (some VB6 files have MIXED line endings and hate it if you change them, so this kind of blanket tinkering would mess that up) and Bash (which hates CRLF files).

(build 20720, 64-bit install on Ubuntu Linux)

[Aaron]

Hello,

I started up an Ubuntu test machine running BC 4.1.2 64bit Linux application, and loaded a folder compare pointing to the same SFTP server based in Linux with two different accounts. If I load a child Text Compare on a pair of files, it detects as Unix line endings. I can edit and save, or copy the entire file from one side to the other, but AUTO (Ascii) preserves the Linux line ending in this case. Could you give a few more details on your setup? We will probably also need a full copy of your Log with debug messages enabled (Tweaks). You can post here or email us privately at [email protected] with a link back to this forum thread.

The default mode is AUTO, which looks at the file extension to determine the method to use. Many FTP clients default to ASCII, as I mention earlier that a binary transfer would crash older web servers when these defaults were picked by various clients years ago. Changing the default for all users is something we've considered, but would probably not be updated in a minor release.

BC4 can be configured to have Binary as the default profile, which is then used for all profiles that do not override this setting. You can change this option in the Profiles dialog, and it will then be the default for you going forward.

[dr_barnowl]

My setup is a 14.04 box, accessing files via SFTP on a 15.10 VM (running as a guest on the main box).

The files are PHP, the text compare detects both of them as UNIX endings. Turning on visible line endings shows the pilcrow (¶) not the other thing (¤).

I have indeed configured Binary as the default profile which fixes this. I'll try and get you some logs tomorrow.

[Dave_L]

Turning on visible line endings shows the pilcrow (¶) not the other thing (¤).

Out of my obsession with irrelevant details, I found that the other thing's proper name is "currency sign".
Reference: https://en.wikipedia.org/wiki/Special_characters

(Both the pilcrow sign and currency signs even have their own Wikipedia pages, with history and so on, linked from that page.)

[Aaron]

Hello,

We use the ¤ sign for Windows line endings, which should appear on a side that is hosted on a Windows machine. If both sides are Linux, the expected sign is ¶ for the Linux line endings. Older Mac line endings use the << character, although many modern OSX editors use Unix ¶ line endings.

If both sides are SFTP locations on Linux servers, and you are running BC4's Linux client, ¶ display makes sense (as well as the text "Unix" in the top status bar of each pane). Is there a method you are using that gives the files the Windows line ending? Or are you trying to inject the Windows line ending (Edit menu -> Convert -> Line endings)?

[dr_barnowl]

Tested as follows :

With profile as "Auto"

* Started session. Session is two folders on the same (Ubuntu) server, via SFTP.
* Content is PHP source code with Unix (LF) file endings
* On compare, both sides displayed as UNIX / Pilcrow correctly
* Left side is a Git clone
* Added/removed space to left file to set dirty flag. Saved file.
* Git now identifies file as changed
* Vim now shows file has DOS file endings
* Reloading compare INCORRECTLY identifies left file as UNIX
* Running dos2unix on file restores it to unchanged condition
* "Auto" saves file with PC endings regardless of detected / presented endings

With profile as "Binary"

* Reload compare
* Correctly identifies file as PC endings
* Also correctly identifies file after being fixed back to LF endings
* Saves file with presented endings

Have attached log with all options on and repeated examples of these steps.

[Aaron]

Thanks for these steps. I had been using BC4 to verify the line ending characters, and using an external tool or BC4 in Binary Mode helps reveal an issue. We'll investigate to figure out what's going on here.

[Chris]

The SFTP ASCII transfer bug is fixed in Beyond Compare 4.1.3.

[mryan]

We use the ¤ sign for Windows line endings, which should appear on a side that is hosted on a Windows machine. If both sides are Linux, the expected sign is ¶ for the Linux line endings. Older Mac line endings use the « character, although many modern OSX editors use Unix ¶ line endings.

If both sides are SFTP locations on Linux servers, and you are running BC4's Linux client, ¶ display makes sense (as well as the text "Unix" in the top status bar of each pane). Is there a method you are using that gives the files the Windows line ending? Or are you trying to inject the Windows line ending (Edit menu -> Convert -> Line endings)?

Sorry to necro an old thread, but I was just curious why the currency symbol ¤ (scarab) was chosen for Windows line endings.

The pilcrow used for Linux line endings is referenced on Wikipedia - Newline, however I wasn't able to find any other examples of the currency symbol being associated with line endings. Was it simply a conveniently available symbol or was there a historical reason for its selection?