Segmentation fault

[David]

Are you on a 64bit system? If so, when you updated glibc did you also update the 32bit glibc?

[Zaister]

I am on a multilib Gentoo system, and, yes, the glibc update also created 32-bit libraries.

[David]

Another user reported problems with glibc 2.18 on gentoo (64bit). I tried with glibc 2.18 on Ubuntu 13.10 64bit (updating 32bit libs) and was not able to repeat the crash.

The other user did a gdb backtrace and an strace trace and sent them to me. From those traces it appears that something is going wrong when BC tries to access the locale information through glibc calls.

This user regressed his glibc back to 2.17 quickly and wasn't able to give more info including whether the 32bit libs had been updated or not. I had hoped that since glibc2.18 worked for me on Ubuntu that the issue was related to a mismatch between glibc 2.17 32 and glibc 2.18 64 on his machine.

Can you pass along the output for 'locale' from a terminal window?

[David]

And perhaps run 'locale-gen' if you haven't already done so.

[Zaister]

Certainly; my locale is somewhat customized. It looks like this (note that LC_MESSAGES is English):

LANG=de_DE.utf8
LC_CTYPE="de_DE.utf8"
LC_NUMERIC="de_DE.utf8"
LC_TIME="de_DE.utf8"
LC_COLLATE="de_DE.utf8"
LC_MONETARY="de_DE.utf8"
LC_MESSAGES=en_US.utf8
LC_PAPER="de_DE.utf8"
LC_NAME="de_DE.utf8"
LC_ADDRESS="de_DE.utf8"
LC_TELEPHONE="de_DE.utf8"
LC_MEASUREMENT="de_DE.utf8"
LC_IDENTIFICATION="de_DE.utf8"
LC_ALL=

EDIT: running locale-gen and logging in again doesn't change any behavior.

[Zaister]

This is what I'm getting from gdb (not 100% sure I'm doing it right, though):

stefan@juliet ~ 2 $ LD_LIBRARY_PATH="$LD_LIBRARY_PATH;/opt/beyondcompare/lib64" gdb /opt/beyondcompare/bin/BCompare
GNU gdb (Gentoo 7.6.2 p1) 7.6.2
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
For bug reporting instructions, please see:
<http://bugs.gentoo.org/>...
Reading symbols from /opt/beyondcompare/bin/BCompare...(no debugging symbols found)...done.
(gdb) run
Starting program: /opt/beyondcompare/bin/BCompare
warning: the debug information found in "/usr/lib64/debug/lib64/ld-2.18.so.debug" does not match "/lib/ld-linux.so.2" (CRC mismatch).

warning: Could not load shared library symbols for linux-gate.so.1.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
0xf787767d in __strstr_sse42 (s1=0xf7e583b7 "objectDestroyedSlot(QObject*)", s2=0xf75b442d "()")
at ../sysdeps/x86_64/multiarch/strstr.c:385
385 ../sysdeps/x86_64/multiarch/strstr.c: No such file or directory.


EDIT: strace output is here: http://pastebin.kde.org/ptpoh4lwj

[David]

The LD_LIBRARY_PATH does not look quite right - it should point to where the beyondcompare libqt-mt.so.3 library is (a 32 bit library - BC is a 32bit app). Under deb or rpm install, the binary (BCompare) and the libs will be installed in

/usr/lib/beyondcompare/BCompare
/usr/lib/beyondcompare/libqt-mt.so.3
/usr/lib/beyondcompare/libqtc.so.1
/usr/lib/beyondcompare/..... (other files)

Executing gdb would look like
$ LD_LIBRARY_PATH="/usr/lib/beyondcompare" gdb /usr/lib/beyondcompare/BCompare

(and then we install script 'bcompare' into /usr/bin to set LD_LIBRARY_PATH and call to /usr/lib/beyondcompare/BCompare)

From your trace, it appears that gdb is trying to use 64 bit linker for some reason

"the debug information found in "/usr/lib64/debug/lib64/ld-2.18.so.debug" does not match "/lib/ld-linux.so.2"

And I am curious about the path /opt/beyondcompare/lib64. Is this where you placed the libqtc.so.1 and libqt-mt.so.3? They are actually 32bit. Though obviously the path name shouldn't make a difference as long as BCompare can see the libs.

[Zaister]

The library path is correct, I've used an ebuild provided a few years ago by a user here on the forum that installs the 32-bit libraries into that path. If the path wasn't correct, I'd get an error listing the libraries (I have seen that output before), and not just "Segmentation fault", though.

The "lib64" of that path is created by the ebuild which calls Portage's get_libdir function. While this isn't quite right obviously, it's not harmful. Anyway the libraries are installed correctly.

I'll go and recompile glibc, maybe something went wrong there, although since I have the problem both on my home and my office machine, and the office machine has no debug symbols at all, I don't think that mismatch is the cause of the segfault.

When this problem is handled, I think, I will create a new ebuild based off the install script that's handling the installation more cleanly. I will then post that ebuild here, if that is OK.

[David]

Given that someone else has seen a similar issue I am also not convinced that you have a bad glibc build i.e. I am not expecting that rebuilding will fix the problem.

The other piece of information that you can get from gdb, is after the program segfaults call 'thread apply all backtrace' (though in this case I don't think any additional threads have been spun up yet so 'backtrace' is probably sufficient). This will print out a stack trace of where the program was when things went south - or approximately.

running

'strace LD_LIBRARY_PATH="$LD_LIBRARY_PATH;/opt/beyondcompare/lib64" /opt/beyondcompare/bin/BCompare'

Will show the system calls being made as it crashes.

You are most welcome to post an updated ebuild here. It would probably be useful to me so see where everything is installed on your machine with the current ebuild. I am assuming that everything is sitting under /opt/beyondcompare? So maybe a 'find /opt/beyondcompare' dump.

[David]

Did a google on __strstr_sse42 and hit this on the glibc wiki page

https://sourceware.org/glibc/wiki/Release/2.18

5.3.2. Issues with ``__strstr_sse42``

It has been reported that after upgrading to 2.18 some users have experienced crashes in __strstr_sse42. The first reports of this were from Allan McRae (Arch Linux) here: http://www.sourceware.org/ml/libc-al.../msg00257.html. It appears that this is a compiler issue, but none the less it results in a non-functioning glibc 2.18 that passes the testsuite cleanly but then causes most of x86 userspace to crash. The fundamental issue is that the psABI is not followed and the stack is not maintained aligned. If __strstr_sse42 is called with an unaligned stack it will fault as this violates the psABI. The only real solution is to patch glibc to make these functions inline again, but that isn't a real fix and only papers over the actual problem which will eventually occur if any of the code is perturbed enough to get a misaligned stack again. At present there are two potential solutions. The first solution is to disable the use of the SSE42 routines in glibc, but that has a performance penalty. The consensus around a correct solution is to rebuild the faulty code that calls __strstr_sse42 with a new enough compiler (any gcc released after the year 2000 when -msse went in with the 16 byte stack alignment) that always ensures the stack is aligned modulo 16 bytes, or to correct any assembly that may leave the stack unaligned. There is still some debate over the issue since glibc should try to be as backwards compatible as possible and should therefore support older binaries with misaligned stacks calling SSE42 strstr, and doing this would require new entry points that do dynamic stack realingment. However, doing the dynamic stack realignment is an added cost to the SSE42 strstr routine which already has poor asymptotic performance whom several argue should be removed entirely or fixed, see bug https://sourceware.org/bugzilla/show_bug.cgi?id=12100.

[Zaister]

Sure, here's the backtrace info (unfortunately it seems glibc symbols are missing):

Program received signal SIGSEGV, Segmentation fault.
0xf787767d in __strstr_sse42 (s1=0xf7e583b7 "objectDestroyedSlot(QObject*)", s2=0xf75b442d "()")
at ../sysdeps/x86_64/multiarch/strstr.c:385
385 ../sysdeps/x86_64/multiarch/strstr.c: No such file or directory.
(gdb) backtrace
#0 0xf787767d in __strstr_sse42 (s1=0xf7e583b7 "objectDestroyedSlot(QObject*)", s2=0xf75b442d "()")
at ../sysdeps/x86_64/multiarch/strstr.c:385
#1 0xf7215213 in QConnection::QConnection(QObject const*, int, char const*, int) ()
from /opt/beyondcompare/lib64/libqt-mt.so.3
#2 0xf72713a5 in QObject::connectInternal(QObject const*, int, QObject const*, int, int) ()
from /opt/beyondcompare/lib64/libqt-mt.so.3
#3 0xf7271d27 in QObject::connect(QObject const*, char const*, QObject const*, char const*) ()
from /opt/beyondcompare/lib64/libqt-mt.so.3
#4 0xf7c8d8d6 in SignalHookBase::SignalHookBase(QObject*) () from /opt/beyondcompare/lib64/libqtc.so.1
#5 0xf7e42eca in new_QTimerSignalHook () from /opt/beyondcompare/lib64/libqtc.so.1
#6 0x08095239 in ?? ()
#7 0x080ea7dd in ?? ()
#8 0x080f8d7a in ?? ()
#9 0x08061b90 in ?? ()
#10 0x08068d80 in ?? ()
#11 0x08068e0e in ?? ()
#12 0xf7765ea3 in __libc_start_main (main=0x8068d9c, argc=1, argv=0xffffce24, init=0x8068e65, fini=0x8068e65,
rtld_fini=0xf7feb800 <_dl_fini>, stack_end=0xffffce18) at libc-start.c:285
#13 0x08068e64 in ?? ()

I posted a link to the strace output in a previous post, but here it is again: http://pastebin.kde.org/ptpoh4lwj

Yes, the ebuild I'm currently using installs under /opt/beyondcompare. It is customary for Gentoo ebuilds to install to /opt any package that is not compiled from sources. Only the documentation is installed into /usr. Here's the find dump:

stefan@juliet ~ 2 $ find /opt/beyondcompare/
/opt/beyondcompare/
/opt/beyondcompare/bin
/opt/beyondcompare/bin/bcompare.sh
/opt/beyondcompare/bin/BCompare
/opt/beyondcompare/bin/help
/opt/beyondcompare/lib64
/opt/beyondcompare/lib64/kde_context_menu.sh
/opt/beyondcompare/lib64/libqtc.so.1
/opt/beyondcompare/lib64/kde_context_menu
/opt/beyondcompare/lib64/context_init.sh
/opt/beyondcompare/lib64/libqt-mt.so.3

[Zaister]

Oh, interesting report about the glibc problem there. Is this something I can fix on my side (it seem my gcc should be new enough), or is that something you need to take into account when building Beyond Compare? I'm not quite sure.

[David]

I am not sure. I think that compiling glibc, on your end, with -mno-sse would work. I think this should work since on our side the libqt-mt.so.3 library that we provide is just calling symbol 'strstr' and doesn't know anything about __strstr_sse42 and so it must be the glibc code that is compiled to change that call into a call to sse42 code. Compiling glibc with -mno-see would prevent a strstr call being turned into a __strstr_msee42 call. The downside, from the bug report, seems to be a performance hit.

At first I thought that that was the only option (we are using an old old Delphi/Kylix toolset to build linux BC and have yet to find an adequate upgrade). However, the gcc compiler we build libqt-mt.so.3 is from 2005 (not quite as old as Kylix that build BCompare) and should have a -msee mode. I will trying doing a rebuild of libqt-mt.so.3 with -msee on. If it compiles and works locally then I will pass it on to you to try on your system.

[Zaister]

Thanks, that sounds good. I will try your recompiled lib then, if it workds for you.

[David]

I have recompiled the lib and it still works for BC on my build machine. Send an email to [email protected] with a link to this thread and we will get you a link to download the -msse2 test library.