Packages have invalid .gnu_debuglink

Bug #261380 reported by Yuri Timenkov
40
This bug affects 2 people
Affects Status Importance Assigned to Milestone
qt4-x11 (Ubuntu)
Fix Released
Medium
Unassigned
Nominated for Jaunty by Martin von Gagern

Bug Description

I've encountered this problem while investigating crash in amarok-kde4 beta1, but this affects every package in my Kubuntu Hardy distribution (x86_64).

I suppose the contents of .gnu_debuglink section in all packages are invalid, therefore gdb fails to load symbols even if appropriate -gdb packages are installed (which makes them useless):
On my Kubuntu:
# objdump -s -j .gnu_debuglink /usr/lib/libQtGui.so.4.4.0

/usr/lib/libQtGui.so.4.4.0: file format elf64-x86-64

Contents of section .gnu_debuglink:
 0000 6c696251 74477569 2e736f2e 342e342e libQtGui.so.4.4.
 0010 30000000 000101b1 0.......

Note, that this is not libQtGui.so.4.4.0.debug shipped with libqt4-dbg

On Fedora 8:
# objdump -s -j .gnu_debuglink /usr/lib/libQtGui.so.4.3.3

/usr/lib/libQtGui.so.4.3.3: file format elf32-i386

Contents of section .gnu_debuglink:
 0000 6c696251 74477569 2e736f2e 342e332e libQtGui.so.4.3.
 0010 332e6465 62756700 0607bf23 3.debug....#

Note that this points to .debug file name which is installed with appropriate debug package.

I can confirm this for libqt4* 4.4.0-1ubuntu5~hardy1 packages and seems every other. Because of this bug I can't step into calls to libqt4 functions (But the main probelm is that I can't get useful backtraces to file crash reports).

System information:
# lsb_release -rd
Description: Ubuntu 8.04.1
Release: 8.04
#uname -a
Linux gigawin2 2.6.24-20-generic #1 SMP Mon Jul 28 13:06:07 UTC 2008 x86_64 GNU/Linux

Revision history for this message
Brian Murray (brian-murray) wrote :

Martin could you provide some insight here?

Revision history for this message
Martin Pitt (pitti) wrote :

In general the links are not invalid, but they look just fine as far as I can see. Fedora's debug symbol packages are structured differently, which is why their debug link needs to point to "libQtGui.so.4.3.debug". However, in Debian/Ubuntu, both the explicit -dbg packages, as well as the automatically created -dbgsym ones on http://ddebs.ubuntu.com usually do *not* append the ".debug" suffix to files, but just store them in /usr/lib/debug/<original path>. That's why a debug link equal to the original file name is correct.

However, in the particular case of libqt4-dbg, I confirm the bug. This source package seems to not use the standard debhelper way to produce -dbg packages, but something custom which uses the Fedora schema (see http://packages.debian.org/sid/amd64/libqt4-dbg/filelist). Thus the package needs to be fixed to either install files under the (ubuntu/debian) standard path (/usr/lib/debug/usr/lib/libQtCore.so.4.4.2 instead of /usr/lib/libQtCore.so.4.4.2.debug), or (less preferred) fix their objcopy calls to attach correct debug links.

Revision history for this message
Martin Pitt (pitti) wrote :

cf. for example http://packages.debian.org/sid/amd64/libc6-dbg/filelist and http://packages.debian.org/sid/amd64/libglib2.0-0-dbg/filelist. I deal with -dbg packages quite a lot, and in fact qt4-x11 is the only package I encountered so far which breaks this schema.

Revision history for this message
Martin von Gagern (gagern) wrote :

Confirming for 4.4.3-0ubuntu1 on Intrepid.

Revision history for this message
Martin von Gagern (gagern) wrote :

The crc32 sum seems to be wrong as well. Therefore simply moving/renaming the files won't solve the issue.

To use the libQtGui.so.4.4.3 from 4.4.3-0ubuntu1 on i386 as an example: here /usr/lib/libQtGui.so.4.4.3.debug has a crc32 value of 0x61598269, computed using the algorithm given in the gdb manual and compared against one calculated using objcopy. The .gnu_debuglink section of /usr/lib/libQtGui.so.4.4.3 states 0x18743161. I manually converted endianess; objdump displayed reverse byte order in both cases.

As to the cause of this issue: I just had a look at the build log from https://launchpad.net/ubuntu/+source/qt4-x11/4.4.3-0ubuntu1/+build/729722 and found the following line:

(test -z "../../lib/" || cd "../../lib/" ;
targ=`basename libQtGui.so.4.4.3`;
objcopy --only-keep-debug "$targ" "$targ.debug" &&
objcopy --strip-debug "$targ" &&
objcopy --add-gnu-debuglink="$targ.debug" "$targ" &&
chmod -x "$targ.debug" ) ;

So there the link seems to get set correctly, according to the Fedora scheme, including the .debug extension and presumably with the correct checksum as well. Something seems to modify the installed so file later on, or perhaps the file modified by above command isn't even the one installed later on. I couldn't find any likely command actually naming libQtGui, but it might be a command using a glob, or I might have overlooked something. Strange.

Generating separate debug files according to Fedora naming scheme seems to be the default configuration of the qt4 build process. A viable solution to both solve the broken links and also get the package to follow the preferred Ubuntu naming scheme would be to disable both debug symbol separation and stripping from the Qt build process, and instead do the debug symbol separation later on the Ubuntu way, with whatever tools other packages might use.

Looking at Gentoo, the way to achieve this seems to be to pass -no-separate-debug-info to configure and CONFIG+=nostrip to qmake. As Ubuntu, in contrast to Gentoo, doesn't seem to invoke qmake directly, I don't know how to achieve this. A viable solution would probably to include it in the default value of QMAKE_VARS in the configure script. See also http://sources.gentoo.org/viewcvs.py/gentoo-x86/eclass/qt4-build.eclass?view=markup for how Gentoo builds its qt4 packages.

Revision history for this message
Martin von Gagern (gagern) wrote :

Another observation: Debian (tested with sid packages) also uses Fedora layout for qt4-x11, but doesn't seem to suffer from broken debug links. So what do they differently?
1. mention *.debug in debian/not-installed
2. some differences in debian/rules that I don't completely understand
3. several other differences that seem unlikely to have anything to do with this
4. using debian tools instead of ubuntu ones to build things

Revision history for this message
Jonathan Thomas (echidnaman) wrote :

Riddell just merged Qt4 for Jaunty. I wonder if the merge fixed this, although nobody should recommend anybody upgrading to Jaunty this early on to test.

Revision history for this message
Jonathan Thomas (echidnaman) wrote :
Revision history for this message
Martin von Gagern (gagern) wrote :

It isn't necessary to update to Jaunty just to check this. It would be enough to get a library package unpacked so you can objdump it, and to compare the file name mentioned there with those from the dbg package file list. "dbg -x" would do the former, while the latter might be possible without even downloading the package itself.

Unfortunately it looks like the 1ubuntu1 build is too brand new. The links to the binary packages on the page you referred to lead to launchpad error pages, and packages.ubuntu.com is still taking about version 0ubuntu1. So if you can lay your hand on these packages, you might try this out, but I didn't manage to yet.

Revision history for this message
Asraniel (asraniel) wrote :

Is there no fix for this in hardy?

Revision history for this message
Martin von Gagern (gagern) wrote :

Still not fixed in 4.4.3-1ubuntu4_i386 for Jaunty.

I had a typo in comment 9; s/"dbg -x"/"dpkg -x"/

To get some traction here, I think the following should work:
1. (debian/patches/) patch configure to include "CONFIG+=nostrip" in QMAKE_VARS
2. (debian/rules) pass "-no-separate-debug-info" to the configure invocation
3. (debian/rules) call "dh_strip --dbg-package=libqt4-dbg" to separte the debugging symbols the Ubuntu way
4. (debian/rules) drop all other packaging instructions for the libqt4-dbg package

Right now I don't even have access to my Ubuntu system, so I can't try this out just now. Might take a few weeks before I can work on this again, so if you want to experiment in that direction, be my guest.

Revision history for this message
Martin von Gagern (gagern) wrote :

I've got a proposed version in my PPA, with debug packages managed by dh_strip, not the qt build process.
See https://launchpad.net/~gagern/+archive

Adding CONFIG+=nostrip to QMAKE_VARS in the configure script didn't work, so I added it to some other spec file. I also found that cdbs provides elaborate support for debug packages, and made use of that. The resulting list of debug symbol files looks somewhat longer, needs to be invedtigated still.

I'll attach the patch to stop the build from stripping its binaries as well as a patch of the rules file later on; the system where I developed this is already down. Until then, you can compare my diff or simply use my version.

Revision history for this message
Martin von Gagern (gagern) wrote :
Revision history for this message
Martin von Gagern (gagern) wrote :

This patch can be added to the debian patch series in order to prevent the qt build system from stripping objects.

Revision history for this message
Martin von Gagern (gagern) wrote :

Complete list of my modifications:
* Modified debian/rules according to attached script.
* Added nostrip patch to debian patch series
* rm debian/*-dbg.install
* sed -i 's: \./\(.*\)\.debug: ./usr/lib/debug/\1:' debian/*-dbg.lintian
* Updated changelog :-)
I could have wrapped all this in a single patch, but I guess that would have been less readable and more difficult to integrate with other changes.

Revision history for this message
Martin von Gagern (gagern) wrote :

Can some Ubuntu qt packager please have a look at this?

I've put quite a lot of personal time, and even more computer runtime, into trying to get this fixed. I've posted the relevant modifications as well as instructions how to integrate them. I'm clobbering my PPA with almost 1 GiB of packages for this, even though they have been outdated less than a month after publishing them.

On the other hand, this bug hasn't seen much useful comments from devs since about 5 months. I would like to know
1. whether you are going to fix this for Jaunty, in order to get usable qt backtraces there
2. whether my patches lead in the right direction, or if they are unacceptable for some reason
3. whether it makes sense to update my PPA, or if I should simply delete those packages 'cause noone else cares
4. whether the updated intrepid packages might be placed in some team PPA, to save space in my own PPA

Changed in qt4-x11:
importance: Undecided → Medium
Revision history for this message
Marco Maini (maini10) wrote :

Please attach to this report a patch following the instructions reported here: https://wiki.ubuntu.com/Bugs/HowToFix.
Then subscribe to this report ubuntu-main-sponsors.
Thanks for your work.

Revision history for this message
Martin Pitt (pitti) wrote :

Marco, there is no patch for this problem yet, so your comment about sponsoring is irrelevant here.

Revision history for this message
Gabriel de Perthuis (g2p) wrote :

The fixing is described in comments 12 through 15. Martin's PPA has it in patch form (debdiff I think): http://launchpadlibrarian.net/20834194/qt4-x11_4.4.3-0ubuntu1_4.4.3-0ubuntu1.2~ppa0i.diff.gz

Revision history for this message
Martin Pitt (pitti) wrote :

Ah, indeed, I overlooked that. I apologize.

Debdiff: http://launchpadlibrarian.net/20834194/qt4-x11_4.4.3-0ubuntu1_4.4.3-0ubuntu1.2%7Eppa0i.diff.gz

I subscribed the sponsoring team.

Revision history for this message
Martin von Gagern (gagern) wrote :

Do I understand you correctly that the diff in my PPA is enough for now, rendering Marco's request for a patch obsolete?
So now I wait for some sponsor to have a look at this, and work out the next steps based on his opinion?
The version against which my patch was created is no longer current. Should I create a new patch, and if so, against Jaunty, Intrepid, or both?

Revision history for this message
Martin von Gagern (gagern) wrote :

> The resulting list of debug symbol files looks somewhat longer, needs to be invedtigated still.

I've investigated this now. The original debug packages only contained debug symbols for libraries. With my patch in place, it contains debug symbols for binaries as well. Debug files for the following additional files end up in my libqt4-dbg:

1 from libqt4-dbus: /usr/bin/qdbus
1 from qt4-designer: /usr/bin/designer-qt4
1 from qt4-qtconfig: /usr/bin/qtconfig-qt4
9 from qt4-dev-tools: assistant-qt4 assistant_adp linguist-qt4 pixeltool qcollectiongenerator qdbusviewer qhelpconverter qhelpgenerator xmlpatterns
10 from libqt4-dev: lrelease-qt4 lupdate-qt4 moc-qt4 qdbuscpp2xml qdbusxml2cpp qmake-qt4 qt3to4 rcc uic-qt4 uic3
217 from qt4-demos: qtdemo, 5 libs from /usr/lib/qt4/plugins/designer, 15 demos and 196 examples

It would seem to me that for the former, the libqt4-dbg package is a suitable place, even if they amount to something like 63MiB unpacked space on the system.

The number of files corresponding to the demos package is excessively large, though, and they consume around 145MiB unpacked. I suggest to either start a separate package qt4-demos-dbg for those, or dropping debug info for them altogether. While the use of those demos to the common qt user is probably pretty small, those actually using the package are developers and thus likely to be interested in debugging them as well. Therefore I'd opt for one more debug package to be built from the qt4-x11 source package.

I'll create an updated patch for this.

Revision history for this message
Martin von Gagern (gagern) wrote :

I created an updated patch for Jaunty. It failed to build on my PPA, though, due to some PostgreSQL-related issue that seems unrelated to my patch, but makes testing a bit difficult. I'll try a local build tomorrow, I think. The attached patch introduces three new debug packages, for qt4-dev-tools and libqt4-dev as well as qt4-demos, in order to reduce the size of installed debug files for each of these packages.

Revision history for this message
Martin Pitt (pitti) wrote :

Martin, thanks for your work on this. Some questions/notes:

 - Do we really need to explicitly build -dbg packages, given that pkg-create-dbgsym already builds them for every package in the archive anyway? The long-term goal is actually to remove -dbg packages entirely (and we are pondering doing that on the Ubuntu builds). As long as the package builds with -O2 -g and leaves stripping to dh_strip, it should Just Work. That works for tens of thousands of other packages.

  So perhaps the -no-separate-debug-info addition and debian/patches/bug261380-nostrip.diff is everything that's required here?

 - the lintian overrides should be removed from the patch. Lintian should't complain about /usr/lib/debug/, that's a lintian bug.

Revision history for this message
Martin von Gagern (gagern) wrote :

It took me a while to find the relevant documentation about pkg-create-dbgsym:
https://launchpad.net/pkg-create-dbgsym
https://wiki.ubuntu.com/AptElfDebugSymbols
https://launchpad.net/ubuntu/+spec/apt-get-debug-symbols

If this is really the way to go, some reference in https://wiki.ubuntu.com/PackagingGuide would be useful.

I'm not an experienced Ubuntu/Debian packager, but here is my impression comparing those two solutions:

1. My patch, using cdbs and old -dbg packages
- additional -dbg packages added to control
- debug symbols from multiple binary packages can be collected in one debug package
- easy to build on ppa and install locally using apt-get

2. Your suggested method, using pkg-create-dbgsym and new ddeb packages
- all -dbg packages removed from control
- exact 1:1 corresponcence between binary packages and debug packages
- don't know how to install those packages locally, nor how ppa deals with them

As https://wiki.ubuntu.com/AptElfDebugSymbols#apt%20changes regarding installation of debug symbols doesn't seem implemented yet, at least not on my intrepid, I'm not convinced that this approach is ready to replace the old -dbg packages yet. I still created a corresponding patch.

About lintian: I haven't understood the functioning of lintian yet, nor the reason for those overrides in the old package. My renames were simply intended to have as little impact on current behaviour as possible. I'd be happy to drop them. If we drop the -dbg packages, the corresponding lintian files have to go in any case.

Revision history for this message
Jonathan Thomas (echidnaman) wrote :

If we dropped the dbg packages now, that would mean we would have to drop the dbg packages from every KDE module since kdelibs5-dbg depends on the qt4 -dbg packages, and every kde dbg package depends on kdelibs5-dbg. That would be a somewhat large change this late in the cycle, and it would create an annoying delta with Debian. I say that for Jaunty we should just fix the -dbg packages, and then maybe we can consider going to dbgsym packages in Karmic.

Revision history for this message
Martin Pitt (pitti) wrote : Re: [Bug 261380] Re: Packages have invalid .gnu_debuglink

Jonathan Thomas [2009-03-18 12:23 -0000]:
> If we dropped the dbg packages now, that would mean we would have to
> drop the dbg packages from every KDE module since kdelibs5-dbg depends
> on the qt4 -dbg packages, and every kde dbg package depends on
> kdelibs5-dbg. That would be a somewhat large change this late in the
> cycle, and it would create an annoying delta with Debian. I say that for
> Jaunty we should just fix the -dbg packages, and then maybe we can
> consider going to dbgsym packages in Karmic.

Thanks, that makes sense.

Revision history for this message
Martin von Gagern (gagern) wrote :

OK, yet another debdiff. Revertet to -dbg packages (including a few extra ones), dropped the lintian overrides, and also completed the mapping from non-debug to debug packages, as the implicit cdbs build logic didn't work as I originally expected. Currently queued in my ppa, let's see...

Revision history for this message
Jonathan Riddell (jr) wrote :

I've uploaded this debdiff following review by Bo Thorsten. Thanks Martin von Gagern. We'll revisit this in Karmic.

Revision history for this message
Steve Langasek (vorlon) wrote :

Jonathan, what more needs to revisited here? Since the proposed patch was uploaded to jaunty, I'm unsubscribing ubuntu-main-sponsors at least.

Revision history for this message
Martin von Gagern (gagern) wrote :

I guess one thing that needs revisiting in Karmic is the proposed migration to dbgsym packages. Is there some page to track progress on this, not only in qt but all over Ubuntu?

By the way, how about backports? Would it make sense to backport the patch to hardy, which is a LTS after all? In its current state, the debug packages are almost unusable. Unfortunately, fixing the debug packages without rebuilding the other packages would be a major effort, so I'd not do this. At the next opportunity when packages need to be updated to fix some other issues, it might make sense to fix this one as well. What do you think?

Revision history for this message
sabby (sabby) wrote :

I saw that the patch was integrated in the last package 4.5.3 uploaded for karmic, note however that the source don't compile because the patch still as the old path in it, i.e.

+--- qt4-x11-4.4.3.orig/mkspecs/common/linux.conf 2008-12-31 11:20:45.000000000 +0100
++++ qt4-x11-4.4.3/mkspecs/common/linux.conf 2008-12-31 11:21:36.000000000 +0100

instead of the usual a/ and b/.

Revision history for this message
Jonathan Thomas (echidnaman) wrote :

Looks like we're not going to be able to drop -dbg packages, but I think fabo from Debian said that they were picking up some of our -dbg work for Qt 4.6. This can be closed, anyways.

Changed in qt4-x11 (Ubuntu):
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.