empty ddebs when dh_strip is called twice

Bug #562418 reported by Alberto Bertogli
32
This bug affects 5 people
Affects Status Importance Assigned to Milestone
pkg-create-dbgsym (Ubuntu)
Fix Released
High
Martin Pitt
Lucid
Fix Released
High
Martin Pitt
xorg-server (Ubuntu)
Invalid
Undecided
Unassigned
Lucid
Invalid
Undecided
Unassigned

Bug Description

SRU INFORMATION:
Impact: Produces empty ddebs and broken debug links on packages which call dh_strip on a package more than once, when being built on Soyuz (not locally).
Patch: http://bazaar.launchpad.net/~ubuntu-core-dev/pkg-create-dbgsym/ubuntu/revision/180
Test case: http://bazaar.launchpad.net/~ubuntu-core-dev/pkg-create-dbgsym/ubuntu/revision/176
Regression potential: p-c-d has quite an extensive test suite which cover a lot of different build systems, debhelper compat levels, etc. Almost all bug reports get a test written first to ensure that it reproduces the bug and the patch fixes it. However, being a toolchain package it can potentially cause all further builds to be broken completely (which is very very unlikely, though); a more realistic regression potential is that this now causes debug packages to stop being built because the check for "has debug symbols" is imprecise.

Update: Probably affects all packages which happen to run dh_strip twice on the same binary package.

Binary package hint: xserver-xorg-core-dbg

After upgrading to Lucid, I get the following error when trying
to debug xorg:

$ gdb /usr/bin/Xorg
GNU gdb (GDB) 7.1-ubuntu
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/bin/Xorg...
warning: the debug information found in "/usr/lib/debug//usr/bin/Xorg" does not match "/usr/bin/Xorg" (CRC mismatch).

warning: the debug information found in "/usr/lib/debug/usr/bin/Xorg" does not match "/usr/bin/Xorg" (CRC mismatch).

(no debugging symbols found)...done.
(gdb)

It worked just fine on 9.10.

I've version 2:1.7.6-2ubuntu3 installed, on amd64.

If you need more information, please let me know.

Thanks,
     Alberto

Related branches

Revision history for this message
Bryce Harrington (bryce) wrote :

Hi Alberto,

Please run the command 'apport-collect BUGNUMBER', which will attach several files we need for debugging.

[This is an automated message. Apologies if it has reached you inappropriately; please just reply to this message indicating so.]

tags: added: needs-xorglog
tags: added: needs-lspci-vvnn
Changed in xorg-server (Ubuntu):
status: New → Incomplete
Revision history for this message
Alberto Bertogli (albertito) wrote :

Assign to the proper package so apport-collect can be run.

Revision history for this message
Alberto Bertogli (albertito) wrote :

I tried running apport-collect, but ran into this:

$ apport-collect 562418
Package xorg-server not installed and no hook available, ignoring

*** Updating problem report

No additional information collected.

Press any key to continue...

No pending crash reports. Try --help for more information.

So I tried to change the package this bug is assigned to, but got
this message:

  'xserver-xorg-core-dbg' is a binary package. This bug has been assigned to its source package 'xorg-server' instead

So, what do you want me to do?

Thanks,
    Alberto

Revision history for this message
Robert Hooker (sarvatt) wrote :

It looks like this is caused by the build rules and only affects packages run through pkg-create-dbgsym (in the archive) because PPA and self built packages work right.

The ddeb's aren't being built properly with xserver 1.7.x where the build rules have changed, they're only a few kb.
http://ddebs.ubuntu.com/pool/main/x/xorg-server/

Also, for mesa which does:

        dh_strip -plibgl1-mesa-swx11 --dbg-package=libgl1-mesa-swx11-dbg
        dh_strip -plibgl1-mesa-glx --dbg-package=libgl1-mesa-glx-dbg
        dh_strip -plibgl1-mesa-dri --dbg-package=libgl1-mesa-dri-dbg

the -dbg packages do not have valid CRC's but the -dbgsym's do work properly. This also does not affect PPA builds where the -dbg packages actually build correctly and work.

The debian/rules for xorg-server:
http://git.debian.org/?p=pkg-xorg/xserver/xorg-server.git;a=blob;f=debian/rules;h=eb5928547f257fa52e2fb52b335ce470870d9cb2;hb=refs/heads/ubuntu

and for mesa:
http://git.debian.org/?p=pkg-xorg/lib/mesa.git;a=blob;f=debian/rules;h=f661e9d59eba8ba5ee45dcc7954a7d6e0d652766;hb=refs/heads/ubuntu

Build log for an improperly stripped xorg-server:
http://launchpadlibrarian.net/43932202/buildlog_ubuntu-lucid-i386.xorg-server_2:1.7.6-2ubuntu3_FULLYBUILT.txt.gz

Changed in xorg-server (Ubuntu):
status: Incomplete → Confirmed
summary: - xserver-xorg-core-dbg debug symbols mismatch
+ many -dbg packages have debug symbols mismatch
Changed in pkg-create-dbgsym (Ubuntu):
status: New → Confirmed
Revision history for this message
Tormod Volden (tormodvolden) wrote : Re: many -dbg packages have debug symbols mismatch

This problem arises for packages which run dh_strip several times in debian/rules. When using only the "real" dh_strip (as in PPAs) it is no problem, because dh_strip will detect an already stripped file and ignore it.

pkg_create_dbgsym, on the other hand, will gladly strip a file a second time, and thus obtain no debug symbols from it, but still create the almost zero symbol file, and to top it off, write this useless CRC to the binary.

So in the end the -dbgsym packages will have no symbols and a good CRC, while the -dbg packages have no matching CRC any longer.

For instance the mesa rules file should be changed so it does not run dh_strip on the same package twice. Now it does:
dh_strip -plibgl1-mesa-dri --dbg-package=libgl1-mesa-dri-dbg
dh_strip -s

Here the second line should have had -Nlibgl1-mesa-dri appended to it so that these packages are not processed again. (Both dh_strip and pkg_create_dbgsym are smart enough to ignore -dbg packages).

But since we often sync packages from Debian where this is no problem, we should make pkg_create_dbgsym handle this better. See the one-liner in my linked branch.

Revision history for this message
Tormod Volden (tormodvolden) wrote :

Interestingly, xorg-server tries this:
dh_strip -pxserver-xorg-core --dbg-package=xserver-xorg-core-dbg
dh_strip -s --remaining-packages

Unfortunately that is wishful thinking, because dh_strip does not have any --remaining-packages option.

I have tested that my fix works on the mesa packages. See the attachment for details on what happens at which steps in the stripping processes.

description: updated
Revision history for this message
Tormod Volden (tormodvolden) wrote :

I was kindly pointed to the debhelper generic option --remaining-packages. I can imagine that the "real" dh_strip calls some debhelper functions to honour this option (it uses $dh{DOPACKAGES}), but I guess the dh_strip wrapper in pkg_create_debug does not so it will do its ddebs work on all packages before handing over to the real dh_strip.

Revision history for this message
Martin Pitt (pitti) wrote :

Ah, *headdesk*, thanks for figuring this out! I'll create and commit a test case and then merge your branch.

Changed in pkg-create-dbgsym (Ubuntu):
assignee: nobody → Martin Pitt (pitti)
status: Confirmed → In Progress
Revision history for this message
Martin Pitt (pitti) wrote :
Changed in pkg-create-dbgsym (Ubuntu):
status: In Progress → Fix Committed
importance: Undecided → High
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package pkg-create-dbgsym - 0.42

---------------
pkg-create-dbgsym (0.42) lucid; urgency=low

  * Add tests/dhtest.doublestrip/: Call dh_strip twice, to check whether this
    keeps the debug symbols intact. This reproduces LP #562418, where debug
    symbols and CRC checksums were damaged due to double-stripping.
  * pkg_create_dbgsym: Do nothing if we already have built a .ddeb during this
    package build, i. e. it is in debian/files already. This fixes
    overwriting good .ddebs and debug links in case dh_strip gets called
    twice. Thanks to Tormod Volden for investigating this! (LP: #562418)
  * debian/control: Bump Standards-Version to 3.8.4 (no changes necessary).
 -- Martin Pitt <email address hidden> Thu, 22 Apr 2010 10:36:38 +0200

Changed in pkg-create-dbgsym (Ubuntu):
status: Fix Committed → Fix Released
Revision history for this message
Martin Pitt (pitti) wrote :

So with that xorg-server just needs a rebuild. If one is done for bug 565981 anyway, then this will just be solved along with it, otherwise you have my ack to do a no-change upload for right-after-RC to get the ddebs back.

Thanks again, Tormod!

Changed in xorg-server (Ubuntu):
status: Confirmed → Fix Committed
milestone: none → ubuntu-10.04
Revision history for this message
Bryce Harrington (bryce) wrote :

-RC is out, I've uploaded the fix for 565981. I forgot to mention this bug in that changelog unfortunately, so this will need to be closed manually.

Revision history for this message
Bryce Harrington (bryce) wrote :

The upload was approved today and should now be included in ubuntu

Changed in xorg-server (Ubuntu):
status: Fix Committed → Fix Released
Revision history for this message
Tormod Volden (tormodvolden) wrote :

Unfortunately this is not fixed in the xorg-server case. The build log for 2:1.7.6-2ubuntu7 confirms that pkg-create-dbgsym 0.42 got installed, but I still get CRC mismatch on /usr/bin/Xorg.

The snippet below from the build log shows that -dbgsym was built twice. Why there is Make messages in there is also a bit funny, it seems like some parallel makes are running, which I would not expect in this phase of the build.

dh_strip -pxserver-xorg-core --dbg-package=xserver-xorg-core-dbg
dh_strip debug symbol extraction: all non-arch-all packages for this build platform i386: xserver-xorg-core xserver-xorg-dev xdmx xdmx-tools xnest xvfb xserver-xephyr xserver-xfbdev xserver-xorg-core-dbg
pkgmaintainermangler: Not overriding Maintainer for domain lists.ubuntu.com
dpkg-deb: warning: 'debian/xserver-common/DEBIAN/control' contains user-defined field 'Original-Maintainer'
dpkg-deb: warning: ignoring 1 warnings about the control file(s)

dpkg-deb: building package `xserver-common' in `../xserver-common_1.7.6-2ubuntu7_all.deb'.
dh_strip debug symbol extraction: not adding gnu debuglinks since --dbg-package is given
dh_strip debug symbol extraction: packages to act on: xserver-xorg-core
dh_strip debug symbol extraction: ignored packages:
make[1]: Leaving directory `/build/buildd/xorg-server-1.7.6'
WARNING: not running pkgbinarymangler for this package, as requested
dpkg-deb: warning: '/build/buildd/xorg-server-1.7.6/debian/xserver-xorg-core-dbgsym/DEBIAN/control' contains user-defined field 'Original-Maintainer'
dpkg-deb: warning: ignoring 1 warnings about the control file(s)

dpkg-deb: building package `xserver-xorg-core-dbgsym' in `../xserver-xorg-core-dbgsym_1.7.6-2ubuntu7_i386.ddeb'.
dh_strip -s --remaining-packages
dh_strip debug symbol extraction: all non-arch-all packages for this build platform i386: xserver-xorg-core xserver-xorg-dev xdmx xdmx-tools xnest xvfb xserver-xephyr xserver-xfbdev xserver-xorg-core-dbg
dh_strip debug symbol extraction: packages to act on: xserver-xorg-core xserver-xorg-dev xdmx xdmx-tools xnest xvfb xserver-xephyr xserver-xfbdev xserver-xorg-core-dbg
dh_strip debug symbol extraction: ignored packages:
WARNING: not running pkgbinarymangler for this package, as requested
dpkg-deb: warning: '/build/buildd/xorg-server-1.7.6/debian/xserver-xorg-core-dbgsym/DEBIAN/control' contains user-defined field 'Original-Maintainer'
dpkg-deb: warning: ignoring 1 warnings about the control file(s)

dpkg-deb: building package `xserver-xorg-core-dbgsym' in `../xserver-xorg-core-dbgsym_1.7.6-2ubuntu7_i386.ddeb'.
WARNING: not running pkgbinarymangler for this package, as requested
dpkg-deb: warning: '/build/buildd/xorg-server-1.7.6/debian/xdmx-dbgsym/DEBIAN/control' contains user-defined field 'Original-Maintainer'
dpkg-deb: warning: ignoring 1 warnings about the control file(s)

Changed in pkg-create-dbgsym (Ubuntu):
status: Fix Released → Confirmed
Martin Pitt (pitti)
Changed in pkg-create-dbgsym (Ubuntu):
status: Confirmed → In Progress
Changed in xorg-server (Ubuntu):
status: Fix Released → Confirmed
milestone: ubuntu-10.04 → none
Revision history for this message
Martin Pitt (pitti) wrote :

Argh, I know why. It's because on Soyuz we currently disable adding the ddebs to debian/files, thus the grep wouldn't find them.

Revision history for this message
Martin Pitt (pitti) wrote :
Changed in pkg-create-dbgsym (Ubuntu):
milestone: none → lucid-updates
Changed in pkg-create-dbgsym (Ubuntu Lucid):
status: In Progress → Fix Committed
Martin Pitt (pitti)
description: updated
Revision history for this message
Martin Pitt (pitti) wrote :

I built xorg-server locally with this package, and it works now:

dh_strip -pxserver-xorg-core --dbg-package=xserver-xorg-core-dbg
dh_strip debug symbol extraction: all non-arch-all packages for this build platform amd64: xserver-xorg-core xserver-xorg-dev xdmx xdmx-tools xnest xvfb xserver-xephyr xserver-xfbdev xserver-xorg-core-dbg
dh_strip debug symbol extraction: not adding gnu debuglinks since --dbg-package is given
dh_strip debug symbol extraction: packages to act on: xserver-xorg-core
dh_strip debug symbol extraction: ignored packages:
dpkg-deb: Warnung: »/home/martin/ubuntu/tmp/xorg-server-1.7.6/debian/xserver-xorg-core-dbgsym/DEBIAN/control« enthält benutzerdefiniertes Feld »Original-Maintainer«
dpkg-deb: Warnung: Ignoriere 1 Warnungen wegen der control-Datei(en)

dpkg-deb: Baue Paket »xserver-xorg-core-dbgsym« in »../xserver-xorg-core-dbgsym_1.7.6-2ubuntu7_amd64.ddeb«.
dh_strip -s --remaining-packages
dh_strip debug symbol extraction: all non-arch-all packages for this build platform amd64: xserver-xorg-core xserver-xorg-dev xdmx xdmx-tools xnest xvfb xserver-xephyr xserver-xfbdev xserver-xorg-core-dbg
dh_strip debug symbol extraction: packages to act on: xserver-xorg-core xserver-xorg-dev xdmx xdmx-tools xnest xvfb xserver-xephyr xserver-xfbdev xserver-xorg-core-dbg
dh_strip debug symbol extraction: ignored packages:
xserver-xorg-core is already stripped, ignoring
xserver-xorg-dev is already stripped, ignoring

-rw-r--r-- 1 martin martin 5204262 2010-04-26 14:28 xserver-xorg-core-dbgsym_1.7.6-2ubuntu7_amd64.ddeb

So I uploaded this to lucid-proposed, awaiting SRU approval now.

Martin Pitt (pitti)
summary: - many -dbg packages have debug symbols mismatch
+ empty ddebs when dh_strip is called twice
Revision history for this message
Robert Hooker (sarvatt) wrote :

Awesome, thanks Martin and Tormod! I tested it locally and indeed it does fix it here as well.

Changed in xorg-server (Ubuntu Lucid):
status: Confirmed → Invalid
Revision history for this message
Martin Pitt (pitti) wrote :

> Confirmed → Invalid

Well, the current ddebs are still broken, so my gut feeling would be to leave it open until p-c-d goes to lucid-updates and xorg-server gets rebuilt. It's of some value to have working debug symbols for an LTS, I guess.

Revision history for this message
John Dong (jdong) wrote :

ACK from SRU for the patch.

Revision history for this message
Martin Pitt (pitti) wrote : Please test proposed package

Accepted pkg-create-dbgsym into lucid-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

tags: added: verification-needed
Revision history for this message
Uwe Geuder (ubuntulp-ugeuder) wrote :

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 10.04 LTS
Release: 10.04
Codename: lucid

I believe I'm affected by the bug. When trying to debug Xorg I get the CRC mismatch warnings mentioned in the original bug description.

So I thought I could help verify the correction. I understood that the problem was in pkg-create-dbgsym and the correction to this package should now be verified.

The first thing I did was building xorg-server source package. That went fine after installing all the required tools and -dev packages.

$ dpkg-buildpackage -rfakeroot -uc -us

... was the command I used.

I still don't have pkg-create-dbgsym installed at all in this phase...

$ apt-cache policy pkg-create-dbgsym
pkg-create-dbgsym:
  Installed: (none)
  Candidate: 0.42
  Version table:
     0.42 0
        500 http://fi.archive.ubuntu.com/ubuntu/ lucid/main Packages

..., but to my surprise the debug package...

../xserver-xorg-core-dbg_1.7.6-2ubuntu7_i386.deb

... has been generated by the "normal" build command

I had guessed the the debug package would be created by using (tools inside) pkg-create-dbgsym.

I do understand that launchpad is not a forum for questions, but if somebody could provide short instructions
how to verify the correction I would be grateful.

Revision history for this message
Uwe Geuder (ubuntulp-ugeuder) wrote :

Oops sorry, I guess the answer is in the bug description:

> Produces empty ddebs and broken debug links ...
> , when being built on Soyuz (not locally).

Not sure what Soyuz stands for here, but I guess the problem affects only Ubuntu build infrastructure.
If I build locally on my own machine I don't have the problem in the first place, so there is no way to verify the correction either?

Revision history for this message
Martin Pitt (pitti) wrote :

Uwe,

you actually have to install pkg-create-dbgsym before building xorg-server. The -dbg package is manually built by xorg-server itself. pkg-create-dbgsym builds automatic -dbgsym packages for everything.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package pkg-create-dbgsym - 0.43

---------------
pkg-create-dbgsym (0.43) lucid-proposed; urgency=low

  * pkg_create_dbgsym: Do not check debian/files for whether we already
    processed a .ddeb, since Launchpad Soyuz disables dpkg-distaddfile for
    ddebs still. Instead, skip the run if we do not find any unstripped binary
    in the current package. (LP: #562418)
 -- Martin Pitt <email address hidden> Mon, 26 Apr 2010 14:00:38 +0200

Changed in pkg-create-dbgsym (Ubuntu Lucid):
status: Fix Committed → Fix Released
Revision history for this message
Martin Pitt (pitti) wrote :

Copied lucid-proposed to maverick.

Changed in pkg-create-dbgsym (Ubuntu):
status: Fix Committed → Fix Released
Changed in pkg-create-dbgsym (Ubuntu Lucid):
status: Fix Released → Fix Committed
Revision history for this message
Martin Pitt (pitti) wrote :

I built current lucid xorg-server with the lucid-proposed pkg-create-dbgsym, and the .ddebs are now alright.

tags: added: verification-done
removed: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package pkg-create-dbgsym - 0.43

---------------
pkg-create-dbgsym (0.43) lucid-proposed; urgency=low

  * pkg_create_dbgsym: Do not check debian/files for whether we already
    processed a .ddeb, since Launchpad Soyuz disables dpkg-distaddfile for
    ddebs still. Instead, skip the run if we do not find any unstripped binary
    in the current package. (LP: #562418)
 -- Martin Pitt <email address hidden> Mon, 26 Apr 2010 14:00:38 +0200

Changed in pkg-create-dbgsym (Ubuntu Lucid):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.