Launchpad itself

retry-depwait is falling over

Bug #492632 reported by Julian Edwards
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Launchpad itself
Fix Released
Critical
Julian Edwards
Launchpad itself 3.1.12

Bug Description

2009-12-04 19:55:36 INFO Processing The Lucid Lynx for sparc (sparc)
2009-12-04 19:55:36 INFO Found 78 builds in MANUALDEPWAIT state.
2009-12-04 19:55:38 INFO Retrying sparc build of gnome-shell 2.28.1~git20091125-1 in ubuntu lucid RE
LEASE
2009-12-04 19:55:38 INFO Processing The Karmic Koala for amd64 (amd64)
2009-12-04 19:55:38 INFO Found 573 builds in MANUALDEPWAIT state.
2009-12-04 19:55:50 INFO Retrying amd64 build of erlang 1:13.b.1-dfsg-2 in ubuntu karmic RELEASE
2009-12-04 19:55:51 INFO Retrying amd64 build of gnome-bluetooth 2.27.8-0ubuntu1 in ubuntu karmic RE
LEASE
Traceback (most recent call last):
  File "/srv/launchpad.net/codelines/current/cronscripts/buildd-retry-depwait.py", line 35, in <module>
    script.run()
  File "/srv/launchpad.net/codelines/soyuz-production-rev-8745/lib/lp/services/scripts/base.py", line 2
48, in run
    self.main()
  File "/srv/launchpad.net/codelines/soyuz-production-rev-8745/lib/lp/soyuz/scripts/buildd.py", line 17
1, in main
    build_set.retryDepWaiting(distroarchseries)
  File "/srv/launchpad.net/codelines/soyuz-production-rev-8745/lib/lp/soyuz/model/build.py", line 1110,
 in retryDepWaiting
    build.buildqueue_record.score()
  File "/srv/launchpad.net/codelines/soyuz-production-rev-8745/lib/lp/soyuz/model/build.py", line 114,
in buildqueue_record
    return results.one()
  File "/srv/launchpad.net/codelines/soyuz-production-rev-8745/eggs/storm-0.15danilo_storm_launchpad_r3
42-py2.5-linux-x86_64.egg/storm/store.py", line 1117, in one
    raise NotOneError("one() used with more than one result available")
storm.exceptions.NotOneError: one() used with more than one result available

Tags: lp-soyuz

Related branches

lp:~julian-edwards/launchpad/die-buildqueue-die--bug-492632
Björn Tillenius (community): Approve (release-critical)
Muharem Hrnjadovic (community): Approve
Diff: 80 lines (+45/-0)
2 files modified
lib/lp/soyuz/model/buildqueue.py (+9/-0)
lib/lp/soyuz/tests/test_build.py (+36/-0)
Revision history for this message
Julian Edwards (julian-edwards) wrote :

retrying builds in RELEASE for a released series doesn't sound very healthy :(

Changed in soyuz:
status: New → Triaged
importance: Undecided → Critical
assignee: nobody → Julian Edwards (julian-edwards)
milestone: none → 3.1.11
Revision history for this message
Julian Edwards (julian-edwards) wrote :

Red herring, that's a copy-archive build.

The basic problem, it seems, is that there's a race condition in creating BuildQueue records and queue-builder is fighting with regular build creation.

http://pastebin.ubuntu.com/336488/ will delete the extra BQ records until we fix the code.

Revision history for this message
Julian Edwards (julian-edwards) wrote :

This query shows the affected builds:
http://pastebin.ubuntu.com/336498/

Revision history for this message
Scott Kitterman (kitterman) wrote :

Due to Partner, it's not entirely unreasonable.

Revision history for this message
Julian Edwards (julian-edwards) wrote :

This seems to have been caused by the recent refactoring of Build.createBuildQueueEntry(). It previously worked fine if two callsites were calling it around the same time, but now it does not since it has to manipulate three tables at once.

@Scott - none of the packages were partner packages or I would have realised that more quickly.

Revision history for this message
Julian Edwards (julian-edwards) wrote :

BTW I fixed the database and re-enabled the retry-depwait script so all the builds that were waiting to get retried should have gone through.

I expect it will fail again at some point though.

Revision history for this message
Scott Kitterman (kitterman) wrote : Re: [Bug 492632] Re: retry-depwait is falling over

Certainly. Just pointing out a case where that's valid. The bug cost me a
lot of time today and yet tonight baby sitting retries.

Revision history for this message
Scott Kitterman (kitterman) wrote :

BTW, glad to see it's restarted, that helped.

Revision history for this message
Julian Edwards (julian-edwards) wrote :

On Monday 07 December 2009 22:31:00 Scott Kitterman wrote:
> Certainly. Just pointing out a case where that's valid. The bug cost me a
> lot of time today and yet tonight baby sitting retries.

Right, sorry for the trouble, it was a nightmare to track down where the
problem was. I'll keep on top of it with manual fixes until we get it fixed.

Julian Edwards (julian-edwards)
Changed in soyuz:
milestone: 3.1.11 → 3.1.12
Revision history for this message
Julian Edwards (julian-edwards) wrote :

I'm cherry picking a tentative fix today to see if it helps.
https://code.edge.launchpad.net/~julian-edwards/launchpad/die-buildqueue-die--bug-492632/+merge/15927

Revision history for this message
Scott Kitterman (kitterman) wrote :

I've noticed a recent change that is problematic from my perspective. It
appears that pacakages that come out of depwait that have uninstallable
build-deps now FTBFS instead of returning to depwait.

Was this inetentional? If it was, I'd like to talk you out of it.

Revision history for this message
Julian Edwards (julian-edwards) wrote :

Definitely not intentional, I'll look into it.

Revision history for this message
Julian Edwards (julian-edwards) wrote :

Scott can you file a new bug about that on launchpad-buildd please. We think it's a problem with apt/dpkg on the buildds.

Revision history for this message
Scott Kitterman (kitterman) wrote :

Done. Bug 495564.

Revision history for this message
Julian Edwards (julian-edwards) wrote :

I haven't seen the problem re-occur for 2 days so marking fix-released. It was happening a couple of times a day previously.

Changed in soyuz:
status: Triaged → Fix Released
Revision history for this message
Julian Edwards (julian-edwards) wrote :

This is happening again, I don't know if it's the same bug. For reference, here's the SQL to fix it:
http://pastebin.ubuntu.com/414262/

Revision history for this message
Julian Edwards (julian-edwards) wrote :

And here's the broken builds SQL dump:
https://pastebin.canonical.com/30566/

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.