db-devel librarian hung with '[Errno socket error] timed out'

Bug #682861 reported by Māris Fogels
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Launchpad itself
Invalid
Low
Unassigned

Bug Description

It looks like the librarian or librarian teardown hung itself with a socket timeout in build 436. The full log is attached.

Here is an excerpt of the log with the relevant information:

{{{
Traceback (most recent call last):
  ...
  File "/srv/buildbot/slaves/launchpad/lucid-db-devel/build/lib/canonical/testing/layers.py", line 861, in _check_and_reset
    "shutdown: " + str(e)
canonical.testing.layers.LayerIsolationError: Librarian has been killed or has hung. ... : [Errno socket error] timed out

...
WARNING: A test appears to be hung. There has been no output for 600 seconds.
Forcibly shutting down the test suite
...
The dying processes left behind the following output:
--------------- BEGIN OUTPUT ---------------

**********************************************************************
Could not communicate with subprocess
**********************************************************************

---------------- END OUTPUT ----------------

Tests failed (exit code -15)
}}}

Revision history for this message
Māris Fogels (mars) wrote :
Revision history for this message
Māris Fogels (mars) wrote :

The LOSAs have confirmed that the librarian pidfile was cleaned up properly after the run, so it wasn't a complete process layer crash.

Revision history for this message
Gary Poster (gary) wrote :

This seems potentially related to bug 671335.

Changed in launchpad-foundations:
status: New → Triaged
importance: Undecided → Medium
Revision history for this message
Graham Binns (gmb) wrote :

This happened again: https://lpbuildbot.canonical.com/builders/lucid_db_lp/builds/448/steps/shell_6/logs/summary. Three out of the last four db-devel builds failed because of this problem (unless I'm reading the waterfall incorrectly, which is quite possible).

Revision history for this message
Māris Fogels (mars) wrote : Re: [Bug 682861] Re: db-devel librarian hung with '[Errno socket error] timed out'

wgrant suspects the librarian fixtures changes in db-devel r10013 may
be to blame. He said the timing of these issues' first occurrence is
suspiciously close to that revision's landing.

Revision history for this message
Māris Fogels (mars) wrote :

From #launchpad-dev:

<wgrant> When did it start?
<wgrant> Around the time of the databasefixture landing?
<gmb> wgrant: The bug was filed on the 29th, FWIW:
https://bugs.edge.launchpad.net/launchpad-foundations/+bug/682861
<wgrant> That's, er, slightly suspicious timing. The databasefixture
stuff landed on the 28th.

and:

<wgrant> mars: It's probably the librarian fixture changes.
<wgrant> In db-devel r10013

Revision history for this message
Robert Collins (lifeless) wrote :

Consider landing my librarian branch then :)

Revision history for this message
Deryck Hodge (deryck) wrote :

Just FYI, Abel will be working on getting Robert's branch landed starting Monday.

Curtis Hovey (sinzui)
Changed in launchpad:
importance: Medium → Low
William Grant (wgrant)
Changed in launchpad:
status: Triaged → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.