Launchpad itself

during sustained overload situation replies are truncated in production

Bug #636713 reported by Robert Collins
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Launchpad itself
Won't Fix
High
Unassigned

Bug Description

See the 2010-09-13-LP-partial%20pages incident report for gory situational details.

We seem to have a bug where:
 - when LP is kept 100% busy (that is, 4 pending requests and 4 active ones, at all times)
 - it is running behind HAproxy and apache
 - clients will get requests (e.g. +activereviews, WADL, merge proposals) truncated.

One user (wgrant) reported the truncation as almost always being at around 12K of data.

Revision history for this message
William Grant (wgrant) wrote :

Here are some example errors from one of my API scripts. Some for WADL, others for collections. The HTTP response headers for the 1.2MB WADL are about 350 bytes.

httplib.IncompleteRead: IncompleteRead(15646 bytes read, 1272188 more expected)
httplib.IncompleteRead: IncompleteRead(15710 bytes read, 39088 more expected)
httplib.IncompleteRead: IncompleteRead(15710 bytes read, 35557 more expected)
httplib.IncompleteRead: IncompleteRead(15710 bytes read, 38408 more expected)
httplib.IncompleteRead: IncompleteRead(15646 bytes read, 1272188 more expected)
httplib.IncompleteRead: IncompleteRead(15646 bytes read, 1272188 more expected)

Gary Poster (gary)
Changed in launchpad-foundations:
status: New → Triaged
importance: Undecided → High
Revision history for this message
Gary Poster (gary) wrote :

See also bug 702134

Revision history for this message
Robert Collins (lifeless) wrote :

We've changed our production deployment to avoid overloading zope appservers. If this happens ever again we'll diagnose (but be pretty confident its an haproxy/apache issue at that point).

Changed in launchpad:
status: Triaged → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.