Comment 32 for bug 380504

On Wed, 31 Mar 2010 15:42:51 -0000, Gary Poster <email address hidden> wrote:
> The following is in regards to the underlying problem of the frequent
> 502 errors, rather than the lazr.restfulclient changes we've been
> discussing, of which I am still in favor.
>
> James offered yesterday to set up a dedicated launchpadlib instance to
> catch the errors if I could tell him what logs to capture. I had in
> mind the logs that Leonard had suggested in comment #2, but I wanted to
> verify that these would be useful, so I said I'd get back to him.
>
> I then talked with Francis about his earlier investigation described in
> comment #10, and what we might be able to investigate now. He proposed
> a reasonable hypothesis, which we could test.
>
> HAProxy will queue requests up to 30 seconds, and send no more than 8 at
> once to the appservers. Busy appservers will mean that HAProxy may drop
> requests. This could be the cause of the 502 errors.

Yes, I very much think that is one of the causes. I have also seen it
sending the offline.html response.

> Things to test for this hypothesis that come to mind:
> - does a 502 typically indicate HAProxy complaining, and 503 indicate Apache complaining? Can we find logs for HAProxy and Apache that correlate errors there with errors reported in the clients?
> - when clients get 502s and 503s, are they typically accompanied with request times of >30 seconds?
>
> Therefore, for client-side logs for this problem, I'd be inclined to get
> logs of what Leonard requests (which is what wgrant shows in Comment #4,
> I think) along with logs of how long the request took (which I do not
> see in that example output). Is that reasonable, and reasonably easy to
> gather?

Yes, pretty simple.

Do we also want to know which server we are talking to as well? That
could be trickier to get hold of. We could just point at an IP address
instead of a hostname to narrow this down if the arrangement is such
that this would help.

Thanks,

James