+project-cloud page timing out

https://pastebin.canonical.com/13489/ shows the output of EXPLAIN ANALYZE using roughly the same query on staging. The query takes only 0.4s -- much less than the 25s we are seeing on production.

https://pastebin.canonical.com/13490/ shows the same EXPLAIN call on production: less than 0.2s.

If I had to guess, I'd say we are being bitten by some sort of lock. I'd rather not guess though, and ask stub instead.

The queries are all fast.

The OOPS reports state a massive amount of non-SQL time.

Looking at how the timeouts are currently implemented, we have no way of knowing if this was genuinely non-SQL time, or time spent waiting for a lock and attempting that last SQL query.

I think canonical.launchpad.webapp.adapter.LaunchpadTimeoutTracer needs to be fixed, checking if get_remaining_time() is positive before even attempting to issue a query - this will tell us if time has been spent in Python for some reason rather than time spend waiting on the db.

Nothing should be blocking that query - the only operations that block a select we only do during rollouts.

Surely the timeout tracer wouldn't raise an error with a query that has already finished executing? Or are you suggesting that the query starts at roughly ~24s into the page request? (Or am I misunderstanding something)