DistroSeriesLanguage:+index pages timing out

Bug #327100 reported by Данило Шеган
4
Affects Status Importance Assigned to Milestone
Launchpad itself
Fix Released
High
Jeroen T. Vermeulen

Bug Description

pt_BR team seems to be refetching https://translations.edge.launchpad.net/ubuntu/jaunty/+lang/pt_BR/+index with batch size of 300 from a script quite often. A lot of time we are spending doing processing in Python (SQL time is usually pretty low), so this is something that might need some profiling.

Sample oopses: OOPS-1135EA1224, OOPS-1135EB947

Changed in rosetta:
importance: Undecided → High
status: New → Triaged
Revision history for this message
Jeroen T. Vermeulen (jtv) wrote :

That page fetches all POFiles for the given DistroSeries with the given Language, unsliced, with a prejoin. It doesn't take the database very long to produce all those rows, but it's a safe bet that it does take a while for all those Python objects to come out the other end of the ORM.

Why isn't the query being sliced? Because DistroSeriesLanguage.po_files_or_dummies puts the cart before the horse. It gets all (yes, all) POFiles for the given DistroSeries in the given language, then goes through all templates for the DistroSeries to identify any DummyPOFiles it needs to create, and finally it produces the combined list for the entire DistroSeries—of which the BatchNavigator then renders just one batch. We need to make the batch navigator navigate the POTemplates, then fetch the POFiles and create DummyPOFiles for just the batch we're showing.

After that it's mostly queries to Person. Bundling all of those into a single "id IN (...)" would help.

Revision history for this message
Jeroen T. Vermeulen (jtv) wrote :

I still felt like fixing this after work, so I've attached a branch that solves the problem. I've got higher priorities than pushing this through right now though, so if somebody else wants to pick it up, please let me know!

Changed in rosetta:
milestone: none → 2.2.4
Changed in rosetta:
milestone: 2.2.4 → 2.2.5
Changed in rosetta:
milestone: 2.2.5 → 2.2.6
Changed in rosetta:
milestone: 2.2.6 → 2.2.7
Changed in rosetta:
status: Triaged → In Progress
Changed in rosetta:
assignee: nobody → Jeroen T. Vermeulen (jtv)
Revision history for this message
Jeroen T. Vermeulen (jtv) wrote :

Fixed in devel 8878

Changed in rosetta:
status: In Progress → Fix Committed
Changed in rosetta:
status: Fix Committed → Fix Released
Revision history for this message
Robert Collins (lifeless) wrote :

sorry for the noise, oops tools is confused.

Changed in rosetta:
status: Fix Released → Triaged
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.