Add option to run puller without subprocesses
Bug #153778 reported by
Jonathan Lange
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| Launchpad itself |
Won't Fix
|
Medium
|
Unassigned | ||
Bug Description
The recent switch to run the branch puller in multiple subprocesses is (probably) a pessimization for HOSTED branches, which (probably) won't benefit from running in parallel.
If we aren't going to run in parallel, then doing the pulling in subprocesses only adds overhead.
Changing the code isn't trivial, but should be fairly straightforward.
Revision history for this message
| David Allouche (ddaa) wrote | #1 |
| Changed in launchpad-bazaar: | |
| importance: | Undecided → Medium |
| status: | New → Confirmed |
Revision history for this message
| David Allouche (ddaa) wrote | #2 |
| visibility: | private → public |
Revision history for this message
| Jonathan Lange (jml) wrote | #3 |
| Changed in launchpad: | |
| status: | Triaged → Won't Fix |
To post a comment you must log in.
Agreed.
Pulling hosted branches is disk-bound, increasing parallelism is not a significant win, and is even likely to be a loss.
It is also very likely that the overhead of spawning subprocesses (in particular, loading the zcml takes whole seconds) will dwarf the actual branch-pulling cost and cause the task to be cpu-bound for no good reason.
On the other hand, using subprocesses has benefits: it reduces the cost of unhandled exceptions that cause a single-process branch-puller to exit prematurely.
Ideally, we should be able to use different amounts of parallelism for different tasks (high paralellism for mirror branches, low parallelism for import and hosted), and to turn subprocess-workers on and off on a per task basis (naturally, turning it off would turn off parallelism too), so we have the greatest flexibility in tuning the system.