staging is overloaded

Staging is being routinely overloaded:
https://lpstats.canonical.com/graphs/AsukaLoadAverage/20100907/20100908/

Some causes are the check-teamparticipation (I filed a bug)

13:46 < spm> 15900 launchpa 20 0 388m 126m 8180 R 98 3.3 0:05.00 /usr/bin/python /srv/staging.launchpad.net/staging/launchpad/cronscripts/process-apport-blobs.py
13:46 < spm> 15901 bzrsyncd 20 0 279m 88m 6284 R 24 2.3 0:03.55 /usr/bin/python /srv/bzrsyncd.staging.launchpad.net/staging/launchpad/cronscripts/merge-proposal-jobs.py -vv
13:46 < spm> 15914 bzrsyncd 20 0 350m 101m 7968 R 24 2.6 0:04.01 /usr/bin/python /srv/bzrsyncd.staging.launchpad.net/staging/launchpad/cronscripts/sendbranchmail.py
13:46 < spm> 15906 bzrsyncd 20 0 279m 89m 6284 R 23 2.3 0:03.28 /usr/bin/python /srv/bzrsyncd.staging.launchpad.net/staging/launchpad/cronscripts/scan_branches.py
13:46 < spm> 15929 launchpa 20 0 376m 118m 8092 R 22 3.1 0:05.06 /usr/bin/python /srv/staging.launchpad.net/staging/launchpad/scripts/cache-country-mirrors.py -q /srv/mirrors.staging.launchpad.net/w
13:46 < spm> 9052 launchpa 20 0 748m 392m 3440 R 7 10.2 22:35.15 /usr/bin/python -S ./foaf-update-karma-cache.py

(the column with 98 on the top is CPU)

We can't QA performance changes to the appservers effectively while we're overloading the system: some noise is expected, but saturated throws a more than 100% delay into things, which makes ok things timeout and as a knockon effect can waste a lot of time analysing perfectly ok code.

We'll probably want to add RT's to fix this all up, and perhaps we need a separate box for these tasks vs the staging appserver instance(s)