want to know breakdown of test run time by area of development

Looks like the existing test runner can log most of the information we are interested in - I'm updating the makefile. Once we have the per test timeings raw data available, we can summarize it.

I've switched on the Z3 test runners timing mechanism. An initial score is that the main launchpad tests take 9 minutes to run, which indicates that:
a) The Z3 test runners timing mechanism isn't counting setup/teardown time
b) Most of our time is spent doing setup/teardown

As the Z3 functional framework is cached and cannot be torn down, the setup/teardown that is hurting us would be the postgresql fixture and to a lesser extent the librarian.

I'd already figured that out ;). The thing that I think steve was asking for was a list of the sections of the tests that take the most aggregate time. This could be estimated by multiplying the number of tests in each area by the ratio of uncounted time/total setup - but a better way is to actually get the aggregate time for each of the areas - not the 'in-test-method' time.

Robert Collins wrote:
> Public bug report changed:
> https://launchpad.net/malone/bugs/5814
>
> Comment:
> I'd already figured that out ;). The thing that I think steve was asking
> for was a list of the sections of the tests that take the most aggregate
> time. This could be estimated by multiplying the number of tests in each
> area by the ratio of uncounted time/total setup - but a better way is to
> actually get the aggregate time for each of the areas - not the 'in-
> test-method' time.

Yup. Just reporting where I got to ;)

--
Stuart Bishop <email address hidden> http://www.canonical.com/
Canonical Ltd. http://www.ubuntu.com/

Will attempt as part of the testsuite-optimization work.

Steve, is the current "Tests and runtime by layer" section of the 'PQM test report' satisfies your need?

After discussion with Steve, it would be beneficial to track the time spent tests with external dependencies like Soyuz or the libraries we use for bzr imports like cscvs.

It might be possible to group those tests in a new layer. Otherwise, probably define a test tag system.

I'll see if grouping these tests using a layer is possible, otherwise, I'll open a new bug about the 'tag' proposal.

I claim that the current report by layer is good enough. And if it ain't, we should file a more specific bug about what is needed.