Many tests use unnecessarily expensive layers

personally, I think we should switch to testresources.

jml and I had a call about what would be involved in switching to testresources. It's quite a bit.

 - we need to do something to do with resources that can't be cleaned up. The zope test runner runs these in subprocesses.
 - we need to think what to do about the layers that have testSetUp and testTearDown methods. possibly resources that are always dirty?
 - similarly, some of our layers check invariants. how do we do that with testresources?
 - zope.testing at least used to ignore any special suite you specified for your tests, which would mean testresources' OptimizingTestSuite would be useless during a transition.
 - we need to define the resources launchpad tests need and come up with a transition strategy.

It's probably also worth enumerating the advantages of using testresources:

 - the main one is that it will make our tests easier to understand, the requirements of a test will be more explicit.
 - we can also more easily use other test runners, which will probably make projects like parallelizing the test run easier.
 - running a small number of tests locally will often be quicker, as test fixtures will be more minimal.

It's not totally clear that it's worth it.

On Wed, Sep 9, 2009 at 10:26 AM, Michael
Hudson<email address hidden> wrote:
> testresources' OptimizingTestSuite would be useless during a transition.

"OptimisingTestSuite"

I don't feel competent to set a priority, but I would say high (because medium is below the fold and this should get done).

See Michael's comment -- it's not clear what the benefits actually are.

@jml the benefits listed:
---
It's probably also worth enumerating the advantages of using testresources:

 - the main one is that it will make our tests easier to understand, the requirements of a test will be more explicit.
 - we can also more easily use other test runners, which will probably make projects like parallelizing the test run easier.
 - running a small number of tests locally will often be quicker, as test fixtures will be more minimal.
---

Are pretty clear to me:
 - clearer test dependencies
 - easier use of other test runners (such as trial, tribunal, or [say, bzr-selftest --parallel=ec2])
 - faster runs of select tests by better dependency management

@mwhudson
 - we need to do something to do with resources that can't be cleaned up. The zope test runner runs these in subprocesses.

You could thunk into subunit for those. Alternatively we could just clean them up - 'cannot' is a very strong statement.

 - we need to think what to do about the layers that have testSetUp and testTearDown methods. possibly resources that are always dirty?

Yes, that would work, or perhaps child resources that go dirty and the parent doesn't.

 - similarly, some of our layers check invariants. how do we do that with testresources?

Point me at some code, I'll have a think.

 - zope.testing at least used to ignore any special suite you specified for your tests, which would mean testresources' OptimizingTestSuite would be useless during a transition.

It rearranges things as it pulls the layers out. I have in my head an adapter for testresources to honour layers-as-resources, which would permit nuking zope.testing's layer support on day one.

 - we need to define the resources launchpad tests need and come up with a transition strategy.

I think an incremental approach might be easier to deploy.

On Thu, Dec 3, 2009 at 11:51 AM, Robert Collins
<email address hidden> wrote:
...
>  - similarly, some of our layers check invariants. how do we do that
> with testresources?
>
> Point me at some code, I'll have a think.
>

lib/canonical/testing/layers.py

http://bazaar.launchpad.net/~launchpad-pqm/launchpad/db-devel/annotate/head:/lib/canonical/testing/layers.py

>  - zope.testing at least used to ignore any special suite you specified
> for your tests, which would mean testresources' OptimizingTestSuite
> would be useless during a transition.
>
> It rearranges things as it pulls the layers out. I have in my head an
> adapter for testresources to honour layers-as-resources, which would
> permit nuking zope.testing's layer support on day one.
>

That'd be aces.

>  - we need to define the resources launchpad tests need and come up with
> a transition strategy.
>
> I think an incremental approach might be easier to deploy.
>

Well, yes. That's what the comments are getting at. Could you please
expand on what you mean here?

jml