Test failures in devel break ec2test runs

Running against stable would mean not doing integration testing against the missing revisions. Which is most likely to happen?

Encountering a broken devel, encountering errors related to your changes or encountering integration problems?

As the bug title is stated, I'd mark it as Won't fix. Maybe what we are looking for is an option to ec2test to run it against --stable.

Personnally, when I encountered a similar situation, I compared and my failures with the one of the failed build. And submitted again once the [testfix] revision was landed without using ec2test (because I confirmed that all the failures were identical, and thus not related to my branch).

On Sat, Aug 29, 2009 at 6:59 AM, Francis J.
Lacoste<email address hidden> wrote:
> Running against stable would mean not doing integration testing against
> the missing revisions. Which is most likely to happen?
>
> Encountering a broken devel, encountering errors related to your changes
> or encountering integration problems?
>

The first one, in my experience.

> As the bug title is stated, I'd mark it as Won't fix.  Maybe what we are
> looking for is an option to ec2test to run it against --stable.
>
> Personnally, when I encountered a similar situation, I compared and my
> failures with the one of the failed build. And submitted again once the
> [testfix] revision was landed without using ec2test (because I confirmed
> that all the failures were identical, and thus not related to my
> branch).
>

But this is not ideal at all:
  - it's repeated work (you have to submit again)
  - it's extra thinking (you have to compare test results)
  - it's extra waiting (you have to wait for test fix)

I think none of this work is creating extra value for the customer,
particularly since the change is *guaranteed* to be tested against the
missing revisions by buildbot.

jml

Either add an option for running against stable, or make it the default and provide options for running against devel.

After spending a huge amount of time over the last week staring at and
resubmitting branches to land via ec2, I would like to second the
proposal to run against stable by default. I know I'm not the only one
feeling the pain this week.

Jono has said that there have been an unusual number of bad landings
to devel this week, so maybe this is an abberation. However, it has
lost me a lot of time. Cumulatively, across the whole team, each time
this kind of abberation happens, this loss of productivity could be
greater than the value of integration testing with the very latest
landings to devel. Testing with the very most recent landings to devel
also does not guarantee a successful devel -> stable merge.

I think an experiment to use stable in ec2 would be worthwhile. In
case of a bad landing or an integration failure, the merge from devel
to stable will stop the line soon enough, at which point anyone on the
team can fix it.

On Thu, Sep 23, 2010 at 9:56 PM, Gavin Panella
<email address hidden> wrote:
> After spending a huge amount of time over the last week staring at and
> resubmitting branches to land via ec2, I would like to second the
> proposal to run against stable by default. I know I'm not the only one
> feeling the pain this week.

I think this would be a mixed blessing. rather than a small degree of
breakage, it can easily cascade. That is, the failure mode is worse.

I'd love to do gary's multi-branch-landing with precommit tests
experiment. I think it has the most legs.

-Rob

On Thu, Sep 23, 2010 at 4:56 PM, Gavin Panella
<email address hidden> wrote:
> After spending a huge amount of time over the last week staring at and
> resubmitting branches to land via ec2, I would like to second the
> proposal to run against stable by default. I know I'm not the only one
> feeling the pain this week.
>
> Jono has said that there have been an unusual number of bad landings
> to devel this week, so maybe this is an abberation. However, it has
> lost me a lot of time. Cumulatively, across the whole team, each time
> this kind of abberation happens, this loss of productivity could be
> greater than the value of integration testing with the very latest
> landings to devel. Testing with the very most recent landings to devel
> also does not guarantee a successful devel -> stable merge.
>
> I think an experiment to use stable in ec2 would be worthwhile. In
> case of a bad landing or an integration failure, the merge from devel
> to stable will stop the line soon enough, at which point anyone on the
> team can fix it.

If you want to experiment, I think you can do this right now is you
use 'test' instead of 'land':

utilities/ec2 test -b launchpad=stable -s '[r=foo]. ..'

--
Stuart Bishop <email address hidden>
http://www.stuartbishop.net/

Stuart is right that you can experiment with this now.

We are close to starting the multi-branch-landing story that Robert mentioned (https://dev.launchpad.net/Foundations/Proposals/SimplifyMergeMachinery). Hopefully it will be done sometime in the upcoming LP release cycle.