Scan branches immediately after pulling them

Blocked on other bugs -- I'm not certain on the numbers. Before we can do this, we have to make sure that the scanner isn't generating diffs.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jonathan Lange wrote:
> Blocked on other bugs -- I'm not certain on the numbers. Before we can
> do this, we have to make sure that the scanner isn't generating diffs.

Why aren't you sure the scanner isn't generating diffs?

Aaron
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkmlYO0ACgkQ0F+nu1YWqI1DcwCeLShE8e3IDuHlFuzgSMuLiikK
e4sAniDDXnSO2MBvZrp3FjuhZplDuu3v
=oavY
-----END PGP SIGNATURE-----

On Thu, Feb 26, 2009 at 2:17 AM, Aaron Bentley <email address hidden> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Jonathan Lange wrote:
>> Blocked on other bugs -- I'm not certain on the numbers. Before we can
>> do this, we have to make sure that the scanner isn't generating diffs.
>
> Why aren't you sure the scanner isn't generating diffs?
>

That's been the blocker historically -- I'm trying to get this stuff
written down.

Are you aware of any blockers now?

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jonathan Lange wrote:
> On Thu, Feb 26, 2009 at 2:17 AM, Aaron Bentley <email address hidden> wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Jonathan Lange wrote:
>>> Blocked on other bugs -- I'm not certain on the numbers. Before we can
>>> do this, we have to make sure that the scanner isn't generating diffs.
>> Why aren't you sure the scanner isn't generating diffs?
>>
>
> That's been the blocker historically -- I'm trying to get this stuff
> written down.

So, I did the work to split diffs out a while back. It's done. It's
landed. Did I fail to communicate that? Or do you think I did it wrong?

> Are you aware of any blockers now?

We're generating logs with -v, and that means doing almost as much work
as diff does. That's what my other branch is meant to address.

Aaron
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkmlx0gACgkQ0F+nu1YWqI0sawCeKCnhCedImuLQqoN+YKquyKUc
jjUAmwa07I+O+4Nr5InOfb9frO73lL5j
=77PD
-----END PGP SIGNATURE-----

On Thu, Feb 26, 2009 at 9:33 AM, Aaron Bentley <email address hidden> wrote:
>
> Jonathan Lange wrote:
>> On Thu, Feb 26, 2009 at 2:17 AM, Aaron Bentley <email address hidden> wrote:
>>>
>>> Jonathan Lange wrote:
>>>> Blocked on other bugs -- I'm not certain on the numbers. Before we can
>>>> do this, we have to make sure that the scanner isn't generating diffs.
>>> Why aren't you sure the scanner isn't generating diffs?
>>>
>>
>> That's been the blocker historically -- I'm trying to get this stuff
>> written down.
>
> So, I did the work to split diffs out a while back.  It's done.  It's
> landed.  Did I fail to communicate that?  Or do you think I did it wrong?
>

Or I could have failed to listen properly.

>> Are you aware of any blockers now?
>
> We're generating logs with -v, and that means doing almost as much work
> as diff does.  That's what my other branch is meant to address.
>

OK, that's good to know. Please comment on this bug when that's landed.

... although I guess we'll probably have to wait until it's released
before we can start moving the scanner around.

jml

We don't get oopses because of this any more.

Oh, I misunderstood, we sort of do, but I'm not sure it should be tagged oops in any case.

This bug is a little confused (possible implementation details mixed in with problem statements).

AIUI the issues are twofold:
 - high latency on scanning
 - oopses when a pull succeeds but the branch can't be scanned correctly

For the former, amqp can help a lot (and with parallelism and just-in-time-scaling too). For the latter, I don't see how the situation would change - if pulling succeeds but scanning fails, we need to issue an error somehow. We could just write that to the branch status field today.

I'm downgrading to low because the gap between pull and scan isn't actually dire today, and improving it isn't on our short term road map. Patches gratefully considered, of course.

Branch scan latency is now sub-second due to celery.