queue builder is way too slow

Currently this really slows down distro work, sometimes leading to a delay of up to three hours between source upload and binaries hitting the archive mirrors.

Will address it as soon as the next soyuz release rollout finish

Let's try to use this smarter approach.

Is it helping?

We may go for a more 'package-centric' approach.
However, the proposed query might help to speed up the "overall checker" that we can run less often than hourly (maybe daily)

Pushing off to 1.1.8 and giving back to Celso for now -- he has a better idea of what's necessary, and I want to make sure the commercial-repo work gets done.

So my suggestion here is to avoid rescoring completely, and instead when deciding what build to process, ordering by the [original, process-upload defined] score and the build record's age.

If you want to preserve the behaviour where we ensure that high scores were given to old records that are for non-priority components, you can use the age in seconds and then fudge it with a divisor, and then add to the score, and order by that. For instance, using a divisor of 100 gives you 36 points per hour, so a build that is 10 hours old gets 360 points; if main's score is 1500 and universe's score is 500, that means that in 30 hours a universe build would be prioritized over a main build -- which I think is close to the behaviour we have today.

Removing the rescoring step, the queue builder becomes necessary only to build stuff which comes in through minority cases (as most build records are today created at upload time) -- for instance, new architectures, PAS changes, source syncs, and maybe another one or two cases I can't recall right now.

It can be run a lot less frequently if that's all that it's handling -- maybe once or twice a day.

I was with kiko right up until the "we can run it less frequently" bit. If we rewrite this flow to make sure new uploads always get scored when accepted, and that the queue-builder is only creating new records for P-a-s changes and the like, or rescoring based on age, it will end up so fast that we could run it every 5 minutes for all we care (obviously, every 15 or so is fine).

The basic idea of scoring at creation time (and only at creation time), and rescoring based on age is a good one, and I tend to like it. Note that we can also do a couple of things about the initial scoring to make it saner:

- Drop the available-deps check entirely. It's pointless, it becomes stale and inaccurate very soon after the record is created, and we have auto-dep-waiting on the buildd side specifically to deal with missing deps.
- Try to score in a similar fashion to dak/wanna-build, taking into account suite, component, priority, and section (libs first, then everything else), in that order.

If you guys want my input on this at any point during the development of it, please ask. This is something pretty near and dear to my hear, and my daily work.

On Wed, Aug 08, 2007 at 11:45:48AM -0000, Adam Conrad wrote:
> I was with kiko right up until the "we can run it less frequently" bit.

Because it is potentially expensive. We need to find out exactly how
expensive it will end up once we've done the changes suggested here, but
the essential problem is that doing the whole-archive check for records
that need creating is expensive.

> The basic idea of scoring at creation time (and only at creation time),
> and rescoring based on age is a good one, and I tend to like it. Note

We can dynamically "rescore" based on age; there is no need to actually
run a script to update the score. Fundamentally, this decision can be
made when dispatching builds -- it doesn't need to be persisted.

> - Drop the available-deps check entirely. It's pointless, it becomes stale and inaccurate very soon after the record is created, and we have auto-dep-waiting on the buildd side specifically to deal with missing deps.
> - Try to score in a similar fashion to dak/wanna-build, taking into account suite, component, priority, and section (libs first, then everything else), in that order.

That'd be cool and definitely something we could roll into this work.
--
Christian Robottom Reis | http://async.com.br/~kiko/ | [+55 16] 3376 0125

Short update:

 * RF 5242 allow builddmaster tasks to run in parallel (slave-scanner from sequencer and queue-builder & retry-depwait from cron).

Next step is to create and score (statically) build records synchronously for source-only uploads.

Confirmed that creating build records straight on package uploads, and having q-b just doing the rescoring based on age, etc. would be wonderful. This is currently a major cause of release engineering slowdown.

The remaining issue mentioned in this bug will fixed during the implementation of https://blueprints.edge.launchpad.net/soyuz/+spec/soyuz-buildd-improvements