code imports should run fewer jobs at once

I talked about this with Aaron a little, and we decided that passing the number of CPUs on the slave to the getJobForMachine XML-RPC call would be less work. "os.sysconf('SC_NPROCESSORS_ONLN')" is a way to find the number of CPUs.

Michael Hudson <email address hidden> writes:

> I talked about this with Aaron a little, and we decided that passing
> the number of CPUs on the slave to the getJobForMachine XML-RPC call
> would be less work. "os.sysconf('SC_NPROCESSORS_ONLN')" is a way to
> find the number of CPUs.

This will misrepresent the capacity of SMT boxes (as our current
generation of servers are) and for something like code import that's
probably bad (it's not the kind of work load which will benefit from
scaling with number of threads rather than cores).

That said, I'm afraid I don't know a particular clean way to detect
active SMT other than to check for duplicate core id's in /proc/cpuinfo.
(And bear in mind that older CPUs won't have a core ID in /proc/cpuinfo)

--
James

James Troup wrote:
> Michael Hudson <email address hidden> writes:
>
>> I talked about this with Aaron a little, and we decided that passing
>> the number of CPUs on the slave to the getJobForMachine XML-RPC call
>> would be less work. "os.sysconf('SC_NPROCESSORS_ONLN')" is a way to
>> find the number of CPUs.
>
> This will misrepresent the capacity of SMT boxes (as our current
> generation of servers are)

Bugger. I thought I checked this, but obviously got it wrong somehow.

> and for something like code import that's
> probably bad (it's not the kind of work load which will benefit from
> scaling with number of threads rather than cores).

Yeah/

> That said, I'm afraid I don't know a particular clean way to detect
> active SMT other than to check for duplicate core id's in /proc/cpuinfo.
> (And bear in mind that older CPUs won't have a core ID in /proc/cpuinfo)

That sounds fairly horrible. Maybe CodeImportMachine.cpu_count is
easier then.

Cheers,
mwh

Will this still be an issue if we have incremental imports?

I can't think of a reason why incremental imports would change this. The issue is that these imports are CPU-intensive. It doesn't matter whether they run for a short or long time-- you don't want to run multiple imports in parallel on a single core.