When timeouts occur, they can be caused by a) inefficient code or b) external influences.
We should gather enough data that we don't spend time debugging the wrong things.
Specifically we should gather:
- system load average
- number of cpucores (to normalise the load average)
- process memory & physical memory (to guesstimate whether we're hitting swap)
- *process* time since the request started. As each request is in a separate thread, the OS's system accounting can tell us whether 5 seconds of wall clock time was 5 seconds of CPU time, or 1 second of CPU time.
The canonical.mem.resident() and canonical.mem.memory() will help in implementing this. os.loadavg will give us load averages. We can grep /proc/cpu as bzr does for the cpu counts, and time.clock() will give us CPU usage ('per process', which may be equivalent to per-thread when we use it from a non main thread. Testing will be needed). If time.clock does not suffice, a small extension could call clock_gettime(CLOCK_THREAD_CPUTIME_ID, ....)
We are hitting many questions we cannot answer today as a result of not knowing these things.
Alternatively:
#RUSAGE_THREAD = 1 on my linux system - we'd want a C extension to get the right constant
resource.getrusage(1)ru_utime
should give us what we need.
AIUI Francis' team is in the best position to actually store this information, and he already has put work into capturing this data into data structures we can output in the OOPS dump.