rebuilding replicas causes all background tasks to stall (no differentiation between latency sensitive tasks and others)

This is by design. When a slave is being rebuilt, an open transaction is kept open for several hours. We don't want data changes being made during this window because of database bloat. We don't want scripts that do bulk changes running when there are long running transactions open.

If it is critical things never block, don't use DBLoopTuner. Its purpose is to block when the DB is unhappy. Use LoopTuner instead.

If we want things to block, but not so aggressively, it needs to be opt in. We could either add flags to DBLoopTuner to not perform some checks, or alternative implementations that strike a different balance between remaining responsive and keeping the database happy.

We can do this right now by setting long_running_transaction to None.

As we discussed on IRC, even setting long_running_transaction to None won't solve everything because we are still reading the cluster lag. When slave is being rebuilt, this is going to grow out of hand as well and any scripts running as part of DBLoopTuner would block. Re-opening.

So long_running_transaction can be used to tune or turn off the long running transaction checks.

acceptable_replication_lag can be used to tune the cluster lag checks.

If we want to avoid blocking on a new node rebuild, we need to change the lag check from cluster lag to (all nodes in the cluster that are not currently being built) lag.

With Slony, there is no way of telling which nodes are currently being built. We would have to store this state somewhere and ensure it is updated somehow. This seems overkill, as we normally only build a new node once per month.

I think its worth figuring out and doing because even though its monthly it is very disruptive; in particular it makes opening new distroreleases just after a db deploy a problem, which we don't need.

Danilo suggested allowing things to continue if lag is 'too high'. I've thought about this, and it seems fine as the only time lag is 'too high' is when alerts have been raised and people are having to fix things, or we are in a known laggy situation such as rebuilding replicas.

So i think we can change the loop tuner to only block if replication lag is between 5 minutes and 45 minutes.

So, we no longer rebuild replicas every month; just when we add nodes which is very rare.

That said, I think we can discriminate between 'big scanning jobs that should respect lag' and 'lively maintenance' that should ignore lag.