Looking into this. The algorithm called "exponential backoff" was designed with a random element to avoid collisions, which isn't really the point here--when we 502 it's not because we got a lot of simultaneous hits. But I do like the idea of waiting for 2 seconds, 4 seconds, etc. and timing out for real after 16 or 32 seconds, and that's what I mean by "exponential backoff" here.
However, if we had real data about how long these 502 episodes last, 1/4 of that time would be a better starting point than 2 seconds. We could time out after waiting a total of 2x the length of a typical episode.
Looking into this. The algorithm called "exponential backoff" was designed with a random element to avoid collisions, which isn't really the point here--when we 502 it's not because we got a lot of simultaneous hits. But I do like the idea of waiting for 2 seconds, 4 seconds, etc. and timing out for real after 16 or 32 seconds, and that's what I mean by "exponential backoff" here.
However, if we had real data about how long these 502 episodes last, 1/4 of that time would be a better starting point than 2 seconds. We could time out after waiting a total of 2x the length of a typical episode.