Bazaar

Bug #582157
Comment #14

Comment 14 for bug 582157

Revision history for this message

Martin Pool (mbp) wrote on 2010-06-25:

#14

Robert and I studied this a bit today, by pulling from chinstrap which is a machine on the network pretty close to Launchpad. Amongst other things we watched the socket buffer size on both sides, and the strace of read/write system calls on both sides. We're using external SSH on both sides so the socket is not directly controlled by bzr.

What we observed is that the server's send buffer is often filled with quite large chunks (1MB) which gradually drain out, then there's a period where the server is not writing anything out. On the client, the recv buffer is almost always empty: SSH and bzr are accepting data as fast as it comes in.

This was in the context of the proposed merge for bug 590638. I didn't see conclusive proof that patch will make things faster, but I'm more convinced that it will help in some cases where there are small parts in the stream and where the sender is choppy. Without this patch, we leave some data in the application buffer so when we do start writing, the OS has a bit more data to send that could have been sent earlier.

There seem to be two independent problems then:
0- There are multi-second periods where the bzr server process is cpu or disk bound and not writing out anything. Because these pauses are relatively long and happen near the start of the connection there's not enough data already buffered to keep the network full, so it goes idle.

1- When the server process has provided a lot of data, the server sends it relatively slowly. The server OS has >1MB of data, the client advises it has a receive window over 500kB, and yet the server sends at only somewhere around 300kB/s.

We see a similar rate fetching a large random file over ssh or https. If we try multiple simultaneous connections the measurement is noisy but it seems like they each reach about this rate: at any rate the same bandwidth is not being divided between them. So this does look like the server is limiting the rate at which it transmits.

Robert and I studied this a bit today, by pulling from chinstrap which is a machine on the network pretty close to Launchpad.  Amongst other things we watched the socket buffer size on both sides, and the strace of read/write system calls on both sides.  We're using external SSH on both sides so the socket is not directly controlled by bzr.

What we observed is that the server's send buffer is often filled with quite large chunks (1MB) which gradually drain out, then there's a period where the server is not writing anything out.  On the client, the recv buffer is almost always empty: SSH and bzr are accepting data as fast as it comes in.

This was in the context of the proposed merge for bug 590638.  I didn't see conclusive proof that patch will make things faster, but I'm more convinced that it will help in some cases where there are small parts in the stream and where the sender is choppy.  Without this patch, we leave some data in the application buffer so when we do start writing, the OS has a bit more data to send that could have been sent earlier.

There seem to be two independent problems then:
 0- There are multi-second periods where the bzr server process is cpu or disk bound and not writing out anything.  Because these pauses are relatively long and happen near the start of the connection there's not enough data already buffered to keep the network full, so it goes idle.

1-  When the server process has provided a lot of data, the server sends it relatively slowly.  The server OS has >1MB of data, the client advises it has a receive window over 500kB, and yet the server sends at only somewhere around 300kB/s.

We see a similar rate fetching a large random file over ssh or https.  If we try multiple simultaneous connections the measurement is noisy but it seems like they each reach about this rate: at any rate the same bandwidth is not being divided between them.  So this does look like the server is limiting the rate at which it transmits.