Codehosting server used a lot of memory and not releasing "old" processes

31*32 is only a gig, which is a bit excessive but where was all the other memory going?

The processes hanging around issue is semi-known, but I don't think understood. At least, *I* don't understand it.

Rsync backups were taking up some memory, but I think 31*31MB including that many stale processes is enough to worry about on it's own.

I didn't know about the old processes issue. What do we know about it? Do we know what's causing it or whether it's reproducible locally?

Oh. It seems that 'bzr lp-serve' process gradually accumulate on the server, long after it seems likely that a client is interested.

Poking around now, it seems that a client is still connected, though, at least according to netstat. I guess if any traffic was sent down the socket we'd get a connection reset by peer error or something.

Maybe we should have an inactivity timeout in the ssh server?

On Fri, Aug 29, 2008 at 7:51 AM, Michael Hudson
<email address hidden> wrote:
> Oh. It seems that 'bzr lp-serve' process gradually accumulate on the
> server, long after it seems likely that a client is interested.
>
> Poking around now, it seems that a client is still connected, though, at
> least according to netstat. I guess if any traffic was sent down the
> socket we'd get a connection reset by peer error or something.
>
> Maybe we should have an inactivity timeout in the ssh server?
>

A timeout is a great idea. I still want to know the root cause, though.

jml

Happening again (28 stale processes this time)

It'd be good to get pystack reports from gdb the next time this happens. At least we'll get a clue as to what the code thinks it's doing.

Can we try and replicate this on staging. I'm not sure diving right into using gdb on the production server is the best way to go here.

On Sun, Sep 14, 2008 at 6:37 AM, Tom Haddon <email address hidden> wrote:
> Can we try and replicate this on staging. I'm not sure diving right into
> using gdb on the production server is the best way to go here.
>

Sure we can try. It will simply take much longer to replicate the
problem, since we're still unaware of what triggers it.

jml

So all these lp-serve processes seem to correspond to an ESTABLISHED connection in netstat. It seems very likely that the connections are in fact dead, but as we're not trying to send any data to the client, noone notices.

Can you set SO_KEEPALIVE on the server end of a socket?

I think I have a fix in progress for this. No idea how to test its effectiveness without cowboying it on to production for a few days though ...

So there's a chance that the server move fixed this by accident.

If not, I'd like to test my patch at some point...

Moving to .12

ping. Even with the codehosting box upgraded to 8Gb of memory, this continues to be an issue...

On Wed, Feb 11, 2009 at 9:26 PM, James Troup <email address hidden> wrote:
> ping. Even with the codehosting box upgraded to 8Gb of memory, this
> continues to be an issue...
>

Yeah, sorry. If could choose what I worked on, this would be it.

Michael's working on codebounce at the moment & I'm flat out on source
packages. Once I finish the branch I'm working on now, I'll fight for
the time to work on this (if no one else has got to it by then).

So this morning I got an unexpected clue as to what might be causing this: I had an ssh process connected to bazaar.launchpad.net from yesterday -- and its parent was the bzr-notify process. Now, this may be bug #335180, or just the fact that bzr-notify doesn't run the cycle collector very often, but if this affects more than 25% of Canonical employees even, it could be responsible for a large enough number of apparently stale processes on the server.

I've just sent off to ec2test a change that will disconnect connections that are idle (no traffic in either direction) for more than an hour, which will hopefully kill the problem off once and for all.

Is this happening again? See https://pastebin.canonical.com/25181/

I did a simple experiment locally.

Make sure I had the latest db-stable
 * Pull the latest db-stable (bzr pull)
 * Update download-cache (bzr up download-cache)
 * Update sourcecode (./utilities/update-sourcecode ~/src/launchpad/devel)

Change the codehosting timeout to 30 seconds:
 * Edit configs/development/launchpad-lazr.conf
 * Add "idle_timeout: 30" to the bottom of the codehosting clause

Make a tool that will do a Bazaar request and then wait:
 * bzr branch lp:bzr-ping ~/.bazaar/plugins/ping
 * Edit ~/.bazaar/plugins/ping/__init__.py, putting "import pdb; pdb.set_trace()" at the bottom of run()

Start the server:
 * make schema
 * make run_all

In a different terminal:
 * Make a user (./utilities/make-lp-user jml)
 * Simulate an idle connection (bzr ping bzr+ssh://bazaar.launchpad.dev)

Watch what happens:
 * Check that the spawned bzr process is still running (ps ax | grep lp-serve)
 * Repeat

30 seconds later, the process no longer appears in the 'ps' output, and the pdb window looks like this:

$ bzr ping bzr+ssh://bazaar.launchpad.dev
Response: ('ok', '2')
--Return--
> /home/jml/.bazaar/plugins/ping/__init__.py(47)run()->None
-> import pdb; pdb.set_trace()
(Pdb) Connection to launchpad.dev closed by remote host.