1.9rc1 can't push to Launchpad

You are pushing via bzr+ssh, I assume?

Did you get a backtrace in ~/.bzr.log when you killed it? If not, can you try again and use SIGQUIT (hit Ctrl-\) to interrupt it when the memory consumption gets high and then type "bt" at the pdb prompt to get a backtrace.

On Tuesday 04 November 2008 22:43:48 Andrew Bennetts wrote:
> You are pushing via bzr+ssh, I assume?

Yep, sorry I should have mentioned that.

> Did you get a backtrace in ~/.bzr.log when you killed it?

Yes:

Using saved push location: lp:~julian-edwards/launchpad/api_copy_packages
/ [===========
] Transferring 0/4^bzr: ERROR: bzrlib.errors.TooManyConcurrentRequests: The
medium '<bzrlib.smart.medium.SmartSSHClientMedium object at 0x3236b50>' has
reached its concurrent request limit. Be sure to finish_writing and
finish_reading on the currently open request.

Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 893, in
run_bzr_catch_errors
    return run_bzr(argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 839, in
run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 539, in
run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line 913, in run
    use_existing_dir=use_existing_dir)
  File "/usr/lib/python2.5/site-packages/bzrlib/push.py", line 162, in
_show_push_branch
    stop_revision=revision_id)
  File "/usr/lib/python2.5/site-packages/bzrlib/decorators.py", line 138, in
read_locked
    result = unbound(self, *args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/branch.py", line 1769, in push
    _override_hook_source_branch=_override_hook_source_branch)
  File "/usr/lib/python2.5/site-packages/bzrlib/branch.py", line 2495, in
_run_with_write_locked_target
    result = callable(*args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/branch.py", line 1808, in
_push_with_bound_branches
    result = self._basic_push(target, overwrite, stop_revision)
  File "/usr/lib/python2.5/site-packages/bzrlib/branch.py", line 1831, in
_basic_push
    graph=graph)
  File "/usr/lib/python2.5/site-packages/bzrlib/decorators.py", line 192, in
write_locked
    result = unbound(self, *args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/remote.py", line 1747, in
update_revisions
    self.fetch(other, stop_revision) ...

Hmm, I can't reproduce your problem. I have found another problem, but it doesn't seem to be this severe, and it is unchanged between 1.8 and 1.9rc1.

Unfortunately that traceback isn't very helpful either, because it's showing an error during the abort, rather than where it was when it was aborted :(

Can you repeat again with -Dhpss, and attach the .bzr.log? Feel free to interrupt it before it consumes all your memory! The -Dhpss output in ~/.bzr.log should give a fairly good indication of where it got up to when it reached that point.

Well I can't re-create this now either :/

The only difference is that I am using 1.9rc1-1 now as that is what I upgraded
to just now to test this out. However, I know I was not the only one with the
issue yesterday, so it might be some particular data that triggers the bug
rather than all pushes.

However, I did get a new error when pushing up a change to an existing branch,
I don't know if this is related:

UnknownErrorFromSmartServer: Server sent an unexpected error: ('error',
'<ProtocolError for xmlrpc.lp.internal:8097/branchfilesystem: -1 >')

Full traceback attached - sorry I was not using -Dhpss at the time, I can re-
do that if you want but I need to break a lock first :)

BTW, I'll carry on using 1.9rc1-1 and let you know as soon as the problem
happens again, if it does.

I also encountered this problem.

I have the pdb backtrace output. I also ran this with -Dhpss and can give the log. Backtrace below, log attached.

gary@gary-ubuntu:~/canonical/lp-branches/bug288064$ bzr push -Dhpss
Using saved push location: lp:~garyposter/launchpad/bug288064
/ [========== ] Transferring 0/4^\** SIGQUIT received, entering debugger
** Type 'c' to continue or 'q' to stop the process
** Or SIGQUIT again to quit (and possibly dump core)
> /usr/lib/python2.5/site-packages/bzrlib/breakin.py(33)_debug()
-> signal.signal(signal.SIGQUIT, _debug)
(Pdb) bt
  /usr/bin/bzr(130)<module>()
-> exit_val = bzrlib.commands.main(sys.argv)
  /usr/lib/python2.5/site-packages/bzrlib/commands.py(884)main()
-> ret = run_bzr_catch_errors(argv)
  /usr/lib/python2.5/site-packages/bzrlib/commands.py(893)run_bzr_catch_errors()
-> return run_bzr(argv)
  /usr/lib/python2.5/site-packages/bzrlib/commands.py(839)run_bzr()
-> ret = run(*run_argv)
  /usr/lib/python2.5/site-packages/bzrlib/commands.py(539)run_argv_aliases()
-> return self.run(**all_cmd_args)
  /usr/lib/python2.5/site-packages/bzrlib/builtins.py(913)run()
-> use_existing_dir=use_existing_dir)
  /usr/lib/python2.5/site-packages/bzrlib/push.py(162)_show_push_branch()
-> stop_revision=revision_id)
  /usr/lib/python2.5/site-packages/bzrlib/decorators.py(138)read_locked()
-> result = unbound(self, *args, **kwargs)
  /usr/lib/python2.5/site-packages/bzrlib/branch.py(1769)push()
-> _override_hook_source_branch=_override_hook_source_branch)
  /usr/lib/python2.5/site-packages/bzrlib/branch.py(2495)_run_with_write_locked_target()
-> result = callable(*args, **kwargs)
  /usr/lib/python2.5/site-packages/bzrlib/branch.py(1808)_push_with_bound_branches()
-> result = self._basic_push(target, overwrite, stop_revision)
  /usr/lib/python2.5/site-packages/bzrlib/branch.py(1831)_basic_push()
-> graph=graph)
  /usr/lib/python2.5/site-packages/bzrlib/decorators.py(192)write_locked()
-> result = unbound(self, *args, **kwargs)
  /usr/lib/python2.5/site-packages/bzrlib/remote.py(1747)update_revisions()
-> self.fetch(other, stop_revision)
  /usr/lib/python2.5/site-packages/bzrlib/decorators.py(192)write_locked()
-> result = unbound(self, *args, **kwargs)
  /usr/lib/python2.5/site-packages/bzrlib/branch.py(285)fetch()
-> pb=nested_pb)
  /usr/lib/python2.5/site-packages/bzrlib/remote.py(824)fetch()
-> return inter.fetch(revision_id=revision_id, pb=pb, find_ghosts=find_ghosts)
  /usr/lib/python2.5/site-packages/bzrlib/decorators.py(192)write_locked()
-> result = unbound(self, *args, **kwargs)
  /usr/lib/python2.5/site-packages/bzrlib/repository.py(2811)fetch()
-> pb, find_ghosts)
  /usr/lib/python2.5/site-packages/bzrlib/fetch.py(110)__init__()
-> self.__fetch()
  /usr/lib/python2.5/site-packages/bzrlib/fetch.py(140)__fetch()
-> self._fetch_everything_for_search(search, pp)
  /usr/lib/python2.5/site-packages/bzrlib/fetch.py(183)_fetch_everything_for_search()
-> not self.to_repository._fetch_uses_deltas))
  /usr/lib/python2.5/site-packages/bzrlib/knit.py(1357)insert_record_stream()
-> [(record.key, len(bytes))], bytes)[0]
  /usr/lib/python2.5/site-packages/bzrlib/knit.py(2407)...

Julian Edwards wrote:
[...]
> However, I did get a new error when pushing up a change to an existing branch,
> I don't know if this is related:
>
> UnknownErrorFromSmartServer: Server sent an unexpected error: ('error',
> '<ProtocolError for xmlrpc.lp.internal:8097/branchfilesystem: -1 >')

That one is a launchpad-bazaar bug. I'm not sure if it's related or not yet
(although my guess is not), so file a new bug for it to track it.

Ok, so being very slow is probably just bug 294479. (If you're keen, you could try the patch there, or more simply just edit bzrlib/repofmt/pack_repo.py so that "self._cache_limit = 1024*1024" rather than 0. It should make the pushing much faster.)

I still have no idea what's causing the memory consumption though!

Andrew, I tried the manual change ("self._cache_limit = 0" -> "self._cache_limit = 1024*1024") and it still didn't get past "step 0 of 4" in 14 minutes, after which I killed it.

I had a similar problem just now pushing a (stacked) branch that I had already pushed several times successfully last week, so I was just pushing up a couple of revisions. But first I had the following bit of fun:

$ bzr push
Using saved push location: bzr+ssh://<email address hidden>/~allenap/launchpad/package-specific-guidelines-bug-43893
bzr: ERROR: Revision {<email address hidden>} not present in "x_Mark_Shuttleworth_<email address hidden>_Sat_Sep_17_05:50:53_2005_14407.0".

Mmm. Tried again, the same. Merged trunk and it seemed fine, except that the push took forever. Ctrl-C yielded:

bzr: ERROR: bzrlib.errors.TooManyConcurrentRequests: The medium '<bzrlib.smart.medium.SmartSSHClientMedium object at 0x1321f10>' has reached its concurrent request
 limit. Be sure to finish_writing and finish_reading on the currently open request.

In the end I deleted the branch from the web UI and pushed again. It seems fine now.

I just had this again. I left it pushing for a *long* time and it stabilised at ~700M resident memory before I got bored and killed it.

This is with the 1.9 release.

See attached log, bzipped, it's massive! Sorry ...

This is happening more and more often now that everybody's submitting stacked branches to Launchpad

So let's at least upgrade to 1.9final which is in the ppa. It does
have some fixes past rc1, though not necessarily one that fixes this.

--
Martin <http://launchpad.net/~mbp/>

I'm running bzr nightly ppa.
Is this the smart server that needs updating?

I landed a change in bzr.dev 3840 that stops bzr from trying to push the entire history into stacked branches. I believe this is the root cause, but I can't be completely sure. If you are experiencing this problem, please see whether bzr.dev 3840 solves it for you.