Bazaar

Bazaar uses too much memory

Bug #277171 reported by Ted Gould
4
Affects Status Importance Assigned to Milestone
Bazaar
New
Undecided
Unassigned

Bug Description

I created a repository using "bzr svn-import" and it did not create a working tree. So I went into the directory and did a "bzr co" which eventually died because of Memory Error. When looking in top, here is the line after the error. It was going down, so I'm guessing it probably topped at 2GB, silly 32-bit OS.

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
 8637 ted 20 0 2969m 1.8g 3636 D 1 90.4 1:32.65 bzr

I would like to commend bazaar on it's effeciency. Filling 2GB of RAM with only 1.5 min of CPU time is very effective :)

Here is the backtrace if required:

$ bzr co
bzr: ERROR: exceptions.MemoryError:

Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 857, in run_bzr_catch_errors
    return run_bzr(argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 797, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 499, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line 954, in run
    source.bzrdir.create_workingtree(revision_id)
  File "/usr/lib/python2.5/site-packages/bzrlib/bzrdir.py", line 1410, in create_workingtree
    accelerator_tree=accelerator_tree, hardlink=hardlink)
  File "/usr/lib/python2.5/site-packages/bzrlib/workingtree_4.py", line 1388, in initialize
    hardlink=hardlink, delta_from_tree=True)
  File "/usr/lib/python2.5/site-packages/bzrlib/transform.py", line 1810, in build_tree
    delta_from_tree)
  File "/usr/lib/python2.5/site-packages/bzrlib/transform.py", line 1902, in _build_tree
    accelerator_tree, hardlink)
  File "/usr/lib/python2.5/site-packages/bzrlib/transform.py", line 1955, in _create_files
    new_desired_files)):
  File "/usr/lib/python2.5/site-packages/bzrlib/revisiontree.py", line 82, in iter_files_bytes
    for result in self._repository.iter_files_bytes(repo_desired_files):
  File "/usr/lib/python2.5/site-packages/bzrlib/repository.py", line 1358, in iter_files_bytes
    for record in self.texts.get_record_stream(text_keys, 'unordered', True):
  File "/usr/lib/python2.5/site-packages/bzrlib/knit.py", line 1221, in get_record_stream
    needed_from_fallback - absent_keys)
  File "/usr/lib/python2.5/site-packages/bzrlib/knit.py", line 992, in _get_content_maps
    record_map = self._get_record_map(keys, allow_missing=True)
  File "/usr/lib/python2.5/site-packages/bzrlib/knit.py", line 1116, in _get_record_map
    self._read_records_iter(records):
  File "/usr/lib/python2.5/site-packages/bzrlib/knit.py", line 1563, in _read_records_iter
    izip(iter(needed_records), raw_data):
  File "/usr/lib/python2.5/site-packages/bzrlib/knit.py", line 2408, in get_raw_records
    for names, read_func in reader.iter_records():
  File "/usr/lib/python2.5/site-packages/bzrlib/pack.py", line 272, in _iter_records
    for record in self._iter_record_objects():
  File "/usr/lib/python2.5/site-packages/bzrlib/pack.py", line 277, in _iter_record_objects
    record_kind = self.reader_func(1)
  File "/usr/lib/python2.5/site-packages/bzrlib/pack.py", line 218, in reader_func
    return self._source.read(length)
  File "/usr/lib/python2.5/site-packages/bzrlib/pack.py", line 177, in read
    self._next()
  File "/usr/lib/python2.5/site-packages/bzrlib/pack.py", line 172, in _next
    length, data = self.readv_result.next()
  File "/usr/lib/python2.5/site-packages/bzrlib/transport/__init__.py", line 680, in _seek_and_read
    data = fp.read(c_offset.length)
MemoryError

bzr 1.6.1 on python 2.5.2 (linux2)
arguments: ['/usr/bin/bzr', 'co']
encoding: 'UTF-8', fsenc: 'UTF-8', lang: 'en_US.UTF-8'
plugins:
  bzr_notification /home/ted/.bazaar/plugins/bzr_notification [unknown]
  gtk /usr/lib/python2.5/site-packages/bzrlib/plugins/gtk [0.95.0]
  launchpad /usr/lib/python2.5/site-packages/bzrlib/plugins/launchpad [unknown]
  power_management /home/ted/.bazaar/plugins/power_management [unknown]
  pqm /usr/lib/python2.5/site-packages/bzrlib/plugins/pqm [1.0.0dev0]
  rebase /usr/lib/python2.5/site-packages/bzrlib/plugins/rebase [0.3.0]
  search /home/ted/.bazaar/plugins/search [1.6.0dev0]
  svn /usr/lib/python2.5/site-packages/bzrlib/plugins/svn [0.4.11]
*** Bazaar has encountered an internal error.
    Please report a bug at https://bugs.launchpad.net/bzr/+filebug
    including this traceback, and a description of what you
    were doing when the error occurred.

Revision history for this message
Ted Gould (ted) wrote :
Revision history for this message
Jelmer Vernooij (jelmer) wrote :

What sort of size was the tree that had to be created ? Thousands of files, many gigabytes of data?

Revision history for this message
Ted Gould (ted) wrote : Re: [Bug 277171] Re: Bazaar uses too much memory

On Fri, 2008-10-03 at 01:11 +0000, Jelmer Vernooij wrote:
> What sort of size was the tree that had to be created ? Thousands of
> files, many gigabytes of data?

Yes and Yes.

About 28,000 files and about 41 GB. This is my photos repository of the
digital photos I've taken, and more importantly for version control,
touched up.

For those reading this bug looking for a work around. It turns out, at
least for me the first 'bzr co' that crashed created working directory,
but marked things as deleted. So they need to be reverted. It seems
that revert has the same problem so I set it up so that each leaf node
directory will get reverted staying bellow the amount of RAM I have. My
repository is JPEGs, so this command worked for me:

    $ bzr status | grep jpg | xargs -n 1 dirname | sort -u | xargs -n 1
--verbose bzr revert

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.