Bazaar

MemoryError in _ensure_content during commit in small repository

Bug #528793 reported by Martin Pool
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Bazaar
Confirmed
Medium
Unassigned

Bug Description

Originally from https://answers.launchpad.net/bzr/+question/102562

I'm getting out of memory error during the commit operation (it stalls for a while showing
'Saving data locally - Stage:repacking texts:texts 14/175') and then aborts.

It's a relatively small repository, and the commit renames 3 files and modifies another 20. Needless to say it had been working perfectly fine for quite some time. I'm running version 2.0.2 on Ubuntu Karmic 32 bits.

Any light on what's going on?

Thanks

>bzr info -v
Standalone tree (format: 2a)
Location:
  branch root: .

Related branches:
    push branch: sftp://[EDITED]
  parent branch: sftp://[EDITED]

Format:
       control: Meta directory format 1
  working tree: Working tree format 6
        branch: Branch format 7
    repository: Repository format 2a - rich roots, group compression and chk inventories

In the working tree:
       219 unchanged
        33 modified
         0 added
         0 removed
         3 renamed
         2 unknown
        67 ignored
        39 versioned subdirectories

Branch history:
        59 revisions
        78 days old
   first revision: Thu 2009-12-10 15:35:19 -0600
  latest revision: Fri 2010-02-26 15:18:08 -0600

Repository:
        59 revisions

[...]
[28446] 2010-02-26 18:56:51.772 INFO: aborting commit write group: MemoryError()
60.861 Traceback (most recent call last):
  File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 842, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 1037, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 654, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.6/dist-packages/bzrlib/builtins.py", line 3058, in run
    exclude=safe_relpath_files(tree, exclude))
  File "/usr/lib/python2.6/dist-packages/bzrlib/decorators.py", line 192, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/usr/lib/python2.6/dist-packages/bzrlib/workingtree_4.py", line 197, in commit
    result = WorkingTree3.commit(self, message, revprops, *args, **kwargs)
  File "/usr/lib/python2.6/dist-packages/bzrlib/decorators.py", line 192, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/usr/lib/python2.6/dist-packages/bzrlib/mutabletree.py", line 229, in commit
    *args, **kwargs)
  File "/usr/lib/python2.6/dist-packages/bzrlib/commit.py", line 375, in commit
    self.rev_id = self.builder.commit(self.message)
  File "/usr/lib/python2.6/dist-packages/bzrlib/repository.py", line 175, in commit
    self.repository.commit_write_group()
  File "/usr/lib/python2.6/dist-packages/bzrlib/repository.py", line 1554, in commit_write_group
    result = self._commit_write_group()
  File "/usr/lib/python2.6/dist-packages/bzrlib/repofmt/pack_repo.py", line 2268, in _commit_write_group
    hint = self._pack_collection._commit_write_group()
  File "/usr/lib/python2.6/dist-packages/bzrlib/repofmt/pack_repo.py", line 2122, in _commit_write_group
    result = self.autopack()
  File "/usr/lib/python2.6/dist-packages/bzrlib/repofmt/pack_repo.py", line 1472, in autopack
    return self._do_autopack()
  File "/usr/lib/python2.6/dist-packages/bzrlib/repofmt/pack_repo.py", line 1512, in _do_autopack
    reload_func=self._restart_autopack)
  File "/usr/lib/python2.6/dist-packages/bzrlib/repofmt/groupcompress_repo.py", line 691, in _execute_pack_operations
    result = packer.pack()
  File "/usr/lib/python2.6/dist-packages/bzrlib/repofmt/pack_repo.py", line 760, in pack
    return self._create_pack_from_packs()
  File "/usr/lib/python2.6/dist-packages/bzrlib/repofmt/groupcompress_repo.py", line 476, in _create_pack_from_packs
    self._copy_text_texts()
  File "/usr/lib/python2.6/dist-packages/bzrlib/repofmt/groupcompress_repo.py", line 459, in _copy_text_texts
    'texts', self._get_progress_stream, 4)
  File "/usr/lib/python2.6/dist-packages/bzrlib/repofmt/groupcompress_repo.py", line 400, in _copy_stream
    reuse_blocks=False):
  File "/usr/lib/python2.6/dist-packages/bzrlib/groupcompress.py", line 1714, in _insert_record_stream
    bytes = record.get_bytes_as('fulltext')
  File "/usr/lib/python2.6/dist-packages/bzrlib/groupcompress.py", line 454, in get_bytes_as
    self._manager._prepare_for_extract()
  File "/usr/lib/python2.6/dist-packages/bzrlib/groupcompress.py", line 554, in _prepare_for_extract
    self._block._ensure_content(self._last_byte)
  File "/usr/lib/python2.6/dist-packages/bzrlib/groupcompress.py", line 158, in _ensure_content
    self._z_content, num_bytes + _ZLIB_DECOMP_WINDOW)
MemoryError

60.871 return code 3

See original description
Tags: 2a memory
Martin Pool (mbp)
tags: added: 2a memory
Changed in bzr:
status: New → Confirmed
importance: Undecided → Medium
Revision history for this message
Martin Pool (mbp) wrote :

Looks like bug 528724

description: updated
Revision history for this message
Robert Collins (lifeless) wrote :

@alejandro, can you include
'du -sh .bzr/repository' - this will tell us how large the repo actually is.
'bzr inventory | wc -l ' - this will tell us how many versioned files there are.

It looks like it compressed your content ok, but can't decompress it for some reason.

Revision history for this message
alejandro (alejandrorm) wrote :

Here it is:

>du -sh .bzr/repository
557M .bzr/repository
>bzr inventory | wc -l
255

Revision history for this message
Greg (gregspecialsource) wrote :

I may be experiencing the same issue.
I am committing a tiny change, just 2 modified text files.
Repository is ~30mb, ~5000 files version controlled.

I get the following output in Bazaar Explorer:

committing to: .....
modified .....
modified .....
aborting commit write group: Memory Error()
bzr: out of memory

bzrlib v2.1.1
QBzr v0.1.8.4
Python v2.5.4

Any news on this or work around?
Trying to use Bazaar on a project and wondering if I've made a big mistake.

Revision history for this message
Robert Collins (lifeless) wrote : Re: [Bug 528793] Re: MemoryError in _ensure_content during commit in small repository

Could you try from the command line? I wouldn't expect a 30MB file to
have ant issues at all.

It may be a bzr explorer bug.

Revision history for this message
Greg (gregspecialsource) wrote :

Thanks for the suggestion Robert.
The command line...
bzr commit -m "my comment"
...produced exactly the same output.

Revision history for this message
Greg (gregspecialsource) wrote :
Download full text (5.1 KiB)

Here is the entire .bzr.log file generated from the comand
bzr commit -m 'my comment'

The critical bit is...

0.597 Adding the key (<bzrlib.btree_index.BTreeGraphIndex object at 0x02F04BF0>, 42, 7045986) to an LRUSizeCache failed. value 611025734 is too big to fit in a the cache with size 41943040 52428800
2.220 aborting commit write group because of exception:

System is Windows 7 Pro 64bit, 6gb RAM

-----begin file-----
this is a debug log for diagnosing/reporting problems in bzr
you can delete or truncate this file, or include sections in
bug reports to https://bugs.launchpad.net/bzr/+filebug

Wed 2010-07-07 13:13:11 +1000
0.048 bazaar version: 2.1.1
0.050 bzr arguments: [u'commit', u'-m', u'my comment']
0.059 looking for plugins in C:/Users/userA/AppData/Roaming/bazaar/2.0/plugins
0.060 looking for plugins in C:/Program Files (x86)/Bazaar/plugins
0.150 encoding stdout as sys.stdout encoding 'cp850'
0.176 opening working tree 'C:/Dev/Proj1/userAAPlay'
0.261 preparing to commit
[ 2044] 2010-07-07 13:13:12.283 INFO: Committing to: V:/Proj1/userAA1/
0.368 Selecting files for commit with filter None
[ 2044] 2010-07-07 13:13:12.358 INFO: modified src/Tests/TestBB1.cpp
[ 2044] 2010-07-07 13:13:12.361 INFO: modified src/Tests/bbEngine.cpp
[ 2044] 2010-07-07 13:13:12.361 INFO: modified src/Tests/bbEngine.h
0.492 Auto-packing repository GCRepositoryPackCollection(CHKInventoryRepository('file:///C:/Dev/Proj1/.bzr/repository/')), which has 11 pack files, containing 20 revisions. Packing 10 files into 1 affecting 10 revisions
0.495 repacking 10 revisions
0.502 repacking 10 inventories
0.507 repacking chk: 10 id_to_entry roots, 5 p_id_map roots, 77 total keys
0.562 repacking 60 texts
0.597 Adding the key (<bzrlib.btree_index.BTreeGraphIndex object at 0x02F04BF0>, 42, 7045986) to an LRUSizeCache failed. value 611025734 is too big to fit in a the cache with size 41943040 52428800
2.220 aborting commit write group because of exception:
2.222 Traceback (most recent call last):
  File "bzrlib\commit.pyo", line 402, in _commit
  File "bzrlib\repository.pyo", line 179, in commit
  File "bzrlib\repository.pyo", line 1563, in commit_write_group
  File "bzrlib\repofmt\pack_repo.pyo", line 2314, in _commit_write_group
  File "bzrlib\repofmt\pack_repo.pyo", line 2174, in _commit_write_group
  File "bzrlib\repofmt\pack_repo.pyo", line 1476, in autopack
  File "bzrlib\repofmt\pack_repo.pyo", line 1516, in _do_autopack
  File "bzrlib\repofmt\groupcompress_repo.pyo", line 693, in _execute_pack_operations
  File "bzrlib\repofmt\pack_repo.pyo", line 761, in pack
  File "bzrlib\repofmt\groupcompress_repo.pyo", line 478, in _create_pack_from_packs
  File "bzrlib\repofmt\groupcompress_repo.pyo", line 461, in _copy_text_texts
  File "bzrlib\repofmt\groupcompress_repo.pyo", line 402, in _copy_stream
  File "bzrlib\groupcompress.pyo", line 1704, in _insert_record_stream
  File "bzrlib\groupcompress.pyo", line 438, in get_bytes_as
  File "bzrlib\groupcompress.pyo", line 538, in _prepare_for_extract
  File "bzrlib\groupcompress.pyo", line 151, in _ensure_content
MemoryError

[ 2044] 2010-07-07 13:13:14.140 INFO: aborting commit write group: MemoryError()
2.29...

Read more...

Revision history for this message
Greg (gregspecialsource) wrote :

I realized I've hijacked this thread and the original bug is nothing to do with mine.
I've opened a new bug #602614 to report this issue.
Admins, feel free to delete my posts from this thread.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Related questions

Remote bug watches

Bug watches keep track of this bug in other bug trackers.