AssertionError while packing large files

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ted Gould wrote:
> ** Attachment added: "Crash File"
> http://launchpadlibrarian.net/51932457/bzr-20100714160709-20874.crash
>

I don't think this is because of the renamed file. It would hint that
something is funny about your pack files triggering this. If you just
need to work around it, probably 'bzr pack' will clear things up by
doing it slightly differently. It would be nice to get a snapshot of
your current state if you can, though I realize '/Movies' is likely to
be a lot of data.

The traceback shows:
   File
"/usr/lib/python2.6/dist-packages/bzrlib/repofmt/groupcompress_repo.py",
line 461, in _copy_text_texts
     'texts', self._get_progress_stream, 4)
   File
"/usr/lib/python2.6/dist-packages/bzrlib/repofmt/groupcompress_repo.py",
line 402, in _copy_stream
     reuse_blocks=False):
   File "/usr/lib/python2.6/dist-packages/bzrlib/groupcompress.py", line
1721, in _insert_record_stream
     nostore_sha=nostore_sha)
   File "/usr/lib/python2.6/dist-packages/bzrlib/groupcompress.py", line
797, in compress
     start, end, type = self._compress(key, bytes, len(bytes) / 2, soft)
   File "/usr/lib/python2.6/dist-packages/bzrlib/groupcompress.py", line
967, in _compress
     delta = self._delta_index.make_delta(bytes, max_delta_size)
   File "_groupcompress_pyx.pyx", line 227, in
bzrlib._groupcompress_pyx.DeltaIndex.make_delta
   File "_groupcompress_pyx.pyx", line 205, in
bzrlib._groupcompress_pyx.DeltaIndex._populate_first_index
 AssertionError

And in context, it would appear:
    cdef _populate_first_index(self):
        cdef delta_index *index
        if len(self._sources) != 1 or self._index != NULL:
            raise AssertionError('_populate_first_index should only be'
                ' called when we have a single source and no index yet')

        # We know that self._index is already NULL, so whatever
        # create_delta_index returns is fine
        with nogil:
            self._index = create_delta_index(&self._source_infos[0], NULL)
        assert self._index != NULL

So... what seems to be happening is that 'create_delta_index' is
returning NULL. I'm guessing you are running out of memory (if it is
trying to repack Movies.)

Can you tell us the size of the content you are working with?

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkw+tT4ACgkQJdeBCYSNAAO+FACfTvY2J2l0qM2dEDT9QkX8KfKv
J+EAoJ41pCH+OKXcjtnaWz+A/9jJCrqO
=sbRN
-----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ted Gould wrote:
> Public bug reported:
>
> Trying to commit a rename I get an AssertionError.

        with nogil:
            self._index = create_delta_index(&self._source_infos[0], NULL)
        assert self._index != NULL

^- Since this likely indicates an out-of-memory error, we may want to
have some sort of "if we run out of memory trying to compute a delta
index, then we fall back to not computing a delta".

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkw+taoACgkQJdeBCYSNAAMe7QCfSxEm5un1tYXSNA2QDX2VRl8d
TS0An2ZlhpPHr8giUra+YkLntlArtZcS
=ByCI
-----END PGP SIGNATURE-----

if it is a memory error, then 'bzr pack' won't help you. We'll have to think of something else.

If not a duplicate, certainly a relative of bug #109114

On Thu, 2010-07-15 at 07:14 +0000, John A Meinel wrote:
> So... what seems to be happening is that 'create_delta_index' is
> returning NULL. I'm guessing you are running out of memory (if it is
> trying to repack Movies.)
>
> Can you tell us the size of the content you are working with?

I think that it is a memory error. I was trying to mention that on IRC
when I got a stack trace. I'm unsure why it switch from an error to a
full blown trace. Interestingly, I don't ever see the Bazaar process
map all the memory in the system. I think it must be trying to allocate
a bunch all at once when this error occurs.

ted@vee:~/Desktop/Movies$ du -h
917M ./.bzr/repository/obsolete_packs
4.0K ./.bzr/repository/lock
136K ./.bzr/repository/upload
1.5M ./.bzr/repository/indices
5.5G ./.bzr/repository/packs
6.4G ./.bzr/repository
4.0K ./.bzr/branch-lock
4.0K ./.bzr/checkout/lock
60K ./.bzr/checkout
4.0K ./.bzr/branch/lock
56K ./.bzr/branch
6.4G ./.bzr
9.0G .

I'm guessing it might be interesting to know about the individual packs
as well:

ted@vee:~/Desktop/Movies/.bzr/repository/packs$ du -h -a
83M ./71dc01bb401621a3c82503e4ab898b94.pack
21M ./d201e5c85844d0311cd7b7283bb9198e.pack
745M ./dead3f314b7de880b9461523df63a7a6.pack
36M ./7bea5240efe69b01cfb1d2fe2c174f4c.pack
51M ./9a7bb71ed50f435a3ae2d3eaa0418597.pack
83M ./4c626509b02b72194de1e0e308176365.pack
253M ./749f15658f1a897292e9abfe3a0ab822.pack
257M ./410ad3f0f37f19b9f0798574fedbbe20.pack
109M ./1520df8dcef5c1250fb99360c6c19088.pack
958M ./f75cbefa740b612beebf261ddcf2a696.pack
34M ./fa4c13252102830b2c2bc21ff5b34122.pack
10M ./c101405d823e23b84918db6378f1c107.pack
65M ./fb51f4d81734ab89b076317aa1fe7f42.pack
332M ./f0fe95784d9092a5525d3236c4dfbc64.pack
51M ./acd07b5e2e8e670175c0c78149780bf1.pack
257M ./b5cc0183943ad37edccc1c527389caf6.pack
39M ./782a66d7cad04c379308250e280c53ff.pack
39M ./5a8b368f33c558cebbba9ac7281e3ab2.pack
85M ./07a9d095ea56a2328e44ac3ca3027926.pack
29M ./0cff12346f60ad75c83735c40de812e6.pack
895M ./6b1c249ca14e31ffa34402fc96711e1a.pack
67M ./bedbcfe03a6417a0f411807fffa2423a.pack
917M ./5658ae393def4ee8d6998f52080dfcf5.pack
209M ./a319feadbbb2cf55bb00d858d7f683c6.pack
5.5G .

I'm guessing it's trying to combine a couple of them at the time.

Perhaps unrelated, but FYI information. I've got 4GB of RAM, no swap,
running amd64.