after conversion chk formats are badly packed

I just did a conversion of bzr.dev, and after conversion it isn't particularly terrible. I believe the converted size was 65MB, and post-pack was 32MB. So about 2:1...

While that is bad, it isn't 1.7GiB => 150MiB that we saw with a different project. Which leads me to believe that the 1.7GiB conversion wasn't being done with InterDifferingSerializer, but was instead done via the plain fetch...

Just a thought, though.

On Fri, May 15, 2009 at 01:57:26PM -0000, John A Meinel wrote:
> I just did a conversion of bzr.dev, and after conversion it isn't
> particularly terrible. I believe the converted size was 65MB, and post-
> pack was 32MB. So about 2:1...

> While that is bad, it isn't 1.7GiB => 150MiB that we saw with a
> different project. Which leads me to believe that the 1.7GiB conversion
> wasn't being done with InterDifferingSerializer, but was instead done
> via the plain fetch...

This particular one was happening when importing from git into Bazaar.
bzr-git fetches happen on a per-revision basis, so multiple versions
of the same file aren't imported in a row (which I would imagine would
be the most optimal ?). The same problem would occur for svn I
imagine.

Cheers,

Jelmer

well, it depends on how you layer the 'commit_write_groups'. InterDifferingSerializer holds a write group across 100 revs, which causes autopack to occur every 1000 revs. If, instead, you hold the write group across all 50k revs, then yes, things will be bad.

In fact, I think under those circumstances it could expand all texts to fulltexts. It sort of depends what gets put into a single stream. (A group only survives a single 'insert_record_stream', so calling VF.add_lines() repeatedly will give all fulltexts.)

Bumping to critical, this will cause serious issues with 2.0 if not fixed.

So *with* IDS, they are at least reasonably packed after the conversion/fetch/whatever. They are not *optimally* packed, but they are not grotesquely bloated, either.

However, as you are trying to nuke IDS anyway, you are welcome to figure out how to handle the packing problem at the same time.

On Thu, 2009-06-11 at 14:27 +0000, John A Meinel wrote:
> So *with* IDS, they are at least reasonably packed after the
> conversion/fetch/whatever. They are not *optimally* packed, but they are
> not grotesquely bloated, either.
>
> However, as you are trying to nuke IDS anyway, you are welcome to figure
> out how to handle the packing problem at the same time.

I think doing a partial pack of the new data is a sane answer:
- in new conversions it will repack everything
- in incremental pulls it will repack only the pulled data

I agree that we don't really have to repack after an IDS conversion,
however I think its nice to give a really good impression:).

-Rob

I've added a bzr-svn task, because the fix involves an optional API that bzr-svn needs to start calling. Jelmer - it should be obvious by looking at rev 4470 in bzr.dev.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

This is fixed in bzr-svn and bzr-git.

  affects bzr-svn
  status fixreleased

  affects bzr-git
  status fixreleased
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iJwEAQECAAYFAkpDXzMACgkQDLQl4QYPZuW9+QP/ZOHi/FcONNm+kMyF7bnqdzp6
ZbowofHxVqe10/8AnYwOCyS35XdBFb7PQoGkSO+XSK3xWw/ROxCp27rHc++xha9/
1Fw2qBRO5wVff2Ay20FU4Ej89I7sOsIKqO6JT6DLCIFR2rvFgyP1ndDgCnFHUy4u
GDoLezfDfYAVvBYmf5w=
=+AUL
-----END PGP SIGNATURE-----

The current 1.16.1 NEWS shows this as merged there, but is that actually true? I'm checking.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Martin Pool wrote:
> The current 1.16.1 NEWS shows this as merged there, but is that actually
> true? I'm checking.
>

I see:

bzr cat http://bazaar-vcs.org/bzr/bzr.1.16/bzrlib/repository.py:

...
    def pack(self, hint=None):
...
        :param hint: If not supplied, the whole repository is packed.
            If supplied, the repository may use the hint parameter as a
            hint for the parts of the repository to pack. A hint can be
            obtained from the result of commit_write_group(). Out of
            date hints are simply ignored, because concurrent operations
            can obsolete them rapidly.
        """
...

The hinting was done as part of this work. So I assume Michael Hudson
cherrypicked across the pack after-conversion-fetch code.

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkpE/KEACgkQJdeBCYSNAAODKgCgqqer5pAo9+ls8/QGGeB0KgRv
ArYAoM872+Si+cORR+4yZtfqGG58gRSd
=gS2A
-----END PGP SIGNATURE-----

So I just realized.
Robert fixed bug #365615 as a side-effect of working on this bug.
Basically, he just started handling an AbsentContentFactory during autopack because he ran into it while testing the fix for this bug.

So yeah, its in 1.16.1