Comment 55 for bug 405251

Revision history for this message
John A Meinel (jameinel) wrote : Re: [Bug 405251] Re: Huge data transfers/bad performance OVERALL

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Frits Jalvingh wrote:
> Ok, the huge amount of texts is horribly clear. Just for info, the
> repository one month ago, before moving to bzr 1.16 (revno 1292) has the
> following stats:
>
> jal@mabillon:~/vp-1292$ bzr repository-details
> Commits: 6408
> Raw % Compressed % Objects
> Revisions: 3463 KiB 0% 2534 KiB 1% 6408
> Inventories: 19270232 KiB 95% 25390 KiB 11% 6408
> Texts: 848675 KiB 4% 185519 KiB 86% 51993
> Signatures: 0 KiB 0% 0 KiB 0% 0
> Total: 20122370 KiB 100% 213445 KiB 100% 64809
>
> So somehow the #of texts has increased from 51993 to a horrible
> 812257.. Quite an increase. This increase seems to be in some 400
> revisions so: (812257-51993)/400=1900 delta files per revision if i'm
> correct? Quite steep, that.
>

I don't have any idea why 1.16 would be causing something like this. Are
you sure something didn't change inside your process?

Anyway, yes. The closest from your earlier postings is:
> Commits: 6622
> Raw % Compressed % Objects
> Revisions: 3573 KiB 0% 2617 KiB 0% 6622
> Inventories: 19979626 KiB 84% 55881 KiB 16% 6622
> Texts: 3722632 KiB 15% 288029 KiB 83% 388846

so 6622 - 6408 = 214 revisions, and 388,846 - 51,993 = 336,853 / 214 =
1,574 changes per revision.

And then on to:
> Commits: 7328
> Raw % Compressed % Objects
> Revisions: 3928 KiB 0% 2885 KiB 0% 7328
> Inventories: 21695982 KiB 70% 108661 KiB 19% 7328
> Texts: 9040343 KiB 29% 453880 KiB 80% 1017118

(1017118 - 388846) / (7328 - 6622) = 628,272 / 706 = 889

Also, you can note that your "inventories" size went from:
19,270,232 to 19,979,626 to 21,696,982

Which doesn't look like it is increasing by much overall. (So it
wouldn't seem that you are adding a lot of new files.) But the
"compressed" size went from 25MB to 108MB, or almost 4x growth,
indicating that it thinks you have a lot of new entries.

Now, it is also possible that we have a bug. For example, what if
'heads()' was giving incorrect information, and causing us to generate a
new revision for texts that didn't actually need it. That would also
point to a lot of the other issues you've been seeing.

Certainly it is worth investigating. Though I'd also be curious as to
why we haven't been seeing this with other projects.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkqA7z8ACgkQJdeBCYSNAAOUlACeOlDEuidCN7AAf8gLiRVJMrE3
t6MAnjS4RNMs6XfD/ui9kqHFqjGR8E36
=fbOy
-----END PGP SIGNATURE-----