Comment 57 for bug 405251

I was forced by our management to do something quick because all 15 developers could hardly get
any work done. I decided to fully recreate our repositories without using anything from the old
ones, loosing all history in the process. We did this because:
* our old repo was appearently gravely damaged in some way
* we tried for more than a week to get some idea of how to fix it but there was no appearent end date for a fix
* Performance went from 30 minutes for a pull to over 2hours for one in a week - not a good trend
* Commit performance also went downhill very fast
* I guessed that we might find the problem, but it would probably be too much to ask for a "fix program" for
  the failed repository. So we would end up creating new ones anyway.
* Since the impression arose that we were the only ones having this problem we it could be that it
  came from something special we did; the only special things were the conversion to 1.14-rich-root and
  a merge-with-history of another, independent branch which could only be repeated a few times; after
  that it hit bug 364305. We hoped one of these caused the corruption; by starting anew without doing any
  of these we hoped the problem was gone.

We were wrong, sadly enough.
At commit 29 I noticed a very long pull, transferring 15MB again. I had not checked for 2 days because there
was a huge backlog of work to complete. But at this point (commit 29) I already have >100.000 texts in
the repository again.

I created the new repositories by just creating an empty, new repository
and adding the files as an initial commit. I started with our oldest software release (3.1) and created an
initial branch from that; I then branched that one to make the 3.2 branch; did a file sync on it so
the 3.2 branch got all of the files belonging to it and commited that (in effect creating 3.2 as a
single big delta from 3.1). This the same for 3.3 and the new trunk.

So I ended op with a new repository without *any* data from the previous one, and commit 4 or so was
the commit which created the trunk. At that point repository-details reported:
jal@odeon:~/vp-trunk-4/vp-trunk$ bzr repository-details ..
Commits: 4
                      Raw % Compressed % Objects
Revisions: 1 KiB 0% 1 KiB 0% 4
Inventories: 14099 KiB 4% 832 KiB 0% 4
Texts: 293404 KiB 95% 123475 KiB 99% 18320
Signatures: 0 KiB 0% 0 KiB 0% 0
Total: 307506 KiB 100% 124309 KiB 100% 18328
So 18K of texts.

The worst update after that was revision 26. Revision 25's repo-details was:
jal@odeon:~/test/vp$ bzr pull -r 25 ~/bzr/vp-split/vp
Now on revision 25.
jal@odeon:~/test/vp$ bzr repository-details ..
Commits: 39
                      Raw % Compressed % Objects
Revisions: 19 KiB 0% 14 KiB 0% 39
Inventories: 147345 KiB 27% 2956 KiB 2% 39
Texts: 390575 KiB 72% 124934 KiB 97% 28624
Signatures: 0 KiB 0% 0 KiB 0% 0
Total: 537939 KiB 100% 127905 KiB 100% 28702

jal@odeon:~/test/vp$ bzr pull -r 26 ~/bzr/vp-split/vp
Now on revision 26.
jal@odeon:~/test/vp$ bzr repository-details ..
Commits: 55
                      Raw % Compressed % Objects
Revisions: 27 KiB 0% 20 KiB 0% 55
Inventories: 211924 KiB 18% 8001 KiB 5% 55
Texts: 919581 KiB 81% 134790 KiB 94% 94270
Signatures: 0 KiB 0% 0 KiB 0% 0
Total: 1131533 KiB 100% 142812 KiB 100% 94380

So an increase of 28624 to 94270 texts, and the pull took a long time again.

Another, easier one is revno 5: it is an upward merge of only a few very small files
from version 3.1 to trunk. This one I am very sure is very small because I did this
one myself; it only touched a few small text files but increased the #of texts by
10K! It might be best to investigate this one 1st.

It looks like merge or a merge commit has a HUGE problem somewhere.

I will attach the full log for the pull session (console log) and a full history
log for all revisions (bzr log --include-merges --verbose --forward)

John: I will try to get you access to this new repository; I hope you are still
willing to look at it... I will try to find you on IRC.