BzrCheckError: Cannot add revision(s) to repository: missing referenced chk root keys

Hi

I have the same problem with this repository https://code.launchpad.net/~openerp/openobject-server/5.0

It recently update to format 2a, in my local copy i launch "bzr check" and "bzr upgrade" in 2.0.4
and today a bzr pull give me an error (check the logfile), i just upgrade to 2.1.0 and have the same problem.

Regards,

Pulling from bzr+ssh://bazaar.launchpad.net/~openerp/openobject-server/5.0/ into my branch (in a repository) I get an other missing sha1 error: missing referenced chk root keys: [StaticTuple('sha1:0c211a78d10233c47ce084b5b2ecbf44c1d09d40',)]

Creating a new branch within the same repository doesn't work either. Creating a completely new branch from https://code.launchpad.net/~openerp/openobject-server/5.0 is successful though.

So, chk root keys are pointed at by inventories, this is an unusual error to get. To debug I think we should dump the references revisions, inventories, referenced chk roots, and see if a pattern emerges.

I suspect the old branch was missing content and thus it wasn't migrated fully.

bug 539052 is a similar occurrence of this, also in the openerp branches. I wonder if they were migrated from bzr-svn or converted using an early version of bzr's 2a support, because similar problems seem to have occurred there.

Just a note that I'm recently getting this problem from lp:exaile. I ran the upgrade locally (bzr upgrade lp:exaile) because I didn't notice the "upgrade branch" link, using bzr 2.1.1. My local branch was on a shared repository, if that's relevant. The remote branch was migrated from svn a long time ago, but I don't know using which tool.

I've thoughtlessly deleted my old repository and can't restore it from a backup, but I've pointed users here if they are experiencing the problem and can help with testing stuff.

Here is a crash log of an occurence of this bug.

I've encountered a similar problem, again with OpenERP server 5.0 ; here is as much detail as I could gather.

I use a local repo to host 3 branches : "5.0", "dev" and "prod".
- "5.0" is a checkout of lp:~openerp/openobject-server/5.0/
- "dev" is where I make my daily work
- "prod" is a checkout of a distant (bzr+ssh) standalone branch called "server", on the production host.
Often when a release is tagged in "5.0", I'll merge it into "dev".
Eventually I'll merge "dev" into "prod", and "bzr update" the distant "server" branch via SSH.
That worked fine for the tag "5.0.7" and the tags before.

Now I merged the tag "5.0.10" from "5.0" into "dev", but when I merged "dev" into "prod" I couldn't commit (see crash file attached)
:
ErrorFromSmartServer: Error received from smart server: ('error', "Internal check failed: Cannot add revision(s) to repository: missing referenced chk root keys: [StaticTuple('sha1:0deb2b3b15a16211b3addbfb524efe48534b139c',)]")
Commit --local worked but then push failed, and uncommit failed too.
"bzr check" on the repo and the distant branch are OK.

I tried to rebuild a new repo and import my branches into it, but met the same error when I pushed to the distant branch.

I rsynced a local copy of the distant branch to make sure it's not a bzr+ssh issue, and met the same error whe I tried to push to it.

Via SSH, I created a new distant branch from the old distant branch and tried to push to it, and met the same error.

Finally I created a new repo, imported my branches into it, then created a new standalone branch from my new "prod" branch and rsync'ed it to the distant host to replace the "server" branch. That seems to work all right now.

I have a backup copy of the initial repo and the old distant branch, I'll make them available if someone wants to investigate.

Lionel.

Hi folks, i think the bug is priority in a 2.1.x release can be pointed in them?

Regards,

Yes, this would be reasonable to do in 2.0 and 2.1

I get the same type of error when I try to do: bzr branch lp:do - actually I already had a branch, which I migrated as requested. Then I couldn't pull anymore, so I deleted the directory and tried to branch again, which doesn't work. However, if I comment launchpad_username in ~/.bazaar/bazaar.conf and the whole [Launchpad] section in ~/.bazaar/authentification.conf, then I can branch without any problem.

I attached the crash file obtained when running "bzr branch lp:do" with my LP/SSH credentials specified.

Nicolas: thanks for that. I can reproduce this with "bzr branch lp:do" too! I'm grabbing a copy of lp:do's files (and the files of its stacked-on branch) now so that I can investigate in more detail.

I've started actively working on this. Unsurprisingly it looks like a bad interaction with stacking.

Specifically for the lp:do example, there are multiple inventories in the stream from the stacked-on branch with a parent_id_basename_to_file_id root key of sha1:576fa262710cb8cadf998f50c33f4fa28ac6f57b, but that key is apparently being filtered out due to also being the parent_id_basename_to_file_id root key of an inventory at the edge of ancestry.

Short answer: the repository has bad data, but it can be repaired. I have a fix for the cause of the bad data (see branch linked to this bug) and will provide a tool to do the repair soon. There is no data loss.

Long, jargon-filled answer:

The root cause here is that CHKMap.apply_delta and CHKMap._check_remap had bugs that meant that it could generate a CHK map with a non-canonical form. In the case of lp:do, there are some inventories records that exist in both lp:do and the branch it is stacked on (lp:do/0.9), but that have different CHK roots. Mostly this sort of inconsistency doesn't cause trouble, but in this case one of the inventories is at the stacking boundary, and fetch relies on the invariant that a given inventory record will be identical in both repositories: it fetches the inventory record for a parent inventory at the stacking boundary from one repo, then fetches CHK roots from the other. If the source repositories disagree on the CHK roots for that inventory then the target finds the CHK roots are missing, causing the error reported in this bug.

Submitting merge directives with bundles to <email address hidden> is causing this to appear in the oops: OOPS-1663CMP3

Can't do a checkout of lp:do because of this bug.

Will be fixed in 2.0.6, in the next 2.1, and it 2.2.0. As a separate task we need a thing to fix repositories affected by this: bug 613698.

The root cause has been fixed, and as per bug 613698 there's now a repair tool available in trunk.

If you have an affected repository, get a copy of bzr trunk (bzr branch lp:bzr), and run "bzr reconcile --canonicalize-chks" on the affected repository.

(I'll copy that instruction into the description of this bug to help people find it.)

Hello everyone,

Seeing that the repair tool has now made it into trunk with bug 613698, I take it Launchpad admins should run it on affected repositories on the Launchpad repositories, right? I don't think I can do it remotely, can I?

Should I open another bug report for it?
Seeing that current one is still Triaged for launchpad-code, I would like to request this for the official OpenObject/OpenERP branches:
   - openobject-server/trunk and openobject-server/5.0
   - openobject-addons/trunk and openobject-addons/5.0
   - openobject-client/trunk and openobject-client/5.0
   - openobject-client-web/trunk and openobject-client-web/5.0

I'm not sure all branches are affected, but I don't know whether I can check that easily. None of them are stacked, however they do have hundreds of dependent stacked branches, so please make sure a fresh backup is available before applying :-)

Thanks a lot!

Olivier I think you should be able to run it remotely, at least I can't see a reason why you couldn't.

Launchpad 10.12 uses bzr 2.2.2