Import fails with missing referenced chk root keys

Branching to my local box also shows the same issue.

Checking the branches directly on LP over the smart server gives:

Traceback (most recent call last):
  File "/home/jw2328/devel/bzr/bzr.dev/bzrlib/commands.py", line 911, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/home/jw2328/devel/bzr/bzr.dev/bzrlib/commands.py", line 1111, in run_bzr
    ret = run(*run_argv)
  File "/home/jw2328/devel/bzr/bzr.dev/bzrlib/commands.py", line 689, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/home/jw2328/devel/bzr/bzr.dev/bzrlib/commands.py", line 704, in run
    return self._operation.run_simple(*args, **kwargs)
  File "/home/jw2328/devel/bzr/bzr.dev/bzrlib/cleanup.py", line 135, in run_simple
    self.cleanups, self.func, *args, **kwargs)
  File "/home/jw2328/devel/bzr/bzr.dev/bzrlib/cleanup.py", line 165, in _do_with_cleanups
    result = func(*args, **kwargs)
  File "/home/jw2328/devel/bzr/bzr.dev/bzrlib/builtins.py", line 3263, in run
    check_dwim(path, verbose, do_branch=branch, do_repo=repo, do_tree=tree)
  File "/home/jw2328/devel/bzr/bzr.dev/bzrlib/check.py", line 452, in check_dwim
    check_repo=do_repo)
  File "/home/jw2328/devel/bzr/bzr.dev/bzrlib/decorators.py", line 140, in read_locked
    result = unbound(self, *args, **kwargs)
  File "/home/jw2328/devel/bzr/bzr.dev/bzrlib/remote.py", line 1585, in check
    callback_refs=callback_refs, check_repo=check_repo)
  File "/home/jw2328/devel/bzr/bzr.dev/bzrlib/decorators.py", line 140, in read_locked
    result = unbound(self, *args, **kwargs)
  File "/home/jw2328/devel/bzr/bzr.dev/bzrlib/repository.py", line 2777, in check
    check_repo=check_repo)
  File "/home/jw2328/devel/bzr/bzr.dev/bzrlib/repository.py", line 2781, in _check
    result.check(callback_refs)
  File "/home/jw2328/devel/bzr/bzr.dev/bzrlib/check.py", line 102, in check
    self.check_weaves()
  File "/home/jw2328/devel/bzr/bzr.dev/bzrlib/check.py", line 292, in check_weaves
    self._check_weaves(storebar)
  File "/home/jw2328/devel/bzr/bzr.dev/bzrlib/check.py", line 308, in _check_weaves
    self.repository.texts)
  File "/home/jw2328/devel/bzr/bzr.dev/bzrlib/repository.py", line 4208, in check_file_version_parents
    return self._check_file_version_parents(texts, progress_bar)
  File "/home/jw2328/devel/bzr/bzr.dev/bzrlib/repository.py", line 4229, in _check_file_version_parents
    knit_parents = parent_map[key]
KeyError: ('control-20090513234116-5pygawx43okof7y3-214', '<email address hidden>')

for 4 out of 4 of the branches I tried.

All of the file ids of those 4 have been something inside debian/, which might give a clue.

Certainly smells of corruption to me, but it's too systematic to be random.

All of the branches were pushed to LP without issue, and then failed when branching them
again. This could be an asymmetry in push/pull, code changes over time, or perhaps these
branches were upgraded to 2a on the server.

Bumped to critical, as this is an unknown source of branch corruption.

Thanks,

James

on whether it's something odd that bzr-builddeb is doing: It uses
a tree transform to manipulate the working tree, and then just commits as normal, so
it's not serializing revisions directly or anything. If it's bzr-builddeb then it's
the tree transform putting the working tree in an odd state that bzr doesn't catch.

Help in working out what these errors mean is wrong with the repository would be
welcome.

Also, is there a chance that this is a stacking issue? We do monkey with stacking
at release time, but I don't think we've done that with Debian branches yet, and
most of the ones I checked were Debian.

Thanks,

James

FWIW I've just reproduced this locally, thanks to the very helpful instructions at <https://wiki.ubuntu.com/DistributedDevelopment/UnderTheHood/Importer/Hacking>.

I'm digging deeper now.

So far this looks somewhat similar to bug 485601: the package import process appears to be creating a situation where two bzr repositories have different ideas about what a particular inventory contains. Although the differences appear even more severe than the differences bzr-svn appears to be creating in bug 485601.

e.g. for my testing with ifenslave-2.6 the repo on Launchpad and the locally generated repo disagree about the inventory called <email address hidden> for seven inventory entries. To take the first one as an example, changelog, the entry has differences in 'revision', 'text_sha1' and 'text_size'. At a glance the other size differ on the same fields.

Incidentally I've created 'bzr cross-check' command to help identify this sort of issue: <lp:~spiv/+junk/bzr-cross-check>.

You can see the conflicting inventories directly (if you install my bzr-cross-check plugin) with:

$ bzr cross-check bzr+ssh://bazaar.launchpad.net/~ubuntu-branches/debian/sid/ifenslave-2.6/sid bzr+ssh://bazaar.launchpad.net/~ubuntu-branches/debian/squeeze/ifenslave-2.6/squeeze

You'll get this output:

1 common inventories to check (10 seen)
disagree id_to_entry for <email address hidden>: StaticTuple('sha1:588829739866c59b8db423021fa0c767d7ddfa45',) StaticTuple('sha1:9dc9c98f24bfcde96b062248ec25fc652c26e19d',)

That revision was apparently created by the Bazaar Package Importer. So in addition to fixing whatever is generating inventories inconsistently, we'll need to repair existing repositories with the wrong data, probably both on Launchpad and users' systems :(

So, here' s a summary of where I'm at:

 * existing package import branches on Launchpad have mutually inconsistent data. This will need to be fixed one way or another
 * a fresh import of affected branches using import_package.py --no-existing works correctly AFAICS
 * nowhere in the lp:udd or lp:bzr-builddeb code appears any likely suspects for generating this sort of inconsistency, and a cursory glance at their revision history doesn't show any likely culprits either.

So I'm currently stumped as to the root cause. Something somewhere must have specified an existing revision ID when generating a new revision or inventory object, I think. But I just can't find any code here that would have done that.

I've polished my cross-check command and have mostly integrated it into lp:bzr-crosscheck. The current work-in-progress is lp:~spiv/bzr-crosscheck/integration.

What's the cost to redoing the affected imports from scratch? They would be effectively parallel imports, but perhaps no-one has made branches of them? I'm thinking that doing this might be the simplest and quickest way to fix the affected package imports.

Another related thought I've had is that it would probably be worth extending bzr's error reporting in this case to include the revid(s) that reference those root keys. it would cost a little time to calculate, but not very much and only affects this serious error case anyway, and would help by pointing straight to the affected revision(s).

On Thu, 21 Oct 2010 06:30:18 -0000, Andrew Bennetts <email address hidden> wrote:
> So, here' s a summary of where I'm at:
>
> * existing package import branches on Launchpad have mutually inconsistent data. This will need to be fixed one way or another
> * a fresh import of affected branches using import_package.py --no-existing works correctly AFAICS
> * nowhere in the lp:udd or lp:bzr-builddeb code appears any likely suspects for generating this sort of inconsistency, and a cursory glance at their revision history doesn't show any likely culprits either.
>
> So I'm currently stumped as to the root cause. Something somewhere must
> have specified an existing revision ID when generating a new revision or
> inventory object, I think. But I just can't find any code here that
> would have done that.

I don't know of anything that would have done that.

Are you able to determine when the revisions that are problematic were
created? (Revision date won't help, but the revid should)

> What's the cost to redoing the affected imports from scratch? They
> would be effectively parallel imports, but perhaps no-one has made
> branches of them? I'm thinking that doing this might be the simplest
> and quickest way to fix the affected package imports.

That's probably ok for the vast majority. I would want to make sure that
the branches were ones only committed to by the importer before doing
that though, so that we would not be losing information.

> Another related thought I've had is that it would probably be worth
> extending bzr's error reporting in this case to include the revid(s)
> that reference those root keys. it would cost a little time to
> calculate, but not very much and only affects this serious error case
> anyway, and would help by pointing straight to the affected revision(s).

That sounds like a good idea. This isn't an error useful to a user, so I
think it should just be loaded with as much information as an expert
would need to dig in to the problem.

Thanks,

James

James Westby wrote:
[...]
> I don't know of anything that would have done that.
>
> Are you able to determine when the revisions that are problematic were
> created? (Revision date won't help, but the revid should)

The 'bzr cross-check' plugin I mentioned earlier can tell you. I gave an
example in comment #6: <email address hidden>
of some ifenslave-2.6 branches.

I wonder if anything was special about 25th May last year, other than it being
my 5 year anniversary at Canonical... ;)

I'll try cross-checking a few more of the affected imports and see if the dates
appear to be similar.

So it's extremely unlikely that this sort of corruption would have
occured other than when revisions were being created? i.e. we are
looking at the time when the importer created the revisions, and not at
any subsequent modifications to the repository?

Thanks,

James

Well, the issue appears to be essentially that there are *two* different creations of the same revision (or at least, of the same inventory). It's conceivable that something like a format upgrade or cross-format fetch is to blame — a format change necessarily has to generate a new serialisation of an inventory from an existing one. Although in this case I'd be surprised if that had happened, because the changes to contents appear to be pretty significant: not just different metadata but different file contents!

However AFAIK in all our repository formats once a record for a given key has been written then it's basically set in stone: even if a later write to the repository attempts to add different contents for an existing key it won't overwrite that (either the apparent duplicate will be silently ignored, or if the discrepancy is noticed bzr may give an error). So yes, I think it's extremely unlikely to have occurred other than when revisions are created. For comparison there's a current bzr-svn bug can trigger the same sort of error after ghosts have been filled in the SVN repo — so at two different times the SVN source generated slightly different inventories with the same name.

More data points from debian sid/squeeze/lenny import branches:

 * apr-utils: <email address hidden>
 * qcontrol: <email address hidden>
 * hevea: <email address hidden>
 * docbook-xsl: <email address hidden>

So far exactly one disagreeing inventory in each case (even though the number of common revisions between the repos varies from just 1 to most).

> not just different metadata but different file contents!

Or even a different set of files: e.g. should <email address hidden> in the espa-nol branches contain debian/README.source (file-id: readme.source-20090903184448-qpjmpa9nnfr18fpx-1) or not? Different copies of that inventory disagree!

So far the latest date in any revid I've seen is 2009-08-21.

moin and backuppc have 2 broken revids each:

moin: <email address hidden> and <email address hidden>
backuppc: <email address hidden> and <email address hidden>

Mildly interesting that it's the same date for the 2 moin ones, but very different dates for the backuppc ones.

I'm curious what the differences are between the inventories, and if
there are file differences do they say they were changed in this
revision?

Martin Pool wrote:
> I'm curious what the differences are between the inventories, and if
> there are file differences do they say they were changed in this
> revision?

So here's some more details about
<email address hidden> of the diffutils package
import branches for debian squeeze vs. debian sid.

6 files have different contents:

debian/changelog
debian/control
debian/rules
debian/copyright
config.guess
config.sub

Let's take a closer look at say debian/control (just because it's the smallest
file). First the boring bits, the parts the inventories agree on:

  file-id: control-20090616020227-q98q6yzybj4rvrua-209
  parent file-id: debian-20090616020227-q98q6yzybj4rvrua-207
  name in parent: control

Now here's what one repo thinks for the rest:

  revision: <email address hidden>
  sha-1: 16b4c3b1659e4ac76332e47c2934a3174d516e07
  size: 997

But here's what another repo thinks:

  revision: <email address hidden>
  sha-1: 1f6e1c1b6a0b5d2b7970ff8ea1046e17dfefcc89
  size: 1320

Now I see something interesting there: the file-revision in one of them has an
embedded date newer than the inventory's own embedded date (2009-09-08 >
2009-05-29). But then, so does the file-id, which apparently isn't
contentious...

Another point of interest is that the sid branch contains one revision, and is
stacked on the squeeze branch, which has 5 revisions (no revisions in common).
The 1 inventory in common is the one copied across the stacking boundary, the
parent of the other inventory in sid!

So perhaps this is or was due to a stacking bug. I wonder if it's a fixed one?

Also of note is the log -v of the sid branch: it claims no changes in the tip revision, but that seems unlikely...

It might be the long-deferred result of bug 354036 or perhaps bug 390563, or perhaps a previously undetected failure mode from the same root cause.

(I should note that the dates on the affected revisions line up with that theory: there aren't any dates from much after it was fixed in bzr trunk...)

is this still in progress, or stuck...?

I think we should just regenerate the affected imports from scratch. This is a bit disruptive, as it will break any branches people have made — but those branches are presumably already affected by this issue.

The theory here is that whatever caused that corruption has since been fixed in bzr: the inconsistencies are quite old, and udd/bzr-builddeb are calling bzr's APIs in reasonable ways AFAICT. If the theory is right, we won't see this problem recur once we redo those imports.

That's fine by me, but I would like a spot check to ensure that we aren't going to be deleting
any branches that we didn't create.

I think a check that we only remove branches owned by ~ubuntu-branches should be enough
for this.

There may be some developer-pushed revisions that we delete, but probably not as the branches
haven't been up to date for a while.

Thanks,

James

<james_w> poolie, you have to delete them from codehosting, and then clear out the info on package-import for them, and then retry

@spiv: the package importer runs bzr-2.1.1 but bug #522637 has been fixed in 2.1.3.

Is it relevant ?

vila: Good catch! That does seem likely. Here's the cross-check output for dsdo:

$ bzr cross-check $(for urlpart in $( (for ubu in breezy dapper edgy feisty gutsy hardy intrepid jaunty karmic lucid maverick natty ; do echo ubuntu/$ubu/dsdo/$ubu; done) ; (for deb in sid squeeze; do echo debian/$deb/dsdo/$deb; done) ); do echo http://bazaar.launchpad.net/~ubuntu-branches/$urlpart; done)
17 common revisions to check (23 seen)
17 common inventories to check (23 seen)
disagree id_to_entry for <email address hidden>: StaticTuple('sha1:61cb883b87b4ff8f95117463cca5eff890639e83',) StaticTuple('sha1:53f99dc4dd7e3d7d3ed9f26f6994c9b6c8554681',)

Which is exactly the sort of error that bug produced.

For some reason when I scanned the NEWS entries for relevant fixes since 2.1.1 I missed that one.

So: next step is to get 2.1.1 upgraded to 2.1.3 (or 2.2.1 or 2.3) on that system. I'll file an RT.

Then we can either reset those imports again, or run "bzr reconcile --canonicalize-chks $repo" on all the affected repos with bzr 2.3. The latter is probably less disruptive if we can run 2.3 on the package-import system.

It's possible, perhaps likely, that other imports have old revisions with this inconsistency. It's probably good enough to just 'bzr reconcile --canonicalize-chks' those if/when they turn up.

bzr has now been upgraded on that system. Unfortunately the problem for at least dsdo turns out to be deeper: the inventory SHAs disagree not because of non-canonical inventories, but because they actually have different contents. Here's a diff of the long testaments for <email address hidden> in sid vs. squeeze:

--- dsdo.sid.testament 2011-02-04 12:34:43.044796998 +1100
+++ dsdo.squeeze.testament 2011-02-04 12:34:51.060796998 +1100
@@ -35,21 +35,21 @@
   directory debian/cdbs cdbs-20090616021458-osacbonnnq87rgjn-25
   directory debian/cdbs/1 1-20090616021458-osacbonnnq87rgjn-26
   directory debian/cdbs/1/class class-20090616021458-osacbonnnq87rgjn-27
- file debian/cdbs/1/class/dict.mk dict.mk-20090616021458-osacbonnnq87rgjn-28 afd1d248a964e760d449d04f878546188d0996fb
+ file debian/cdbs/1/class/dict.mk dict.mk-20090616021458-osacbonnnq87rgjn-28 5f2ce2df751a29da14a947496586a9a14b0ffb09
   directory debian/cdbs/1/rules rules-20090616021458-osacbonnnq87rgjn-29
   file debian/cdbs/1/rules/buildinfo.mk buildinfo.mk-20090616021458-osacbonnnq87rgjn-31 e0d23276a1a31e076d26a1cc7536668ae211a4d2
   file debian/cdbs/1/rules/copyright-check.mk copyrightcheck.mk-20090616021458-osacbonnnq87rgjn-53 18bb4b6ab155e58a9841224eca04a10e05b8ef76
   file debian/cdbs/1/rules/package-relations.mk packagerelations.mk-20090616021458-osacbonnnq87rgjn-86 228575c081f448eaadf403cc42182c310cf30d0f
   file debian/cdbs/1/rules/upstream-tarball.mk upstreamtarball.mk-20090616021458-osacbonnnq87rgjn-64 4e02fba31a1abbdaf4480d8eb5d500b5bd14cedf
- file debian/changelog changelog-20090616021458-osacbonnnq87rgjn-39 126be1eea1780e5ef44f0b92ca9e679c87cc198d
+ file debian/changelog changelog-20090616021458-osacbonnnq87rgjn-39 02bdc6b9562576ec97cce215984591df8c61067a
   file debian/compat compat-20090616021458-osacbonnnq87rgjn-42 ccf271b7830882da1791852baeca1737fcbe4b90
- file debian/control control-20090616021458-osacbonnnq87rgjn-34 26d6c96ebd8883b8d0d74cc4d1c4944adec21b18
+ file debian/control control-20090616021458-osacbonnnq87rgjn-34 677f89f9321be77ad43ebbb35385ef06c02c7dea
   file debian/control.in control.in-20090616021458-osacbonnnq87rgjn-37 051c9140259db174e6aa0bc2b9ab0520c6b8a32c
   file debian/copyright copyright-20090616021458-osacbonnnq87rgjn-40 68a10afb19ae03a3e018d21dd056a32053772c62
   file debian/copyright_hints copyright_hints-20090616021458-osacbonnnq87rgjn-54 751df3c96824a6cfe7fef0390ea29d4fccc54d0d
   file debian/idanish.info-ispell idanish.infoispell-20090616021458-osacbonnnq87rgjn-36 8802d6a3ffcd933c26d13905727c838551efd07d
   file debian/myspell-da.info-myspell myspellda.infomyspel-20090616021458-osacbonnnq87rgjn-43 693b404a3f13e9cbb4a38b76e6d2326523ff9384
- file debian/myspell-da.links myspellda.links-20090616021458-osacbonnnq87rgjn-73 1b677c610bc1c8083acf0b373a9f44aea1180014
+ file debian/myspell-da.links myspellda.links-20090616021458-osacbonnnq87rgjn-73 763d68a0ee4761407284b498ece6880c7d492cf3
   directory debian/patches patches-20090616021458-osacbonnnq87rgjn-32
   file debian/patches/1001_enable_aspell_rules.patch 1001_enable_aspell_r-20090616021458-osacbonnnq87rgjn-67 6bbef08f28677116b8e170603f68a9e8c1145b14
 ...

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

...
>
> One more strange detail: james_w ran a script to delete all the branches
> of these affected imports, and yet http://bazaar.launchpad.net/~ubuntu-
> branches/debian/squeeze/dsdo/squeeze/.bzr/repository/packs/ and
> http://bazaar.launchpad.net/~ubuntu-
> branches/debian/sid/dsdo/sid/.bzr/repository/packs/ show timestamps
> almost 12 months old for the pack files! I guess the deletion failed?
>

Or he deleted the branch but not the repository? Or some of the packs
aren't referenced in pack-names, or...?

The differing content for the same revision-id is, indeed, pretty
worrying. The importer doesn't use deterministic revision ids, so I
really don't know how that could have happened.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk1MF8wACgkQJdeBCYSNAANAjACfUz3sbclASWYFJpwNOK4nAlcy
fpMAn0vTkEq8/8d96e02pYUsnFmFFDS8
=7EXG
-----END PGP SIGNATURE-----

> > One more strange detail: james_w ran a script to delete all the branches
> > of these affected imports, and yet http://bazaar.launchpad.net/~ubuntu-
> > branches/debian/squeeze/dsdo/squeeze/.bzr/repository/packs/ and
> > http://bazaar.launchpad.net/~ubuntu-
> > branches/debian/sid/dsdo/sid/.bzr/repository/packs/ show timestamps
> > almost 12 months old for the pack files! I guess the deletion failed?
>
> Or he deleted the branch but not the repository? Or some of the packs
> aren't referenced in pack-names, or...?

The deletes were performed through the API, i.e. hopefully equivalent to
clicking the 'Delete' button in the web UI.

> The differing content for the same revision-id is, indeed, pretty
> worrying. The importer doesn't use deterministic revision ids, so I
> really don't know how that could have happened.

Right :(

Ok, I can finally reproduce this locally, even with trunk lp:bzr (as well the old 2.1.1 that was being used). This command:

  ./import_package.py --persistent-download-cache --local-branches dsdo

gives me this traceback:

Traceback (most recent call last):
  File "./import_package.py", line 1098, in <module>
    persistent_download_cache=options.persistent_download_cache))
  File "./import_package.py", line 1002, in main
    revid_db, bstore, possible_transports=possible_transports)
  File "./import_package.py", line 642, in find_unimported_versions
    possible_transports=possible_transports)
  File "./icommon.py", line 1339, in get_branch
    possible_transports=possible_transports, readonly=readonly)
  File "./icommon.py", line 1503, in get_branch_parts
    dir = br_from.bzrdir.sprout(local_location)
  File "/home/andrew/code/bzr/bzrlib/controldir.py", line 375, in sprout
    create_tree_if_local=create_tree_if_local)
  File "/home/andrew/code/bzr/bzrlib/cleanup.py", line 131, in run
    self.cleanups, self.func, self, *args, **kwargs)
  File "/home/andrew/code/bzr/bzrlib/cleanup.py", line 165, in _do_with_cleanups
    result = func(*args, **kwargs)
  File "/home/andrew/code/bzr/bzrlib/controldir.py", line 416, in _sprout
    result_repo.fetch(source_repository, fetch_spec=fetch_spec)
  File "/home/andrew/code/bzr/bzrlib/repository.py", line 1790, in fetch
    find_ghosts=find_ghosts, fetch_spec=fetch_spec)
  File "/home/andrew/code/bzr/bzrlib/decorators.py", line 194, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/home/andrew/code/bzr/bzrlib/repository.py", line 3514, in fetch
    find_ghosts=find_ghosts)
  File "/home/andrew/code/bzr/bzrlib/fetch.py", line 76, in __init__
    self.__fetch()
  File "/home/andrew/code/bzr/bzrlib/fetch.py", line 103, in __fetch
    self._fetch_everything_for_search(search_result)
  File "/home/andrew/code/bzr/bzrlib/fetch.py", line 131, in _fetch_everything_for_search
    stream, from_format, [])
  File "/home/andrew/code/bzr/bzrlib/repository.py", line 4243, in insert_stream
    hint = self.target_repo.commit_write_group()
  File "/home/andrew/code/bzr/bzrlib/repository.py", line 1652, in commit_write_group
    result = self._commit_write_group()
  File "/home/andrew/code/bzr/bzrlib/repofmt/pack_repo.py", line 2333, in _commit_write_group
    hint = self._pack_collection._commit_write_group()
  File "/home/andrew/code/bzr/bzrlib/repofmt/pack_repo.py", line 2171, in _commit_write_group
    "Cannot add revision(s) to repository: " + problems_summary)
bzrlib.errors.BzrCheckError: Internal check failed: Cannot add revision(s) to repository: missing referenced chk root keys: [StaticTuple('sha1:53f99dc4dd7e3d7d3ed9f26f6994c9b6c8554681',)]

This is with current versions of lp:udd (plus a small hack to workaround bug 718569) and lp:bzr.

Disregard my last comment (#28), that failure turns out to be due to the already-broken data on Launchpad. I still can't reproduce this problem in a local run of the importer isolated from the branches on Launchpad. I've tried arranging my localbranches/ directory to have the branches stacked upon each other similarly to how they are on Launchpad, but that doesn't reproduce it. I still have no plausible theory for how this system caused the same randomly generated revision ID to be associated with different tree contents.

The list of currently affected imports appears to be:

http://package-import.ubuntu.com/status/dsdo.html
http://package-import.ubuntu.com/status/ayaspell-dic.html
http://package-import.ubuntu.com/status/commons-daemon.html
http://package-import.ubuntu.com/status/docbook-xml.html
http://package-import.ubuntu.com/status/igaelic.html
http://package-import.ubuntu.com/status/libcommons-net-java.html
http://package-import.ubuntu.com/status/icu.html
http://package-import.ubuntu.com/status/autofs.html
http://package-import.ubuntu.com/status/libzip.html

At least that's shorter than the original list.

In the absence of a better idea, I suggest we try deleting those imports again, verifying that the deletions really worked this time (i.e. that those branches are completely absent from Launcphad), then reimport, and cross our fingers.

On Fri, 18 Feb 2011 04:57:29 -0000, Andrew Bennetts <email address hidden> wrote:
> In the absence of a better idea, I suggest we try deleting those imports
> again, verifying that the deletions really worked this time (i.e. that
> those branches are completely absent from Launcphad), then reimport, and
> cross our fingers.

Hi Andrew,

Thanks for investigating. This sounds like the right thing to do to me.

Seeing what I did to attempt this last time, would you be able to
attempt it yourself this time, or would you like assistance from me
again?

Thanks,

James

James Westby wrote:
> Thanks for investigating. This sounds like the right thing to do to me.

Great, thanks for second opinion.

> Seeing what I did to attempt this last time, would you be able to
> attempt it yourself this time, or would you like assistance from me
> again?

I'd like to, but I think I need at least a bit of assistance to get
started: I don't recall where to find the script you used to do the
deletes, and a look around on jubany and your branches on Launchpad
doesn't show any obvious candidates.

On Mon, 21 Feb 2011 02:53:32 -0000, Andrew Bennetts <email address hidden> wrote:
> I'd like to, but I think I need at least a bit of assistance to get
> started: I don't recall where to find the script you used to do the
> deletes, and a look around on jubany and your branches on Launchpad
> doesn't show any obvious candidates.

Sorry, I thought it was committed. You'll now find
delete_branches_from_lp.py in lp:udd. You'll have to run it as someone
that is authorised to delete those branches, so either on jubany, or as
a LOSA.

Thanks,

James

delete_branches_from_lp.py could not delete these branches, apparently due to differences in behaviour of the oldish launchpadlib on jubany vs. current versions:

 * the login_with didn't exist. I changed it to use get_lp from icommon.py
 * "branch == other_branch" always evaluated as false, so no series links were ever detected or removed. I changed it to "if other_branch is not None and branch.unique_name == other_branch.unique_name".
 * lp_delete doesn't exist. I made an lp_delete function (copied from the lp_delete method in current versions) and used that instead.

With those hacks in place it appears to be deleting those branches now.

The final wrinkle appears to be that these packages had their sid branch stacked on their squeeze branch, but the script tried to delete squeeze first. A second run of the script cleans that up though.

All the BzrCheckErrors have now been cleared!

Most of the fresh imports now appear to be working fine. icu as mentioned previously is failing with an (apparently) unrelated error shared by some other imports.

This bug is, finally, fixed.

I'm still disturbed about how that corruption ever occurred, but at this point I think it's likely to remain an unsolvable mystery. C'est la vie.