RemoteStreamSource does not stream from fallback locations of fallback locations

$ bzr branch lp:~salgado/canonical-identity-provider/import-person-from-lp-registry-3
bzr: ERROR: exceptions.ValueError: A request was made for key: ('sha1:f4bb3cbd77a7fa6ed7b4a9df79d4e3b69c0cc1c6',), but that content is not available, and the calling code does not handle if it is missing.

Traceback (most recent call last):
  File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 835, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 1030, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 647, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.6/dist-packages/bzrlib/builtins.py", line 1236, in run
    source_branch=br_from)
  File "/usr/lib/python2.6/dist-packages/bzrlib/bzrdir.py", line 1194, in sprout
    hardlink=hardlink)
  File "/usr/lib/python2.6/dist-packages/bzrlib/bzrdir.py", line 1598, in create_workingtree
    accelerator_tree=accelerator_tree, hardlink=hardlink)
  File "/usr/lib/python2.6/dist-packages/bzrlib/workingtree_4.py", line 1408, in initialize
    wt.set_parent_trees(parents_list, allow_leftmost_as_ghost=True)
  File "/usr/lib/python2.6/dist-packages/bzrlib/mutabletree.py", line 52, in tree_write_locked
    return unbound(self, *args, **kwargs)
  File "/usr/lib/python2.6/dist-packages/bzrlib/workingtree_4.py", line 1105, in set_parent_trees
    dirstate.set_parent_trees(real_trees, ghosts=ghosts)
  File "/usr/lib/python2.6/dist-packages/bzrlib/dirstate.py", line 2429, in set_parent_trees
    for path, entry in tree.inventory.iter_entries_by_dir():
  File "/usr/lib/python2.6/dist-packages/bzrlib/inventory.py", line 872, in iter_entries_by_dir
    for child_name, child_ie in sorted(cur_dir.children.iteritems()):
  File "/usr/lib/python2.6/dist-packages/bzrlib/inventory.py", line 2114, in children
    key_filter=[(self.file_id,)]):
  File "/usr/lib/python2.6/dist-packages/bzrlib/chk_map.py", line 998, in iteritems
    for node, node_filter in self._iter_nodes(store, key_filter=key_filter):
  File "/usr/lib/python2.6/dist-packages/bzrlib/chk_map.py", line 1141, in _iter_nodes
    bytes = record.get_bytes_as('fulltext')
  File "/usr/lib/python2.6/dist-packages/bzrlib/versionedfile.py", line 182, in get_bytes_as
    % (self.key,))
ValueError: A request was made for key: ('sha1:f4bb3cbd77a7fa6ed7b4a9df79d4e3b69c0cc1c6',), but that content is not available, and the calling code does not handle if it is missing.

The backtrace shows that a CHK page is missing locally, after the branch is completed.

The page is being requests from the parent_id_basename index.

The problem doesn't occur when salgado branches within his shared repo.

One hypothesis is a height-change corner case in the CHKMap, which is plausible as this tree has shrunk by an order of magnitude - most of the files and directories were removed.

Now, CHKStreamSource does inclue the parent_id_basename CHKMap, and that code is meant to error on any missing key references. Just pushing salgado's branch to /tmp/foo results in an apparently good branch - or at least, one that can build its inventory.

When I branch from this repo, I get dumped down to VFS calls:
bzr -Dhpssdetail -Dhpss -Dhpssvfs branch lp:~salgado/canonical-identity-provider/test4
VFS Repository access triggered
  File "/home/robertc/bin/bzr", line 142, in <module>
...
  File "/home/robertc/source/baz/bzr-test-fixes/bzrlib/fetch.py", line 76, in __init__
    self.__fetch()
  File "/home/robertc/source/baz/bzr-test-fixes/bzrlib/fetch.py", line 102, in __fetch
    self._fetch_everything_for_search(search)
  File "/home/robertc/source/baz/bzr-test-fixes/bzrlib/fetch.py", line 139, in _fetch_everything_for_search
    stream, from_format, resume_tokens)
  File "/home/robertc/source/baz/bzr-test-fixes/bzrlib/repository.py", line 3979, in insert_stream
    return self._locked_insert_stream(stream, src_format, is_resume)
  File "/home/robertc/source/baz/bzr-test-fixes/bzrlib/repository.py", line 4008, in _locked_insert_stream
    for substream_type, substream in stream:
  File "/home/robertc/source/baz/bzr-test-fixes/bzrlib/repository.py", line 4221, in get_stream_for_missing_keys
    vf = getattr(self.from_repository, substream_kind)
  File "/home/robertc/source/baz/bzr-test-fixes/bzrlib/remote.py", line 1564, in signatures
    self._ensure_real()
  File "/home/robertc/source/baz/bzr-test-fixes/bzrlib/remote.py", line 733, in _ensure_real
    ''.join(traceback.format_stack()))

I suspect this is related.

So, there are a few things here.

Something has changed, I think, but I'm having trouble identifying exactly what.

Firstly, the damage is this: we're ending up in get_stream_for_missing_keys which is not defined on RemoteStreamSource. CHKStreamSource doesn't override it, so this isn't the problem. However there is a bug that if CHKStreamSource *did* define it this wouldn't work. Removing the CHK specific source will fix that, and Andrew and I want to do that - or at least make it a requirement that the common code path *always* work even if it might be slower.

To end up in get_stream_for_missing_keys, the missing_parents_chain in the RemoteSourceStream has to be exiting successfully but not delivering everything.

And to not get an error from the target, some validation code must be missing or broken in the StreamSink code path. I'm going to focus on why the error isn't caught earlier, for now.

We do request a stream from the trunk, but it only gives back a vanishingly small number of revisions.

The stacked branch has 1 signed commit, with one chk page, one inventory and one revision. This implies no new or removed paths in the tip commit, otherwise two chk pages (at minimum) would be needed.

The stream from trunk returns 14 revisions, -
signatures 1
signatures 1
signatures 1
signatures 1
signatures 1
signatures 1
signatures 8
revisions 1
revisions 1
revisions 1
revisions 1
revisions 1
revisions 1
revisions 8
inventories 1
inventories 1
inventories 1
inventories 1
inventories 1
inventories 1
inventories 8
chk_bytes 1
chk_bytes 1
chk_bytes 1
chk_bytes 1
chk_bytes 8
chk_bytes 72
chk_bytes 1
chk_bytes 6
chk_bytes 1
chk_bytes 1
chk_bytes 1
chk_bytes 1
chk_bytes 1
chk_bytes 1
chk_bytes 1
chk_bytes 1
chk_bytes 1
chk_bytes 1
chk_bytes 1
chk_bytes 1
chk_bytes 1
chk_bytes 1
chk_bytes 20

Earlier errors: the missing keys request does not include CHK pages, and get_stream_for_missing_keys doesn't include the CHK pages the inventory references. I'm filing a bug on that first, as thats causing corruption of any stacked or stacked-like scenario.

Fix is in 1.18