hpss<->client should stream inventory deltas for xml to chk inventory situations

Bumping to critical, we really want this for 2.0. The delta serialiser has landed.

So the plan for implementing inventory-deltas during fetch is:

 - define a new record storage_kind, "inventory-delta". get_bytes_as('fulltext') should fail for this record.
 - update _extract_and_insert_inventories in StreamSink to check each record to see if it is an inventory delta, if so:
   - look up the parent inventory in the target
   - apply the delta
   - add_inventory
   - (otherwise, use get_bytes_as('fulltext') as before)
 - update StreamSource to generate deltas rather than fulltexts when going from chk->non-chk, or vice-versa. (Or maybe just cross format in general?)
 - update the network protocol
   - serialise & deserialise inventory-delta records, e.g. update NetworkRecordStream. (bzrlib.inventory_delta already exists and will do most of the heavy lifting)
   - perhaps need new insert_stream and get_stream verbs to gracefully handle backwards compatibility.

On Mon, 2009-06-29 at 01:28 +0000, Andrew Bennetts wrote:
>
> - look up the parent inventory in the target
> - apply the delta
> - add_inventory

This is one call: repository.add_inventory_by_delta

> - (otherwise, use get_bytes_as('fulltext') as before)
> - update StreamSource to generate deltas rather than fulltexts when
> going from chk->non-chk, or vice-versa. (Or maybe just cross format
> in general?)

Cross format in general IMO.

> - update the network protocol
> - serialise & deserialise inventory-delta records, e.g. update
> NetworkRecordStream. (bzrlib.inventory_delta already exists and will
> do most of the heavy lifting)
> - perhaps need new insert_stream and get_stream verbs to gracefully
> handle backwards compatibility.

Likely yes. We should try to do progress at the same time (by which I
mean before considering the verb stable in trunk). Or something.

-Rob

John asks: how does this fit in with InterDifferingSerializer? Is code in this place actually active?

On Mon, 2009-06-29 at 23:11 +0000, Martin Pool wrote:
> John asks: how does this fit in with InterDifferingSerializer? Is code
> in this place actually active?

There is another critical bug open about disabling IDS for network push
and pull. Related to that was doing packs automatically on cross-format
fetch (which is done now).

-Rob

Landed!