ZODB error when copy-pasting a blob file (zope.file)

A first fix is :

In ZODB/blob.py:85, replace:
        for ref in self.readers+self.writers:
with
        for ref in (self.readers or []) + (self.writers or []):

But that's not enough, the copied file is empty.

Happens here as well.

There are two actual problems on this:

1. zope.copypastemove pickles/unpickles the blob object without calling it's __setstate__ method. This is unexpected from the Blob class and I can't see immediately what happens in pickle/cPickle that prevents this.
2. The blob class uses its object id and a temporary filename to maintain its state. A simple pickle/unpickle/store routine will turn it into an empty blob. I don't see an obvious solution on this either. It sounds like the blob class would have to get a lot smarter to support this use case.

I'm unassigning myself for now.

Where is the blobfile supposed to be copied?
Should it be copied during the pickling/unpickling of the blob object, by sort some of hook ?

Actually, the pickling/unpickling is initiated in zope.location.pickling.locationCopy,
and I've tried an small hack that writes the new blobfile at the end of this function.

By replacing:

    return unpickler.load()

with:

    copied = unpickler.load()
    if hasattr(copied, '_data') and type(copied._data) is Blob:
        copied._data._create_uncommitted_file()
        targetfile = open(copied._data._current_filename(), 'w')
        sourcefile = loc.openDetached()
        chunk = sourcefile.read(1000000)
        while chunk:
            targetfile.write(chunk)
            chunk = sourcefile.read(1000000)
        targetfile.close()
    return copied

It also need the fix mentionned in my previous comment (blob.py).

What is the status of this? This looks like a limitation in the approach of Zope 3's copyy/paste approach. Basically, a blob is a case of an object that can't be copied through pickling alone. I suspect there's a feature request lurking here. And I have a vague idea that someone "fixed" this.

Now, when zope.copy (formerly zc.copy) pluggable mechanism is integrated into zope.copypastemove, it can be easily fixed using copy hook adapters. I've done that for z3c.blobfile some time ago, please see http://svn.zope.org/z3c.blobfile/trunk/src/z3c/blobfile/copy.py?view=auto

So no further change is needed in ZODB?

Jim

On Sun, Aug 23, 2009 at 8:17 AM, Dan Korostelev<email address hidden> wrote:
> Now, when zope.copy (formerly zc.copy) pluggable mechanism is integrated
> into zope.copypastemove, it can be easily fixed using copy hook
> adapters. I've done that for z3c.blobfile some time ago, please see
> http://svn.zope.org/z3c.blobfile/trunk/src/z3c/blobfile/copy.py?view=auto
>
> --
> ZODB error when copy-pasting a blob file (zope.file)
> https://bugs.launchpad.net/bugs/240381
> You received this bug notification because you are a member of Zope 3
> Developers, which is subscribed to Zope 3.
>

--
Jim Fulton

I guess so.

This seems to have been addressed at the zope level.

Actually this is interesting on the ZODB level. With current discussion about high quality backup strategies on the ZODB development list and the pointer back to this bug I found a solution how this might be solved easily on the ZODB level.

The underlying issue here is that blobs use the OID as the pointer to which file belongs to it. When copying a blob object that piece of data goes away. IMHO the ZODB kinda nurishes the assumption that the content of an object is what is stored in the pickle, so as a consequence if I pickle/unpickle such an object its value should be the same.

Thus th idea of a solution: we could save the OID in the state of the blob object and if that is different from the _p_oid then we know we need to initialise the blob from that object's state, e.g. by providing a new hard link to that file.

Maybe we don't even have to store that marker in the object's value persistently, but I don't know, we'll see.

On Fri, Nov 20, 2009 at 2:49 PM, Christian Theune <email address hidden> wrote:
> Actually this is interesting on the ZODB level. With current discussion
> about high quality backup strategies on the ZODB development list and
> the pointer back to this bug I found a solution how this might be solved
> easily on the ZODB level.
>
> The underlying issue here is that blobs use the OID as the pointer to
> which file belongs to it.

All objects use the oid as a key to where the data are.

> When copying a blob object that piece of data
> goes away.

Also when copying a regular object.

> IMHO the ZODB kinda nurishes the assumption that the content
> of an object is what is stored in the pickle, so as a consequence if I
> pickle/unpickle such an object its value should be the same.

If you buy the premise, which I don't. Copying objects is always tricky
and somewhat application dependent.

> Thus th idea of a solution: we could save the OID in the state of the
> blob object and if that is different from the _p_oid then we know we
> need to initialise the blob from that object's state, e.g. by providing
> a new hard link to that file.
>
> Maybe we don't even have to store that marker in the object's value
> persistently, but I don't know, we'll see.

You haven't fleshed this out enough for me to judge this. You also
haven't really defined copying for that matter. There are many ways to
copy an object and it's not clear what's meant in the context of ZODB,
which is why I view this as an application issue, rather than a ZODB issue.

Perhaps you should come up with some definition of copying in the context of
ZODB and then we can decide if there is a compelling reason to
have such a feature in ZODB.

Given other priorities, I don't think we should try to deal with this.

Jim

--
Jim Fulton