MIR for python-dateutil

I am not sure if one should confirm MIR bugs, but this definitely needs some attention. It blocks elisa.

We went through some effort to eliminate all copies of the Olson time zone database in python-tz, postgresql, and a couple of other places, to use "tzdata" everywhere (which we keep up to date in stable releases). python-dateutil is a step backwards again, since it yet again contains a (very old) copy of that database (see http://bugs.debian.org/416204).

Does eliza use the time zone information in python-dateutil? If so, could it either use python-tz for that (which does the right thing), or can python-dateutil be fixed to use the system time zone information?

#ubuntu-devel:
10:38 < lool> pitti: Re: python-dateutil MIR; there's an embedded storm copy in
              elisa which is the code which relies on python-dateutil
10:38 < lool> I don't know how easy it would be for storm to move to python-tz;
              I'll try to ask them, but it's unlikely that it happens in time
              for intrepid

#storm:
10:41 < lool> Hey folks
10:41 < lool> The elisa packages embed a copy of storm because storm in
              intrepid is a bit too old (in particular, they need twisted
              integration)
10:42 < lool> I noticed that storm requires python-dateutil in this embedded
              copy, but this is in universe
10:42 < lool> So I asked for its promotion to Ubuntu main; bug #271680; the
              problem is that we killed copies of the tz information in all
              packages in main, and python-dateutil would add a copy to main
              again
10:42 < mup> Bug #271680: MIR for python-dateutil <python-dateutil
             (Ubuntu):Incomplete> <https://launchpad.net/bugs/271680>
10:43 < lool> So pitti asked whether it would be possible to use python-tz
              instead of something which doesn't embed tz info

#storm:
10:46 < therve> lool: it's unlikely to happen before intreprid release
10:46 < therve> intrepid even
10:46 < therve> it may be easier to fix python-dateutil, now?
10:51 < lool> Probably

#ubuntu-devel:
10:50 < pitti> lool: so eliza actually uses the time zone db in python-dateutil?
10:50 < pitti> lool: I guess it's much easier in the end to make p-dateutil use
               the system tzdata than trying to convert storm
10:50 < lool> pitti: elisa uses storm which uses dateutil for db dates I guess
10:51 < lool> pitti: Got the same comment on #storm that it'd be simpler to fix
              python-dateutil

Guess someone, hopefully not me, needs to sit down and make python-dateutil parse tzdata.

I have a working patch now to make it use the system tzdata. I'll forward it to upstream and Debian.

Hi Gustavo,

in http://bugs.debian.org/416204 and
https://bugs.launchpad.net/bugs/271680 we requested that the
python-dateutil package in Debian and Ubuntu should use the system
tzdata files instead of the shipped obsolete copy. This will keep a
single reference of the pretty volatile tzdata (which is updated
regularly in stable releases) instead of spreading it over multiple
packages.

I made a patch which prefers /usr/share/zoneinfo/ over the internal
tarball. In the Ubuntu package (just uploaded to intrepid) I don't
ship the tarball any more, because we can rely on tzdata being
present. The upstream patch merely prefers looking there first.

Deleting the tarball breaks the test suite, and applying the patch
makes it work again. Also, the change is pretty unintrusive, so I
think I didn't screw it up too much.

Thank you for considering,

Martin

--
Martin Pitt | http://www.piware.de
Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org)

python-dateutil (1.4-1ubuntu1) intrepid; urgency=low

  * debian/patches/system_tzdata.diff: Use system tzdata instead of builtin
    obsolete copy of the Olson timezone database. (LP #271680, Debian #416204)
  * debian/rules: Do not ship the obsolete tzdata tarball.
  * debian/rules: Run the test suite during build, have it fail the build if
    it fails.

 -- Martin Pitt <email address hidden> Mon, 29 Sep 2008 12:07:52 +0200

Uploaded, sent to Debian, and Gustavo.

Thanks a lot Martin!

Hello Martin,

> in http://bugs.debian.org/416204 and
> https://bugs.launchpad.net/bugs/271680 we requested that the
> python-dateutil package in Debian and Ubuntu should use the system
> tzdata files instead of the shipped obsolete copy. This will keep a
> single reference of the pretty volatile tzdata (which is updated
> regularly in stable releases) instead of spreading it over multiple
> packages.
>
> I made a patch which prefers /usr/share/zoneinfo/ over the internal
> tarball. In the Ubuntu package (just uploaded to intrepid) I don't
> ship the tarball any more, because we can rely on tzdata being
> present. The upstream patch merely prefers looking there first.

dateutil has always preferred the installed timezone information when
it is available. The documented way to ask for a timezone is via the
dateutil.tz.gettz() function, which will process the system's timezone
and give precedence to it. The embedded copy is only used if it is
available, and if the system one isn't found.

> Deleting the tarball breaks the test suite, and applying the patch
> makes it work again. Also, the change is pretty unintrusive, so I
> think I didn't screw it up too much.

The patch changes the behavior of dateutil.zoneinfo, which is the
interface for the embedded copy. It's semantically inappropriate to
make it look for information in the system, since its only purpose is
indeed being the embedded copy API, and the generic interface will
already look for the system timezone information before considering
the embedded one.

It's natural that the tests break if the timezone file isn't found,
since the unittests verify that the embedded functionality actually
works. That doesn't mean that the embedded copy has to be shipped,
though. For Ubuntu, I agree that it's more appropriate to not package
the tarball file, and simply add a requirement on the package to the
timezone information (which I believe is already installed by default
anyway).

Note, with the current python-dateutil:

>>> import dateutil.tz
>>> dateutil.tz
<module 'dateutil.tz' from '/var/lib/python-support/python2.5/dateutil/tz.pyc'>
>>> dateutil.tz.gettz("Brazil/East")
tzfile('/usr/share/zoneinfo/Brazil/East')

Approved, promoted

Hi Gustavo,

Gustavo Niemeyer [2008-09-29 11:41 -0000]:
> dateutil has always preferred the installed timezone information when
> it is available. The documented way to ask for a timezone is via the
> dateutil.tz.gettz() function, which will process the system's timezone
> and give precedence to it. The embedded copy is only used if it is
> available, and if the system one isn't found.

Oh, thanks for the explanation. The patch is unnecessary then.

> The patch changes the behavior of dateutil.zoneinfo, which is the
> interface for the embedded copy. It's semantically inappropriate to
> make it look for information in the system, since its only purpose is
> indeed being the embedded copy API, and the generic interface will
> already look for the system timezone information before considering
> the embedded one.

Right, I agree. That wasn't obvious when searching for the place where
the tarball was consulted, and the failing test suite made me think
that this was the official interface. Sorry for not having checked
more carefully.

> It's natural that the tests break if the timezone file isn't found,
> since the unittests verify that the embedded functionality actually
> works. That doesn't mean that the embedded copy has to be shipped,
> though. For Ubuntu, I agree that it's more appropriate to not package
> the tarball file, and simply add a requirement on the package to the
> timezone information (which I believe is already installed by default
> anyway).

Right, will do that.

Guido, that means for the Debian package we don't need the patch, just
the packaging changes (plus a dependency to "tzdata").

Thanks,

Martin
--
Martin Pitt | http://www.piware.de
Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org)