Integration of Mailman and htdig for archiving

The htdig-2.0.6-03.patch version of the patch makes some
previously hard-coded things configurable and enhances the
capability to run the htdig searches and indexing on a
different machine to the one delivering Mailman and
Mailman's web UI.

This patch should also apply without problems to Mm 2.0.7

This patch should also apply without problems to MM 2.0.8

The htdig-2.0.8-0.1.patch version of the patch resolves a problem that can
arise with htdig indexing if the
web_page_url for a list uses other than the http addressing (some folks
want to use https). While specified
as for MM 2.0.8 the revised patch should work OK with 2.0.7, 2.0.6 and
probably back as far as 2.0.3. If
you do not have the requirement for using other than http addressing in
you lists web_page_urls it probably
isn't worth the trouble of upgrading to this patch level.

htdig-2.1a3-0.1.patch is a revised version of the patch that is compatible
with the code published in
mailman-2.1a3.tgz on sourceforge.

The only known deficiency is that the non-English versions of files under
$build/templates still contain text
in English and need translations I cannot do. Also the necessary pygettext
activity and subsequent
translations in files under $build/messages remain to be done.

htdig-2.1cvs-20011217.patch is a revised version of the
patch that is compatible with the code published in mailman
CVS on sourceforge as 11:50 GMT 17 Dec 2001

The only known deficiency is that the non-English versions
of files under $build/templates still contain text in
English and need translations I cannot do. Also the
necessary pygettext activity and subsequent translations in
files under $build/messages remain to be done.

htdig-2.1cvs-20020306.patch is a revised version of the patch that is
compatible with the code published in
mailman CVS on sourceforge as 12:30 GMT 6 Mar 2002

Known deficiency is that the non-English versions of files under
$build/templates still contain text in English
and need translations I cannot do. Also the necessary pygettext activity
and subsequent translations in
files under $build/messages remain to be done.

htdig-2.0.9-0.1.patch is a revised version of the patch
that is compatible with MM 2.0.9

htdig-2.0.10-0.1.patch is a revised version of the patch
that is compatible with MM 2.0.10

htdig-2.0.11-0.1.patch is a revised version of the patch that
is compatible with MM 2.0.11

This version removes an incompatibility with Python 2.2
which caused warning messages to be generated when any
of the family cron/nightly_htdig scripts were run.

Some guidance on file access permissions for some htdig
database files needed by rundig have been added to
installation notes.

Other attachments

htdig-2.0.12-0.1.patch is a revised version of the patch that
applies without complaint to MM 2.0.12.

It also add a facility for adding site wide htdig configuration
attributes to all list specific htdig configuration files.

Do not use htdig-2.0.12-0.1.patch there is an error in it.
Use htdig-2.0.12-0.2.patch instead

htdig-2.0.13-0.1.patch is purely cosmetic to get no mumble
application to MM 2.0.13

htdig-2.1b2-0.1.patch is a revised version of the patch
that is compatible with MM 2.1b2

htdig-2.0.13-0.2.patch just adds a GPL notice to the patch

I've sent Richard some comments off-line about this patch.

Meta comments: the 2.0.x patches can't be officially
supported, but I'm going to create an unofficial patches
page off the wiki for where the 2.0 patches can be migrated.

I think this patch set is too big for MM2.1, but if it's
cleaned up as per my private message, let's re-evaluate it
for MM2.2 (or 3.0).

htdig-2.1b3-0.1.patch is a revised version of the patch that is
compatible with MM 2.1b3

htdig-2.1b3-0.2.patch corrects a dumb syntax error in htdig-
2.1b3-0.1.patch which will typically show up as logged errors
in the operation of the ArchRunner qrunner at line 721 of
HyperArch.py

Just rebuilt MM as 2.1b3 with htdig.
Upgraded lists which had htdig before work fine
New lists give the obvious error:
  Unable to read word database file
  Did you run htmerge?
Running the cronjob doesn't fix as it used to, message is:
  Output from command /usr/bin/python -
S /usr/local/mailman/cron/nightly_htdig ..

Traceback (most recent call last):
  File "/usr/local/mailman/cron/nightly_htdig", line 153, in ?
    main()
  File "/usr/local/mailman/cron/nightly_htdig", line 118, in
main
    file(rundig_run_file, 'w').close()
NameError: global name 'file' is not defined

The archive/htdig folder only contains the xx.conf file, but no
db.xx files

If I copy in db.xx files from another list then the problem goes
away (except I've now got an invalid set of references!)

Is this my elementary error or is it more sinister?!

Got a workaround!

The line referred to in the traceback:
 file(rundig_run_file, 'w').close()
is used to create a 'rundig_last_run' file of lenght 0 bytes
Creating this manually (or copying it) means the line isn't
called and everything seems to work.

Either file() is not a valid function call or my python is broken -
I'm not literate enough in python to know the answer though!

htdig-2.1b3-0.3.patch removes use of the file() function, used
instead of the open() function, in three cron scripts added by
the patch. Use of the file() function created an unnecessary
dependency on Python 2.2

htdig-2.1b4-0.1.patch is a revised version of the patch that is
compatible with MM 2.1b4

htdig-2.1b5-0.1.patch is a revised version of the patch that is
compatible with MM 2.1b5

htdig-2.0.13-0.3.patch corrects a minor typo in text appearing
in the list TOC after the patch is applied.

htdig-2.1b6-0.1.patch is a revised version of the patch that is
compatible with MM 2.1b6

htdig-2.1-0.1.patch is a revised version of the patch that is
compatible with MM 2.1

mailer-2.0.13-0.4.patch improves the content type and
security handling by htdig.py for MM 2.0.13 version of patch

Uploaded wrong file mailer-2.0.13-0.4.patch on last attempt.

Should have been htdig-2.0.13-0.4.patch which improves the
content type and security handling by htdig.py for MM 2.0.13
version of patch.

Please ignore mailer-2.0.13-0.4.patch file

htdig-2.1-0.2.patch corrects a bug in htdig.py and deals with
an adverse interaction between htdig.py and a bug in
$prefix/scripts/driver (see #668685 for a patch to fix this).

It also improves the content type and security handling by
htdig.py for MM 2.1 version of patch

htdig-2.1-0.3.patch corrects yet another bug in htdig.py. Hope
that all of them!

Stops use of obsolete config variable DEFAULT_HOST in
several files.

It seems it is possible, if this patch is installed, for a list's
htdig conf file and the list specific htdig index db files to be
read directly through the web interface for list archives.

Even if this patch isn't installed it seems a list's pipermail.pck
file can also be read directly through the web interface for list
archives.

This seems to be true for accesses via /pipermail for public
lists and via /mailman/private for private lists.

The problem does not occur for htdig search results
accessed via /mailman/htdig as the htdig.py script is more
protective than private.py

Broadly speaking the data affected is availble to a user in
normal operation which is why I do not consider the issue to
be a security breach as such.

Adding the following RewriteRule to Apache's httpd.conf
prevents the situation, assuming you got the RewriteEngine
On:

RewriteRule ^(/pipermail/.*)/(pipermail.pck|htdig/[^/]*)$
$1/index.html [F]

RewriteRule ^(/mailman/private/.*)/(pipermail.pck|htdig/[^/]*)$
$1/index.htm
l [F]

You could, of course, substitute an R flag for the F flag on the
RewriteRules and be more hacker friendly.

htdig-2.1.1-0.1.patch.gz introduces no functional change but
applies without offset warnings to MM 2.1.1

htdig-2.1.1-0.2.patch.gz close a security exploit which allows
leakage of information held in htdig's per-list search indexes
to users not authorized to view private list archives.

Read file INSTALL.htdig-mm installed by this patch for details
and instructions for upgrading MM installations using earlier
versions of this patch

htdig-2.1.1-0.3.patch.gz fixes a fault when mmsearch.py is
rasing an excpetion because it has had a problem running
htsearch

htdig-2.1.1-0.4.patch.gz fixes a problem with mmsearch
handling multi-page search results from htsearch.

htdig-2.1.2-0.1.patch.gz is a revised version for MM 2.1.2
compatibility.

It also incoporates a previosuly unpublished change to
overcome a potential problem with htdig excluced urls - see
the INSTALL.htdig-mm file for more information

htdig-2.1.2-0.2.patch.gz corrects error in file uploaded as
htdig-2.1.2-0.1.patch.gz. Sorry for any inconvenience.

htdig-2.1.2-0.3.patch.gz adds some minor performance
improvement in template handling in MM 2.1.2

You should consider also applying this bug-fis patch:

[ 730769 ] template access hierarchy is broken

http://sourceforge.net/tracker/index.php?
func=detail&aid=730769&group_id=103&atid=100103

htdig-2.1.2-0.3.patch.gz corrects an error in 2 scripts,
mmsearch.py and remote_mmsearch, which caused an
exception if list archives were being accessed via HTTPS and
a search was performed.

last comment should have read:

htdig-2.1.2-0.4.patch.gz corrects an error in 2 scripts,
mmsearch.py and remote_mmsearch, which caused an
exception if list archives were being accessed via HTTPS and
a search was performed.

htdig-2.1.3-0.1.patch is a MM 2.1.3 compatible version of
the patch

htdig-2.1.3-0.3.patch.gz add minor private archive security
improvements to the patch for MM 2.1.3

htdig-2.1.3-0.4.patch provides minor improvements in
handling of HTTP request handled by htidg.py which lead to
the user receiving an authentication challenge.

htdig-2.1.3-0.5.patch modifies htdig.py and private.py; the
security changes introduced by htdig-2.1.3-0.2 patch to
these scripts incorrectly blocked access to the
listname.mbox/listname.mbox file. The 0.5 revision of the
patch corrects this error. This problem and a suggested fix
were pointed out to me in a private email by Stephan Berndts
stb-mm at spline.de

htdig-2.1.4-0.1.patch is a MM 2.1.4 compatible version of
the patch

htdig-2.1.5-0.1.patch.gz is a MM 2.1.5 compatible version of the patch

htdig-2.1.6-0.1.patch.gz is a MM 2.1.6 compatible version of
the patch

Other attachments

Use htdig-2.1.7-0.1.patch.gz for both MM 2.1.7 and MM 2.1.8

htdig-2.1.9-0.1.patch.gz is a MM 2.1.9 compatible version of the
patch

Originator: YES

File Added: htdig-2.1.10-0.1.patch.gz

Marked this bug as 'Invalid' due to its age and nothing further has been added in a long time. New versions have been released since as well as some underlying stuff in the OS platform itself.

If this bug still affects then please change status back to 'Confirmed'.

It's not a bug, it's a patch, and it's still relevant.

More recent versions of this patch can be found at <http://www.openinfo.co.uk/mm/index.html> and <http://www.msapiro.net/mm/>.