rules for amavisd-new are not catching all mails

gavin@robin ~> apt-cache policy amavisd-new
amavisd-new:
  Installed: 1:2.4.2-6.2ubuntu1
  Candidate: 1:2.4.2-6.2ubuntu1
  Version table:
 *** 1:2.4.2-6.2ubuntu1 0
        500 http://ie.archive.ubuntu.com gutsy/universe Packages
        100 /var/lib/dpkg/status
gavin@robin ~> apt-cache policy logcheck-database
logcheck-database:
  Installed: 1.2.61ubuntu0.1
  Candidate: 1.2.61ubuntu0.1
  Version table:
 *** 1.2.61ubuntu0.1 0
        500 http://ie.archive.ubuntu.com gutsy-updates/main Packages
        100 /var/lib/dpkg/status
     1.2.61 0
        500 http://ie.archive.ubuntu.com gutsy/main Packages
gavin@robin ~> cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=7.10
DISTRIB_CODENAME=gutsy
DISTRIB_DESCRIPTION="Ubuntu 7.10"

I have a very similar problem with hardy beta (logcheck-database v1.2.63) - it looks like the log files have changed again. The attached patch works for me (though I only had 'CLEAN' emails to test with)

There are a couple more changes that I didn't catch in the last patch - there's now an IPv6 identifier, and the 'Hits:' value may not have a '-' prefix.

logcheck files are shipped with amavisd-new itself.

$ apt-file list amavisd|grep logcheck
amavisd-new: /etc/logcheck/ignore.d.server/amavisd-new
amavisd-new: /etc/logcheck/violations.ignore.d/amavisd-new

What file is your patch against?
The bug then should get moved over to amavisd-new probably, if you're not using obsolete rule files or something similar.

Sorry, I forgot diff -u :)

I've re-attached the patch (to /etc/logcheck/ignore.d.server/amavisd-new), based on the original from amavisd-new v2.5.3.

Please have a look at the Hardy package and see if the same problem exists
there.

The patch was made against amavisd-new_2.5.3-1ubuntu3 - is this what you meant?

Would you please show how it looks once it's fixed. For a non-logcheck user it's kind of hard to tell exactly what you're missing.

Sorry I took my eye off the ball a little on this one.

As I'm sure you know logcheck uses a set of standard regexps to filter out the "normal" log messages and send the admin the rest. amavisd-new's filters are not quite catching all the normal mail now.

The situation seems to have changed a little since hardy (sorry I didn't test the mail server prior to release but I guess it would be worth fixing for the .1 release of hardy). An example mail delivery log which escapes filtering for me is:

May 22 23:50:26 robin amavis[2067]: (02067-07) Passed CLEAN, LOCAL [127.0.0.1] [195.113.31.123] <email address hidden> -> <gavin@localhost>, Message-ID: <email address hidden>, mail_id: 2yKeML25dBUl, Hits: -, size: 4325, queued_as: 0B198205BA, 288 ms

The existing filter set in hardy is this:

^\w{3} [ :[:digit:]]{11} [._[:alnum:]-]+ amavis\[[[:digit:]]+\]: \([-[:digit:]]+\) Passed CLEAN,( \[[.:[:xdigit:]]+\]){0,2} <[^>]*> -> <[^>]*>(,<[^>]*>)*, Message-ID: <[^>]+>( \((added by[^)]+|sfid-[_[:xdigit:]]+)\))?,( Resent-Message-ID: <[^>]+>,)? mail_id: [-+[:alnum:]]+, Hits: (-[.[:digit:]]*)+, size: [[:xdigit:]]+, queued_as: [[:xdigit:]]+ OK id=[-[:alnum:]]+, [[:digit:]]+ ms$

so it would seem that the " OK id=[-[:alnum:]]+" is now optional or perhaps has even been dropped (I never see it in logs now).

If it's optional, I suggest the filter change to:

^\w{3} [ :[:digit:]]{11} [._[:alnum:]-]+ amavis\[[[:digit:]]+\]: \([-[:digit:]]+\) Passed CLEAN,( \[[.:[:xdigit:]]+\]){0,2} <[^>]*> -> <[^>]*>(,<[^>]*>)*, Message-ID: <[^>]+>( \((added by[^)]+|sfid-[_[:xdigit:]]+)\))?,( Resent-Message-ID: <[^>]+>,)? mail_id: [-+[:alnum:]]+, Hits: (-[.[:digit:]]*)+, size: [[:xdigit:]]+, queued_as: [[:xdigit:]]+( OK id=[-[:alnum:]]+)?, [[:digit:]]+ ms$

if it's not optional, I suggest we just remove it:

^\w{3} [ :[:digit:]]{11} [._[:alnum:]-]+ amavis\[[[:digit:]]+\]: \([-[:digit:]]+\) Passed CLEAN,( \[[.:[:xdigit:]]+\]){0,2} <[^>]*> -> <[^>]*>(,<[^>]*>)*, Message-ID: <[^>]+>( \((added by[^)]+|sfid-[_[:xdigit:]]+)\))?,( Resent-Message-ID: <[^>]+>,)? mail_id: [-+[:alnum:]]+, Hits: (-[.[:digit:]]*)+, size: [[:xdigit:]]+, queued_as: [[:xdigit:]]+, [[:digit:]]+ ms$

I'm attaching patches (solution-1.patch to make OK optional and solution-2.patch to remove OK) to fix this.

I'm testing solution-2.patch now.

Testing solution-2 above (ie removing the OK field from the end) has removed most of my normal mail logs. However, a few are still slipping through:

May 23 10:10:30 robin amavis[17682]: (17682-03) Passed CLEAN, LOCAL [127.0.0.1] [87.42.170.254] <email address hidden> -> <gavin@localhost>, Message-ID:
+<email address hidden>, mail_id: ci7WDJAqDHG9, Hits: -, size: 2404, queued_as: 06B6D205BC, 313 ms

apparently due to the word ", LOCAL" near the start. I'm going to send updated patches to take account of this.

bah! an extra comma went in there where it shouldn't updated patches attached.

OK. If you can give me a test procedure that I can use to replicate the problem and then verify that it's corrected with the new package, I should be able to get this uploaded.

I guess a possible test procedure would be:

1. install and configure amavisd-new (I presume as maintainer you probably have servers
    running it).
2. sudo apt-get install logcheck logcheck-database
3. logcheck will begin hourly checks and mail output to root. If you want it to go elsewhere,
    configure /etc/logcheck/logcheck.conf
4. Send a few mails through so that logs appear.
5. Running:
       sudo -u logcheck logcheck -o -t
    You should get output including those logs for successful mail delivery (which is not desirable).
6. Apply the patches and re-run the above. You should no longer get the output from those mails.

Is this sufficient or do you need something more automatic?

Gavin

That should work. Thanks.

I'll include this with the new amavisd-new 2.6 in Intrepid.

This bug was fixed in the package amavisd-new - 1:2.6.0-1ubuntu1

---------------
amavisd-new (1:2.6.0-1ubuntu1) intrepid; urgency=low

  * Merge from Debian unstable. Remaining Ubuntu changes:
    - Updated logcheck/ignore.d.server/amavisd-new (LP: #188754)
    - Add libmail-dkim-perl (>= 0.31) to recommends for new DKIM based
      whitelisting
    - Added 81_fqdn-warning.dpatch so that correct config file path for
      Debian/Ubuntu is displayed if it needs to be set manually
      - Renumbered from 71 to put it after the new Debian patch
      - Added patch description
    - Add commented out $myhostname definition to debian/etc/conf.d05-node_id
      so that it will be easier for people who don't know Perl syntax to set
    - Updated debian/patches/40_fix_paths.dpatch to use the new path
      (/var/lib) for amavis-release
    - Remove amavisd-new-milter package for transition to Main
      - Remove libmilter-dev from build-dep
      - Comment out debian/rules related to amavisd-new-milter
    - Change maintainer to Ubuntu Core Developers

amavisd-new (1:2.6.0-1) unstable; urgency=low

  * New upstream version
  * Updated debconf translations:
    - German. Closes: #448244
    - Finnish. Closes: #480503
    - Italian. Closes: #480508
    - Russian. Closes: #480622
    - Basque. Closes: #481550
    Thanks to all translators and Christian Perrier for their work.

 -- Scott Kitterman <email address hidden> Wed, 11 Jun 2008 21:07:08 -0400