initgroups() fails when using libnss-ldap (but not nscd)

AppArmor profile? Seems clamav had a similar issue recently.

433764

Uninstalled apparmor and repeated test; still failed.

Hi,

could you please run strace on atd and then submit a job:

    strace -f -o strace.txt -p $pid
    echo echo foo | at now

(replace $pid with the process id of atd). This should produce a file strace.txt. Please attach it to this bug report.

Regards,
Ansgar

strace output is attached

Hi,

greenmoss writes:
> strace output is attached
Thanks.

This is the relevant section from atd.c:
   342 if (chdir(ATJOB_DIR) < 0)
   343 perr("Cannot chdir to " ATJOB_DIR);
   344 PRIV_START
   345 nice((tolower((int) queue) - 'a' + 1) * 2);
   346 if (initgroups(pentry->pw_name, pentry->pw_gid))
   347 perr("Cannot delete saved userids");

And the same part in the output from strace:
   522 20241 chdir("/var/spool/cron/atjobs") = 0
   523 20241 setreuid32(1, 0) = 0
   524 20241 setregid32(1, 0) = 0
   525 20241 getpriority(PRIO_PROCESS, 0) = 20
   526 20241 setpriority(PRIO_PROCESS, 0, 2) = 0
   527 20241 getpriority(PRIO_PROCESS, 0) = 18

So far everything looks ok. Now only initgroups() is left:
   528 20241 open("/proc/sys/kernel/ngroups_max", O_RDONLY) = 5

While looking up the groups, suddenly the following happens:
   828 20241 getuid32() = 1
   829 20241 mlock(0xb7348000, 32768) = 0
   830 20241 geteuid32() = 0
   831 20241 setuid32(1) = 0
   832 20241 getuid32() = 1
   833 20241 geteuid32() = 1
   834 20241 setuid32(0) = -1 EPERM (Operation not permitted)

It looks like the NSS module drops privileges?!
Of course, setgroups thus fails:

  2005 20241 setgroups32(2, [0, 512]) = -1 EPERM (Operation not permitted)

You seem to be using the libnss-ldap module. Does at work correctly if
you disable it?

Regards,
Ansgar

I removed libnss-ldap, re-tried the at test, and it worked. So you are correct: libnss-ldap and at do not like each other.

Hi,

Ansgar Burchardt <ansgar@43-1.org> writes:

> While looking up the groups, suddenly the following happens:
> 828 20241 getuid32() = 1
> 829 20241 mlock(0xb7348000, 32768) = 0
> 830 20241 geteuid32() = 0
> 831 20241 setuid32(1) = 0
> 832 20241 getuid32() = 1
> 833 20241 geteuid32() = 1
> 834 20241 setuid32(0) = -1 EPERM (Operation not permitted)

I think I found the suspect: libgcrypt11/1.4.4-2ubuntu2.
The function lock_pool from src/secmem.c contains the following code:

  uid = getuid ();
  [...]
  err = mlock (p, n);
  [...]
  if (uid && ! geteuid ())
    {
      /* check that we really dropped the privs.
       * Note: setuid(0) should always fail */
      if (setuid (uid) || getuid () != geteuid () || !setuid (0))
        log_fatal ("failed to reset uid: %s\n", strerror (errno));
    }

This matches the output from strace above.

(libgcrypt is used via libnss-ldap → openldap → libgnutls → libgcrypt)

Regards,
Ansgar

Hi,

greenmoss <email address hidden> writes:

> I removed libnss-ldap, re-tried the at test, and it worked. So you are
> correct: libnss-ldap and at do not like each other.

You can try to install "nscd" (name service cache daemon). Then the
communication with the LDAP server should be handled by nscd instead of
the atd process. This should avoid the issue with gcrypt messing up the
user ids.

Regards,
Ansgar

I've had nscd installed since before I noticed this problem. So that didn't work for me. My nscd configuration is the Ubuntu default:

#
# /etc/nscd.conf
#
# An example Name Service Cache config file. This file is needed by nscd.
#
# Legal entries are:
#
# logfile <file>
# debug-level <level>
# threads <initial #threads to use>
# max-threads <maximum #threads to use>
# server-user <user to run server as instead of root>
# server-user is ignored if nscd is started with -S parameters
# stat-user <user who is allowed to request statistics>
# reload-count unlimited|<number>
# paranoia <yes|no>
# restart-interval <time in seconds>
#
# enable-cache <service> <yes|no>
# positive-time-to-live <service> <time in seconds>
# negative-time-to-live <service> <time in seconds>
# suggested-size <service> <prime number>
# check-files <service> <yes|no>
# persistent <service> <yes|no>
# shared <service> <yes|no>
# max-db-size <service> <number bytes>
# auto-propagate <service> <yes|no>
#
# Currently supported cache names (services): passwd, group, hosts, services
#

# logfile /var/log/nscd.log
# threads 4
# max-threads 32
# server-user nobody
# stat-user somebody
        debug-level 0
# reload-count 5
        paranoia no
# restart-interval 3600

        enable-cache passwd yes
        positive-time-to-live passwd 600
        negative-time-to-live passwd 20
        suggested-size passwd 211
        check-files passwd yes
        persistent passwd yes
        shared passwd yes
        max-db-size passwd 33554432
        auto-propagate passwd yes

        enable-cache group yes
        positive-time-to-live group 3600
        negative-time-to-live group 60
        suggested-size group 211
        check-files group yes
        persistent group yes
        shared group yes
        max-db-size group 33554432
        auto-propagate group yes

# hosts caching is broken with gethostby* calls, hence is now disabled
# per default. See /usr/share/doc/nscd/NEWS.Debian.
        enable-cache hosts no
        positive-time-to-live hosts 3600
        negative-time-to-live hosts 20
        suggested-size hosts 211
        check-files hosts yes
        persistent hosts yes
        shared hosts yes
        max-db-size ho...

Hi,

greenmoss <email address hidden> writes:
> I've had nscd installed since before I noticed this problem. So that
> didn't work for me. My nscd configuration is the Ubuntu default:

That is strange. Enabling nscd works around the problem on Debian for
me. Did you restart atd after installing nscd?

Regards,
Ansgar

Here's what I tried:

on 1st Karmic machine:
manually restart atd
atd mail test
== it works

on another Karmic machine:
atd mail test
== it fails
manually restart atd
atd mail test
== it works

In order to be thorough, I rebooted the first machine, did *not* manually restart atd, and re-tried my mail test. It still worked. So my workaround for this problem appears to be manually restarting atd everywhere.

The quirk in this is that all of my Karmic machines had already had nscd installed, and then rebooted several times. So presumably atd had *already* been restarted during those reboots.

Maybe it's also relevant that I first had Jaunty installed on all of these machines, then upgraded to Karmic?

So the following questions still remain:

- shouldn't an atd stop/start due to a machine reboot have fixed this?
- should there be package modifications to atd, nscd, and/or libgcrypt to fix, document, or work around this problem?

Hi,

greenmoss writes:
> - shouldn't an atd stop/start due to a machine reboot have fixed this?

This might not be enough if nscd is started after atd, but I don't know
how they interact in detail. In any case this is a bug that should be
fixed:

> - should there be package modifications to atd, nscd, and/or libgcrypt
> to fix, document, or work around this problem?

Yes, I already filed a bug for libgcrypt in Debian [1]. I think the
library's behavior is not right here.

Regards,
Ansgar

[1] http://bugs.debian.org/566351

Also affects libnss-ldap.
This only concerns programs running with effective uid = 0 and real uid != 0. The system must be configured to use LDAP via SSL.

Back with more information:

After running for "a while" with at jobs being successfully executed, atd will start giving errors as described above. I have not yet managed to discover the maximum amount of time between an atd restart and a successful atd job execution.