autofs: regression on focal->jammy upgrade: SASL binds to Samba AD broken

Upstream has been informed about this security weakness: https://marc.info/?l=autofs&m=166004593318481&w=2

Above patches have been tested with Samba AD DC (Version 4.15.9-Ubuntu) and OpenLDAP slapd (2.5.12+dfsg-0ubuntu0.22.04.1).

As already experienced with real showstopper bug #1982219 we should expect a solution before next automounter version. Therefore, it could make sense to provide own Debian patches until a new upstream version will be provided in a subsequent Ubuntu release.

The attachment "Upstream proposed patches (1/4)" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

As per LP: #1982219, I am marking this bug as triaged and adding it to the server team backlog.

We may want to wait for the upstream review process to complete before applying this to kinetic. In the mean time, would you happen to have a short reproducer for the bug?

Thank you.

As requested here's more detailed information about this bug:

$ automount -f -d -v
Starting automounter version 5.1.8, master map auto.master
using kernel protocol version 5.05
lookup_nss_read_master: reading master ldap auto.master
parse_server_string: lookup(ldap): Attempting to parse LDAP information from string "auto.master".
parse_server_string: lookup(ldap): mapname auto.master
parse_ldap_config: lookup(ldap): ldap authentication configured with the following options:
parse_ldap_config: lookup(ldap): use_tls: 0, tls_required: 0, auth_required: 2, sasl_mech: GSSAPI
parse_ldap_config: lookup(ldap): user: (null), secret: unspecified, client principal: <email address hidden> credential cache: /tmp/krb5cc_0
do_init: parse(sun): init gathered global options: (null)
find_server: trying server uri ldap://dc.example.com
do_bind: lookup(ldap): auth_required: 2, sasl_mech GSSAPI
sasl_do_kinit_ext_cc: using external credential cache for auth: client principal <email address hidden>
sasl_do_kinit_ext_cc: external credential cache default principal <email address hidden>
sasl_do_kinit_ext_cc: Kerberos authentication was successful!
sasl_bind_mech: Attempting sasl bind with mechanism GSSAPI
sasl_log_func: GSSAPI client step 1
getuser_func: called with context (nil), id 16385.
sasl_log_func: GSSAPI client step 1
getuser_func: called with context (nil), id 16385.
sasl_log_func: GSSAPI client step 2
sasl_bind_mech: sasl bind with mechanism GSSAPI succeeded
do_bind: lookup(ldap): autofs_sasl_bind returned 0
get_query_dn: lookup(ldap): check search base list
get_query_dn: lookup(ldap): query failed for search dn ou=automount,dc=example,dc=com: Operations error
get_query_dn: lookup(ldap): failed to find query dn under search base dns
do_connect: lookup(ldap): failed to get query dn
lookup(ldap): couldn't connect to server ldap://dc.example.com
do_reconnect: lookup(ldap): failed to find available server
lookup_nss_read_master: no map - continuing to next source
no mounts in table

Challenge here is the limited logging by autofs. Switching TLS off and monitoring the traffic with wireshark the root cause can be retrieved:

Lightweight Directory Access Protocol
    LDAPMessage bindResponse(3) strongAuthRequired (SASL:[GSSAPI]: Sign or Seal are required.)
        messageID: 3
        protocolOp: bindResponse (1)
            bindResponse
                resultCode: strongAuthRequired (8)
                matchedDN:
                errorMessage: SASL:[GSSAPI]: Sign or Seal are required.
                serverSaslCreds: <MISSING>
        [Response To: 26]
        [Time: 0.000164267 seconds]

Samba AD DC rejects SASL bind as it relies on LDAP traffic encryption using SASL GSSAPI data security layer.

/etc/autofs_ldap_auth.conf:

<?xml version="1.0" ?>
<!--
This files contains a single entry with multiple attributes tied to it.
See autofs_ldap_auth.conf(5) for more information.
-->
<autofs_ldap_sasl_conf
        usetls="no"
        tlsrequired="no"
        authrequired="yes"
        authtype="GSSAPI"
        credentialcache="/tmp/krb5cc_0"
        clientprinc="DC$@EXAMPLE.COM"
/>

And here's the output after applying proposed patches using the same configuration as above:

$ automount -f -v -d

Starting automounter version 5.1.8, master map auto.master
using kernel protocol version 5.05
reading ldap master auto.master
parse_server_string: lookup(ldap): Attempting to parse LDAP information from string "auto.master".
parse_server_string: lookup(ldap): mapname auto.master
parse_ldap_config: lookup(ldap): ldap authentication configured with the following options:
parse_ldap_config: lookup(ldap): use_tls: 0, tls_required: 0, auth_required: 2, sasl_mech: GSSAPI
parse_ldap_config: lookup(ldap): user: (null), secret: unspecified, client principal: <email address hidden> credential cache: /tmp/krb5cc_0
do_init: parse(sun): init gathered global options: (null)
find_server: trying server uri ldap://dc.example.com
init_ldap_connection: lookup(ldap): ldap_initialize( ldap://dc.example.com )
do_bind: lookup(ldap): auth_required: 2, sasl_mech GSSAPI
sasl_do_kinit: initializing kerberos ticket: client principal <email address hidden>
sasl_do_kinit: calling krb5_parse_name on client principal <email address hidden>
sasl_do_kinit: Using tgs name <email address hidden>
sasl_do_kinit: Kerberos authentication was successful!
do_bind: Attempting sasl bind with mechanism GSSAPI
do_bind: SASL username: <email address hidden>
do_bind: SASL authcid: root
do_bind: SASL SSF: 256
do_bind: sasl bind with mechanism GSSAPI succeeded
get_query_dn: lookup(ldap): check search base list
get_query_dn: lookup(ldap): found search base under ou=automount,dc=example,dc=com
get_query_dn: lookup(ldap): found query dn CN=auto.master,OU=automount,DC=example,DC=com
connected to uri ldap://dc.example.com
lookup_read_master: lookup(ldap): searching for "(objectclass=nisObject)" under "CN=auto.master,OU=automount,DC=example,DC=com"
lookup_read_master: lookup(ldap): examining entries
validate_string_len: lookup(ldap): string /pub encoded as /pub
validate_string_len: lookup(ldap): string /- encoded as /-
validate_string_len: lookup(ldap): string /home/EXAMPLE encoded as /home/EXAMPLE
[...]

And the wireshark captures:

Samba AD DC -> autofs client
Lightweight Directory Access Protocol
    LDAPMessage bindResponse(4) success
        messageID: 4
        proto...

And finally here's the log when trying to bind to Samba AD DC with SASL GSSAPI and TLS using the patched automounter with following configuration:

/etc/autofs_ldap_auth.conf:

<?xml version="1.0" ?>
<!--
This files contains a single entry with multiple attributes tied to it.
See autofs_ldap_auth.conf(5) for more information.
-->
<autofs_ldap_sasl_conf
        usetls="yes"
        tlsrequired="no"
        authrequired="yes"
        authtype="GSSAPI"
        credentialcache="/tmp/krb5cc_0"
        clientprinc="DC$@EXAMPLE.COM"
/>

$ automount -f -v -d

Starting automounter version 5.1.8, master map auto.master
using kernel protocol version 5.05
reading ldap master auto.master
parse_server_string: lookup(ldap): Attempting to parse LDAP information from string "auto.master".
parse_server_string: lookup(ldap): mapname auto.master
parse_ldap_config: lookup(ldap): ldap authentication configured with the following options:
parse_ldap_config: lookup(ldap): use_tls: 1, tls_required: 0, auth_required: 2, sasl_mech: GSSAPI
parse_ldap_config: lookup(ldap): user: (null), secret: unspecified, client principal: <email address hidden> credential cache: /tmp/krb5cc_0
do_init: parse(sun): init gathered global options: (null)
find_server: trying server uri ldap://dc.example.com
init_ldap_connection: lookup(ldap): ldap_initialize( ldap://dc.example.com )
do_bind: lookup(ldap): auth_required: 2, sasl_mech GSSAPI
sasl_do_kinit: initializing kerberos ticket: client principal <email address hidden>
sasl_do_kinit: calling krb5_parse_name on client principal <email address hidden>
sasl_do_kinit: Using tgs name <email address hidden>
sasl_do_kinit: Kerberos authentication was successful!
do_bind: Attempting sasl bind with mechanism GSSAPI
do_bind: lookup(ldap): ldap_sasl_interactive_bind failed with error 53
do_bind: ldap_sasl_interactive_bind: SASL:[GSSAPI]: Sign or Seal are not allowed if TLS is used
lookup(ldap): couldn't connect to server ldap://dc.example.com

Samba AD DC rejects the SASL bind as it does not allow SASL (encryption) over TLS due to security constraints (CVE-2016-2112). See https://wiki.samba.org/index.php/Configuring_LDAP_over_SSL_(LDAPS)_on_a_Samba_AD_DC#SASL_over_TLS:_A_bad_idea.
The improved logging now outlines the reason for SASL bind rejection. Wireshark captures are not needed anymore.

Upstream has reviewed the patches and put them into the update queue. As for #1982219 release will be delayed and most probably will make the way into a new version 5.1.9.
I've prepared a PPA with the latest version from upstream:
PPA: https://launchpad.net/~rdratlos/+archive/ubuntu/lp-1984073-autofs-broken-sasl-binds-to-samba-ad

Please feel free to test and provide feedback.

Thank you very much for your work on autofs, rdratlos.

I have subscribed myself to the bug and will try to reproduce the issue (and test your PPA) later. Since this is affecting Jammy, it will require an SRU. One of the first steps here is to write the template for the SRU bug: it involves writing an Impact section, detailing how to reproduce the issue in a Test Plan section, and mentioning possible regressions that can be introduced by the fix in a "Where problems could occur" section.

If you already have a step-by-step procedure to reproduce the bug, that would be awesome. Otherwise, we can start working on it.

Thanks.

Dear Sergio,
reproducing the issue requires a Samba AD DC and a domain workstation with autofs-ldap installed. AutoFS maps need not to be defined in Samba AD as this issues relates to SASL authentication to Samba AD only. But Samba AD needs to be prepared to allow Kerberos authentication of the domain workstation using service principals.

Are there any Test Plans available for setting up a test Samba AD DC and join a test workstation to the domain? This would be a good basis for adding the autofs required configuration changes.

The main area of possible regression is for connecting the autofs test workstation to an OpenLDAP directory service instead of a Samba AD. Using the proposed patches SASL authentication is now completely handled by the OpenLDAP client. Before authentication was controlled by an own SASL client implementation within autofs. Risk for regression is very low as all OpenLDAP client tools (e. g. ldapsearch) use the same mechanism.

No regression is expected for interworking with older Ubuntu releases. We have still Ubuntu 18.04 Samba AD DC in the net and interworking has been successfully tested to them.

Hi rdratlos,

Thanks for your feedback.

A few months ago I worked on bug #1934997 and had to devise a Test Plan to reproduce it. I think it covers most of what you said above: setting up a Samba AD DC and joining the realm from another client (using realmd and sssd). I believe you can use that as a base for writing a Test Plan for this bug. Feel free to adjust it as needed, of course.

I also intend to perform tests over here in order to make sure that things will still work OK after the proposed change.

Thanks.

[ Impact ]

to be provided based on the communication above

[ Test Plan ]

The test case for this bug requires a quite complex test environment. The objective is to reuse the test setup for bug #1934997 as much as possible.

We can reuse one VM from bug #1934997 running a Samba Active Directory Domain Controller (AD DC), which will be our main server that holds the autofs maps. This needs only to be configured once. We then need to setup two privileged containers, which will join our AD domain using realmd and SSSD or Winbind. One container will act as autofs client, which wants to mount a file share from the second container that acts as NAS.
The use of privileged containers is not strictly mandatory for this test plan. LDAP SASL authentication binds and autofs map retrieval work also using unprivileged containers. But in order to demonstrate that autofs mounts really work correctly NFS is required, which only works in privileged containers.

For the autofs client we will use Winbind instead of SSSD to serve domain user logins. The reason for this is that SSSD provides its own autofs client and we want to avoid a mix and match of both implementations.

== Virtual Network setup ==

The first step is to create a dedicated virtual network for our tests. This is not strictly mandatory, but it will simplify things. The best way to create this network is via virt-manager. Install it if needed, open the program and select the "QEMU/KVM" line. Go to Edit > Connection Details > Virtual Networks, click on the "+" icon (bottom left), give this network a name, make sure that "Enable IPv4" and "Enable DHCPv4" are selected (under "IPv4 configuration"). Go to "DNS domain name" and select "Custom". For the domain name, type "test.lan". Click on "Finish".

Take note of the "Device" name that shows up after you create the network. We will use it when creating the containers. For this test plan, let's assume the device name is "virbr1".

== Samba AD DC VM setup ==

We need to setup a Samba AD DC server. It doesn't matter which Ubuntu release we use for it. Note that we have to use "-n virbr1" when creating the VM, otherwise it won't use our virtual network.

$ lxc launch ubuntu-daily:jammy sambadc -n virbr1 --vm
$ lxc shell sambadc
server# apt update
server# ip a

Make sure to grab this VM's IP address.

server# cat >> /etc/hosts << _EOF_
IP_ADDRESS_HERE sambadc sambadc.test.lan
_EOF_
server# reboot
$ lxc shell sambadc
server# apt install -y samba winbind

server# mv /etc/samba/smb.conf /etc/samba/smb.conf.bkp
server# samba-tool domain provision --use-rfc2307 --realm TEST.LAN --domain TEST --server-role dc --dns-backend SAMBA_INTERNAL --adminpass MyPassword1

server# systemctl mask smbd.service nmbd.service winbind.service
server# systemctl disable --now smbd.service nmbd.service winbind.service
server# systemctl unmask samba-ad-dc.service
server# systemctl enable --now samba-ad-dc.service

We now have to adjust the DNS server settings of the server. We are going to disable systemd-resolved.service and use samba as our DNS service.

You will notice that the samba-tool command issued above has added 127.0.0.53 as the "dns forwarder" in /etc/samba/smb.conf. Edi...

Dear Sergio,

above you will find the proposal for a test plan of this bug (fix). I tried to align it to your test plan style. Please feel free to adapt it and enter missing information.

Risks are outlined but should be minor. Automated testing is difficult to achieve.

Hope this helps you.

Kind regards

Thomas

Patches are not yet published at https://mirrors.edge.kernel.org/pub/linux/daemons/autofs/v5/patches-5.1.9/

Upstream says they will be, eventually[1] (message related to another thread, but valid in this context I think).

I'm subscribed to the mailing list, and will be notified of further activity there, or in this bug.

For now, I'll postpone working on this bug until the patches are at least published.

1. https://www.spinics.net/lists/autofs/msg02549.html

Still no upstream formal ACK on this patch. We will consider it again for the next development cycle.

This patch (and others) has been published to https://mirrors.edge.kernel.org/pub/linux/daemons/autofs/v5/patches-5.1.9/, this bug can move forward.

I might have found a way to simulate this with just an openldap server, without having to provision a full samba AD DC. I can set "olcSaslSecProps: minssf=256" in the openldap server, and that will block GSSAPI with an SSF=0. It's a different error code, though, but pristine autofs is failing nonetheless. Let's see if it starts to work after your patches, in this scenario.

I may have found a bug in the patchset, I emailed the mailing list about it (link to the ML archive pending). Looks like we lost support for the credentialcache config parameter when using gssapi auth.

Note to self: just applying these patches made NTLM sasl auth fail[1]:

2023-05-17T14:26:41.509016+00:00 autopkgtest systemd[1]: Started autofs.service - Automounts filesystems on demand.
2023-05-17T14:26:41.518732+00:00 autopkgtest slapd[3229]: conn=1019 fd=12 ACCEPT from IP=127.0.0.1:48900 (IP=0.0.0.0:389)
2023-05-17T14:26:41.519035+00:00 autopkgtest slapd[3229]: conn=1019 op=0 BIND dn="" method=163
2023-05-17T14:26:41.519176+00:00 autopkgtest slapd[3229]: conn=1019 op=0 RESULT tag=97 err=14 qtime=0.000008 etime=0.000119 text=SASL(0): successful result:
2023-05-17T14:26:41.519449+00:00 autopkgtest automount[3488]: do_bind: lookup(ldap): ldap_sasl_interactive_bind failed with error -2
2023-05-17T14:26:41.519695+00:00 autopkgtest slapd[3229]: connection_operation: error: SASL bind in progress (tag=66).
2023-05-17T14:26:41.519748+00:00 autopkgtest slapd[3229]: conn=1019 fd=12 closed
2023-05-17T14:26:41.520035+00:00 autopkgtest slapd[3229]: conn=1020 fd=12 ACCEPT from IP=127.0.0.1:48914 (IP=0.0.0.0:389)
2023-05-17T14:26:41.520206+00:00 autopkgtest slapd[3229]: conn=1020 op=0 BIND dn="" method=163
2023-05-17T14:26:41.520327+00:00 autopkgtest slapd[3229]: conn=1020 op=0 RESULT tag=97 err=14 qtime=0.000006 etime=0.000076 text=SASL(0): successful result:
2023-05-17T14:26:41.520588+00:00 autopkgtest automount[3488]: do_bind: lookup(ldap): ldap_sasl_interactive_bind failed with error -2
2023-05-17T14:26:41.520787+00:00 autopkgtest slapd[3229]: connection_operation: error: SASL bind in progress (tag=66).
2023-05-17T14:26:41.520830+00:00 autopkgtest slapd[3229]: conn=1020 fd=12 closed
2023-05-17T14:26:41.520920+00:00 autopkgtest automount[3488]: key "storage" not found in map source(s).

There are other follow-up patches that came up after this initial set. Investigate which are still needed. I think I saw something about interactive bind.

1. https://autopkgtest.ubuntu.com/results/autopkgtest-mantic-ahasenack-autofs-sasl-fixes/mantic/amd64/a/autofs/20230517_142653_91010@/log.gz

This bug was fixed in the package autofs - 5.1.8-1ubuntu6

---------------
autofs (5.1.8-1ubuntu6) mantic; urgency=medium

  * d/t/ldap-map-sasl-auth: wait for slapd to be ready (LP: #2023232)

 -- Andreas Hasenack <email address hidden> Thu, 08 Jun 2023 14:02:00 -0300

While working on this bug for the ubuntu devel release (mantic at this time), I found at least one regression[1] (that I fixed and sent upstream[2]), and several other cases around the config file options that changed behavior.

The general pattern has been a somewhat incorrect config in some cases, but that let autofs run and even succeed in mounting the filesystem, provided the ldap server was configured in a particular way.

In the new code, autofs will complain about the (somewhat) invalid configuration, or correctly detect failed authentication attempts, and not even attempt the ldap search, which could have worked (and was working in the previous version).

In the old sasl code, autofs would apparently not detect an authentication failure[3], and proceed anyway. If the LDAP server allowed the (now anonymous, after the auth failure) search, it would happily return the requested attributes and autofs would mount the filesystem. In the new version, the authentication failure is detected, and the mount will not happen. This would regress deployments that were either using an unsupported SASL mechanism, like NTLM[4], or which just had an incorrect credential.

I also found possible regressions around the handling of the "authrequired autodetect" configuration option. Again, with good intentions, but still a change in behavior. It's even hard to understand what actually will happen given the manpage documentation, and patch description.

For example, with "authrequired=autodetect", and "user", "secret", "authtype" not set, there is change in behavior. The "old" autofs starts and tries to use gss-spnego (or gssapi). I didn't have that setup, so I don't know if it would have succeeded or not, but I saw the attempts in the logs.

The patched autofs complains about this configuration: it wants "user" and "secret". It starts, but does not try to use gssapi/gss-spnego at all. So what if someone had that config working and using gssapi, even by accident, and now upgrades? I conceed that having a gssapi setup "working by accident" is a bit rare, though ;)

There is a lack of discussion upstream about these, as can be seen in the mailing list links I provided, which does not give me confidence that these patches are final. Specially given that no new upstream version has been released with them yet, tells me that perhaps these patches have not yet seen widespread use.

There can be other changes in behavior lurking behind these changes. These are the ones I detected while writing autopkgtests for autofs.

Given these reasons:
- there don't seem to be many people affected by this bug (yet?)
- at least when using samba, there is a configuration workaround in samba (ldap server require strong auth)
- according to this bug's activity, not many people are affected
- there is risk of regressing existing deployments, even if one could argue that their configuration is incorrect
- there has been no new upstream autofs release with these changes included yet, but we have confirmation that the patches were accepted and committed. We can deduce that these changes have not yet seen widespread use.
- my simple DEP8 tests caught at least one regression introduced b...

I agree with Andreas' proposal not to put higher regression risks on stable releases.

The proposed patch in https://www.spinics.net/lists/autofs/msg02582.html looks fine and should help deployments using external credential caches.

For all those who might still run into an interworking issue with autofs and Samba AD approved upstream fixes are available for jammy in a PPA: https://launchpad.net/~rdratlos/+archive/ubuntu/lp-1984073-autofs-broken-sasl-binds-to-samba-ad