pam_hbac: A PAM module to enforce IPA access control rules

Written by Jakub Hrozek and Pavel Reichl

FreeIPA is an open source identity management software. One of the nice features is the ability to limit which users can log in to which servers using Host Based Access Control (HBAC). Previously, only SSSD was able to parse and evaluate the HBAC rules – however, some users run operating systems that either do not support SSSD at all, or SSSD can only be configured in a way that doesn’t include the HBAC engine.

To provide HBAC support for these platforms, we wrote a new PAM module called pam_hbac. The first release of pam_hbac just happened and in this blog post we would like to introduce the project and show how it works.

pam_hbac connects to an IPA server, downloads the HBAC access control rules, evaluates them and decides whether access should be allowed or denied. Even though pam_hbac is a project on its own, it uses the same code to evaluate the HBAC rules as SSSD does, so the resulting access control decision will be the same for the same rules provided the input is the same as well.

Using pam_hbac should come with a disclaimer – if your operating system supports SSSD and you can use its IPA id_provider, please use SSSD instead of pam_hbac. SSSD is maintained by a large team of developers, it is included in distributions with commercial support available and has several advantages over pam_hbac, including offline caching or Kerberos authentication using the host keytab by default.

Who should use pam_hbac

There are legitimate use-cases where using a standalone PAM module like pam_hbac is required, though. These include:

  • if your IPA client runs an OS that doesn’t support SSSD at all like Solaris or its derivatives
  • if your IPA client runs an OS that does support SSSD but not its IPA provider. At the moment, this is the case with Amazon’s Linux distribution and also FreeBSD.
  • if your IPA client runs an OS that supports the IPA provider, but the IPA provider lacks the support for users from trusted Active Directory domains. This is the case for clients running RHEL-5 and leveraging the “compat” LDAP tree provided by the slapi-nis plugin of IPA.

At the moment, pam_hbac runs on Linux, FreeBSD and Solaris (tested on Oracle Solaris 11 and Omnios)

Obtaining and installing pam_hbac

Development of pam_hbac happens on github. There you can clone the git repository and compile pam_hbac yourself. Because the required dependencies or configure flags differ for each platform a bit, please refer to per-platform README files in the doc directory. As an example, there is a FreeBSD-specific README file that contains step-by-step instructions for building and installing pam_hbac on FreeBSD.

For RHEL-5 and RHEL-6, we have also provided a COPR repository with prebuilt binaries. On these platforms, you can just drop the .repo file into /etc/yum.repos.d/ and then yum install pam_hbac.

We would certainly welcome contributors who would like to provide prebuilt binaries for different platforms!

Examples

In the rest of the blog posts, we illustrate two examples of pam_hbac in action. One would be using pam_hbac on FreeBSD to grant access to a subset of IPA users and another one would be to use pam_hbac on CentOS-5 to restrict access to a single Active Directory group in a setup with trusts. For both examples, we use the same environment – the IPA domain is called IPA.TEST and is managed by an IPA server called unidirect.ipa.test. Our clients are called freebsd.ipa.test and centos5.ipa.test.

Example 1: using pam_hbac to provide access control for IPA users on FreeBSD

Even though the recent FreeBSD releases do ship SSSD, it is not built with the IPA provider by default (only through extra flags) and therefore HBAC enforcement might not be available easily. However, we can configure SSSD with the LDAP id_provider or just nss-pam-ldapd on FreeBSD and use pam_hbac for access control separately.

Our goal is to make it possible only for the bsduser to log in to the FreeBSD client machine and nobody else. Start the configuration by making sure you can resolve and authenticate the IPA users. Once that is done, we can configure pam_hbac to provide access control for the FreeBSD machine. Without access control configured, any user should be able to log in:

% su - bsduser
Password:
$ id
uid=1207000884(bsduser) gid=1207000884(bsduser) groups=1207000884(bsduser)
% ^D
% su - tuser
Password:
$ id
uid=1207000883(tuser) gid=1207000883(tuser) groups=1207000883(tuser)

The next step is to install and configure and pam_hbac. Because at the moment there are no prebuilt binary packages for FreeBSD, you’ll need to compile the module from source. The steps are documented in the FreeBSD README in pam_hbac’s git repo . After configuring the source tree, building and installing pam_hbac, you’ll end up with the module installed to /usr/local/lib/pam_hbac.so.

Because much of the information that pam_hbac reads is only accessible to an authenticated user, we need to create a special bind user that pam_hbac will authenticate as. To do so, prepare an LDIF file with the following contents:

dn: uid=hbac,cn=sysaccounts,cn=etc,$DC
objectClass: account
objectClass: simplesecurityobject
objectClass: top
uid: hbac
userPassword: $PASSWORD

Replace the $PASSWORD value with the desired password of the bind user and $DC with the base DN of your IPA server. Then add this LDIF to the IPA server:

ipaserver $ ldapadd -ZZ -H ldap://$IPA_HOSTNAME -D"cn=Directory Manager" -W < hbac_sysuser.ldif

Now we can create the configuration file for pam_hbac. The configuration options are documented in the pam_hbac.conf manpage, but in general it’s enough to point pam_hbac to the IPA server and specify the bind user and its credentials. The config file for pam_hbac on FreeBSD is located at /usr/local/etc/pam_hbac.conf:

[root@freebsd ~]# cat /usr/local/etc/pam_hbac.conf
URI = ldap://unidirect.ipa.test
BASE = dc=ipa,dc=test
BIND_DN = uid=hbac,cn=sysaccounts,cn=etc,dc=ipa,dc=test
BIND_PW = Secret123
SSL_PATH = /usr/local/etc/ipa.crt

Next, we add pam_hbac to the PAM configuration so that it enforces access control during the PAM account phase. Because pam_hbac only handles the account phase, we only add a single line to the account stack of /etc/pam.d/system to make it look like this:

account required pam_login_access.so
account required /usr/local/lib/pam_ldap.so no_warn ignore_authinfo_unavail ignore_unknown_user
account required /usr/local/lib/pam_hbac.so ignore_authinfo_unavail ignore_unknown_user
account required pam_unix.so

Finally, we can disable the allow_all rule on the server and instead only allow access to bsduser to the freebsd.ipa.test machine. Please don’t forget to add other rules in your test environment so that you can at least access your IPA masters!

ipaserver $ ipa hbacrule-add freebsd-bsd-user
ipaserver $ ipa hbacrule-add-host --hosts=freebsd.ipa.test freebsd-bsd-user
ipaserver $ ipa hbacrule-add-user --users=bsduser freebsd-bsd-user
ipaserver $ ipa hbacrule-mod --servicecat=all freebsd-bsd-user
ipaserver $ ipa hbacrule-disable allow_all

Time to test pam_hbac! First, we can make sure bsduser is still able to log in:

% su - bsduser
Password:
$ id
uid=1207000884(bsduser) gid=1207000884(bsduser) groups=1207000884(bsduser)
$ ^D

OK, now for a negative test, see if tuser is denied access:

% su - tuser
Password:
su: Sorry

Great! Looking to /var/log/auth.log reveals that it was indeed the account control module that denied access:

May 25 13:26:37 su: pam_acct_mgmt: permission denied

Example 2: using pam_hbac to provide access control for AD users on CentOS-5

One important use-case for pam_hbac is to provide access control for setups that resolve users from a trusted AD domain using the ‘legacy client’ setup in which a CentOS-5 machine is set up with id_provider=ldap pointing to the IPA server’s compat tree. Please note that if your IPA domain doesn’t have a trust relationship established with an AD domain, you can already use HBAC provided by SSSD and you don’t need pam_hbac at all in that case.

In our setup, let’s have an AD group linux_admins. Our goal will be to grant access to the CentOS-5 machine to members of linux_admins and nobody else. First, make sure your CentOS-5 client is able to resolve and authenticate AD users and the user is a member of the linux_admins group. You can use the output of ipa-advise config-redhat-sssd-before-1-9 as a starting point. Once the CentOS-5 client is set up, you’ll be able to resolve the user, id should report the group and you should be able to authenticate as that user. Because HBAC rules can only be linked to IPA POSIX groups, we also need to add the AD group as a member of an IPA external group which in turn needs to be added to an IPA POSIX group:

ipaserver $ ipa group-add --external linux_admins_ext
ipaserver $ ipa group-add-member --groups=linux_admins_ext ipa_linux_admins
ipaserver $ ipa group-add-member --external=linux_admins@win.trust.test linux_admins_ext

Try logging in:

$ su - linuxop@win.trust.test
Password:
-sh-3.2$ id
uid=300403108(linuxop@win.trust.test) gid=300403108(linuxop@win.trust.test) groups=300400513(domain users@win.trust.test),300403108(linuxop@win.trust.test),300403109(linux_admins@win.trust.test),1207000858(ipa_linux_admins) context=user_u:system_r:unconfined_t

OK, the id output reports the user is a member of the ipa_linux_admins group, so we can proceed with setting up the HBAC rules.

In order to set up the HBAC rules correctly, it’s important to understand how AD users are authenticated when using the compat tree – the SSSD client on the CentOS-5 machine does an LDAP bind using the user’s password against the IPA compat tree. This password bind is intercepted by the slapi-nis plugin running on the IPA server which in turn authenticates against the system-auth service on the IPA server itself. Therefore, it’s important that all users who should be allowed to authenticate against the compat tree are allowed access to the system-auth service on the IPA server. More details about the authentication against the compat tree can be found in the

Let’s add the system-auth rule first, together with its HBAC service that allows access to everyone to the IPA server itself using the system-auth PAM service:

ipaserver $ ipa hbacsvc-add system-auth
ipaserver $ ipa hbacrule-add system-auth-everyone
ipaserver $ ipa hbacrule-add-host --hosts=unidirect.ipa.test system-auth-everyone
ipaserver $ ipa hbacrule-mod --usercat=all system-auth-everyone

The resulting HBAC rule would look like this:

ipaserver $ ipa hbacrule-show system-auth-everyone
Rule name: system-auth-everyone
User category: all
Enabled: TRUE
Hosts: unidirect.ipa.test
Services: system-auth

To avoid allowing all access, disable the allow_all rule

ipaserver$ ipa hbacrule-disable allow_all

As a pre-flight check, you can disable the system-auth-everyone rule, then your user should be denied access and the server-side journal should show something like:

pam_sss(system-auth:account): Access denied for user linuxop@win.trust.test: 6 (Permission denied)

Enabling the rule would allow access again. Of course, don’t forget to allow access to your IPA servers and clients with other services and for other users as appropriate!

As the final step on the server, we can define the HBAC rule for our particular CentOS-5 machine. While the access was already checked against the system-auth rule on the server, that rule cannot discriminate between different hosts and applies to all authentication requests coming from the slapi-nis plugin.

Please note that the a bind user must be configured, refer to the BSD example for details.

The rule will permit all members of ipa_linux_admins group to access all PAM services on the host called centos5.ipa.test:

ipaserver $ ipa hbacrule-add centos5-ipa-linux-admins
ipaserver $ ipa hbacrule-add-host --hosts=centos5.ipa.test centos5-ipa-linux-admins
ipaserver $ ipa hbacrule-mod --servicecat=all centos5-ipa-linux-admins
ipaserver $ ipa hbacrule-add-user --groups=ipa_linux_admins centos5-ipa-linux-admins

Now we’re ready to configure the CentOS-5 client. Make sure pam_hbac is installed first – you can use our COPR repository for that. Just drop the repo file to /etc/yum.repos.d and then yum install pam_hbac.

The configuration file for CentOS-5 machine is quite similar to the one we used on BSD earlier:

URI = ldap://unidirect.ipa.test
BASE = dc=ipa,dc=test
BIND_DN = uid=hbac,cn=sysaccounts,cn=etc,dc=ipa,dc=test
BIND_PW = Secret123

The file is located at /etc/pam_hbac.conf.

Next, add pam_hbac to the /etc/pam.d/system-auth file on the CentOS-5 machine to enable HBAC rules enforcement. This snippet shows the whole account stack on my CentOS-5 machine:

account required pam_unix.so
account sufficient pam_succeed_if.so uid < 500 quiet
account [default=bad success=ok user_unknown=ignore] pam_sss.so
account sufficient pam_localuser.so
account [default=bad success=ok user_unknown=ignore] pam_hbac.so
account required pam_permit.so

You can see I also added the pam_localuser.so module just before the line with pam_hbac.so. Adding the pam_localuser.so module ensures that pam_hbac wouldn’t be called for local users defined in /etc/passwd – we only want the HBAC policies to apply to IPA and AD users.

It’s time to check the rule. As a positive check, we log in with the linuxop user:

$ su - linuxop@win.trust.test
Password:
su: warning: cannot change directory to /home/win.trust.test/linuxop: No such file or directory
-sh-3.2$

As a negative test, we can try logging in as the AD administrator perhaps:

$ su - administrator@win.trust.test
Password:
su: incorrect password

And indeed, /var/log/secure would tell us it was pam_hbac that denied access:

May 25 22:48:35 centos5 su: pam_hbac(su-l:account): returning [6]: Permission denied

Awesome, now even your CentOS-5 server enforces HBAC rules!

Troubleshooting pam_hbac

In case something doesn’t work as expected, there are several ways to debug pam_hbac and the HBAC access control in general. The first step should be to check with the ipa command line tools and the hbactest plugin. For example, this is how we’d test the bsduser‘s access to the freebsd.ipa.test machine:

ipaserver $ ipa hbactest --user=bsduser --service=su --host=freebsd.ipa.test
--------------------
Access granted: True
--------------------
Matched rules: freebsd-bsd-user
Not matched rules: centos5-ipa-linux-admins
Not matched rules: ph_test_trule
Not matched rules: system-auth-everyone

The pam_hbac module by default only logs failures. If you want to see more verbose output, add the debug parameter to the PAM service file configuration. All logging, including debug logs is done using the standard pam_syslog() PAM calls, so the location really depends on your operating system. But as an illustration, this is the tail of the debug output for the AD user case when the allowed user logs in:

May 25 22:48:47 centos5 su: pam_hbac(su-l:account): ALLOWED by rule [centos5-ipa-linux-admins].
May 25 22:48:47 centos5 su: pam_hbac(su-l:account): hbac_evaluate() >]
May 25 22:48:47 centos5 su: pam_hbac(su-l:account): Allowing access
May 25 22:48:47 centos5 su: pam_hbac(su-l:account): returning [0]: Success

The full debug output breaks down all the rules into their components and shows what matched and what did not.

Conclusion

This blog post described the first version of pam_hbac. We will continue the development to add more supported platforms – in the next version, we would like to add support for IBM AIX and Apple OS-X. There are also several bugs and minor enhancements we would like to add. Feel free to file an issue on github if there is something you would like to see improved or something doesn’t work for you!

Advertisements

Creating large AD test environments from the Linux command line

I’ve been recently working on making SSSD perform better in large environments, that contain thousands of users or groups. And because two of the types of setup I wanted to test were SSSD directly enrolled into an Active Directory domain and SSSD as a client of an IPA server that trusts an Active Directory domain, I needed to create a large Active Directory environment for testing.

Of course I didn’t want to create the users in the AD Users and Groups GUI snap-in one by one. And I preferred to create the users and groups from the Linux command line, because that’s what I’m familiar with. After some tinkering, I realized the adcli command already does exactly what I need to with its “create-user”, “create-group” and “add-member” commands.

For example, creating an active directory user is as simple as:

 $ adcli create-user --domain=your.ad.domain adusername

To create a group, you’d call:

 $ adcli create-group --domain=your.ad.domain adgroupname

And finally add a user to the group:

 $ adcli add-member --domain=your.ad.domain adgroupname adusername

By default, adcli would ask you for the AD Administrator, but using a Kerberos ccache is as simple as adding the “-c” option.

Using these three commands I wrote three simple shell scripts that helped me create a thousand users, a thousand groups and add members to these groups in one go. The scripts are available at: https://github.com/jhrozek/adcli_scripts

Of course, adcli does much more than just creating user or group object in Active Directory. adcli is a powerful command-line tool for performing actions on an Active Directory domain. You probably used it already without realizing if you ever added an Active Directory client with realmd, because realmd uses adcli internally. In particular, it would use the “adcli join” command.

On RHEL-6, adcli is available already in EPEL and will be available in RHEL-6.8 as a regular RHEL package. In RHEL-7 and supported Fedora distributions, both realmd and adcli are already available in repositories.

SSSD local overrides

Written by Pavel Březina and Jakub Hrozek

In most cases, using the SSSD is all about connecting a client machine to a central user database, like FreeIPA or Active Directory precisely because you want all users on all machines across the domain to have exactly the same properties. However, even though it would be best to centralize all the things, there will always be exceptions. There can be an odd legacy server where a particular user has a UID set to a value that predates the migration to a central domain controller. Or you might administer a machine where a user stores her files in a different home directory than any other machines.

If the client is a member of an FreeIPA domain, you can just define an “ID view” and define the custom values centrally on the IPA server. Using ID views is the recommended way of setting any overriden attributes – however, not all environments use a FreeIPA server..

To solve those use-cases, the SSSD provides a command-line tool that allows the administrator to set one or more POSIX attributes to a different value on that particular system. The tool is called sss_override and is part of the sssd-tools package since version 1.13.1. Running sss_override --help would present the user with a list of available sub-commands. We’ll show some examples of using them later in this text. Please note that the local overrides can not be used in combination with IPA provider. Instead, the server side overrides can be defined directly through IPA web or command line interface.

One very important aspect to keep in mind is that the overrides are stored in the cache, along with the data from the directory server. This conflicts with a very commonly used practice among admins who often remove the database and try to start from a clean cache when they encounter any problems with sssd. Therefore it is necessary to manually export the override data prior deleting the cache and import them again later, otherwise the override data would be lost!

Creating user and group overrides

Let’s consider our domain has a user “tuser” that resolves with his UID and GID as defined on the server:
$ getent passwd tuser
tuser:*:1190000015:1190000015:test user:/home/tuser:/bin/sh
But on this particular host, we need this user to have a different IDs – maybe the user already owned some files before this host was added as a member to the domain. We can call the sss_override tool and assign him a different ID:
$ sudo sss_override user-add tuser -u 1234
SSSD needs to be restarted for the changes to take effect.
As the info message tells us, currently SSSD needs to be restarted when the first override is added in order for the overriden values to take effect. All subsequent overrides will take effect immediately.
$ sudo systemctl restart sssd
Now, let’s request the user again:
$ getent passwd tuser
tuser:*:1234:1190000015:test user:/home/tuser:/bin/sh
And the changes are visible now! Keep in mind that user-add always replaces the whole local override, so if we wanted to override this user’s name, too, the previous UID change would be lost:
$ sudo sss_override user-add tuser -n xuser
$ getent passwd xuser
xuser:*:1190000015:1190000015:test user:/home/tuser:/bin/sh
As you can see, only the login name was overriden, but the UID was reverted to the original value. This is because the -add command replaces the override completely. The correct way to override both values would be to specify both switches at the same time:
$ sudo sss_override user-add tuser -n xuser -u 1234
It’s possible to also override group attributes – at the moment, overriding group’s name or GID is supported. A very common use is to override a group ID, so let’s take a look:
$ id tuser
uid=1190000015(tuser) gid=1190000015(tuser) groups=1190000015(tuser),1190000016(tgroup)
$ sudo sss_override group-add tgroup -g 5555
$ id tuser
uid=1190000015(tuser) gid=1190000015(tuser) groups=1190000015(tuser),5555(tgroup)

Import and export

As explained earlier, the overrides are stored in the sssd cache itself. To provide some recovery mechanism for admins who sometimes need to purge the cache in order to reset the environment into clean state, we added import and export commands to the sss_override tool. With the example above, we can export the override for tuser with:
$ sudo sss_override user-export /var/log/sssd/user_overrides
This creates a passwd-like file that contains the override data:
$ sudo cat /var/log/sssd/user_overrides
tuser@ipaldap:xuser:1234::::
 In case we decide to remove the cache, we can later add the override data back with:
$ sudo sss_override user-import  /var/log/sssd/user_overrides


Listing

To see what overrides are actually configured on the system, you can use sss_override commands user-find, user-show, group-find and group-show.

The command sss_override user-find allows to find all overrides from all configured sssd domains:
$ sudo sss_override user-find
tuser@ipaldap:xuser:1234::::
admin@anotherdomain:administrator:4321::::
We can also direct the user-find command to a particular domain:
$ sudo sss_override user-find -d ipaldap
tuser@ipaldap:xuser:1234::::
 Or, using the user-show command, even a particular user:
$ sss_override user-show tuser
tuser@ipaldap:xuser:1234::::

Conclusion

The local overrides are a powerful mechanism to deal with irregularities in an otherwise centralized environment. That said, using a completely centralized solution, like IPA’s idviews feature avoids issues with keeping the overrides in the cache itself as well as provides a unified view of all overrides in the UI.

As an incremental improvement, the SSSD team is looking into ways of making the import and export more automated so that the danger of removing the database is mitigated by the SSSD tools.

Get rid of manually calling kinit with SSSD’s help

For quite some time, my laptop doesn’t have a UNIX /etc/passwd user for the account I normally use at all. Instead, I’ve used SSSD exclusively, logging in with our Red Hat corporate account. That allows me to use some cool SSSD features, most notably the krb5_store_password_if_offline option.

When that option is enabled and you log in while SSSD is offline (typically when a laptop was just started and is not connected to corporate VPN yet) using your cached password, SSSD stores the password you use to the kernel keyring. Then, when SSSD figures out that the servers are reachable, it attempts to kinit using that password on your behalf.

Because SSSD can detect when you connected to the VPN by listening to libnl events or watching resolv.conf for changes, the routine of connecting to Kerberos simplifies rapidly to:

  • open laptop
  • login using Kerberos password
  • connect to VPN

That’s it. kinit happens on the background and Kerberos ticket is acquired without any user intervention. Now you can just start Firefox and start accessing HTML pages or start Thunderbird and read your mail. No more starting an extra terminal and typing kinit or starting a separate Gnome dialog.

However, until now, this setup required that you also use an account from a centralized store that contains the Kerberos principal as one of its attributes or at least has the same user name as the first part of the Kerberos principal (called primary, up to the ‘@’ sign). And that’s a deal breaker for many people who are used to a certain local user name for ages, irrespective of their company’s Kerberos principal.

SSSD 1.12.5 that was released recently includes a very nice new feature, that allows you to work around that and map a local UNIX user to a particular Kerberos principal. The setup includes a fairly simple sssd.conf file:

 [sssd]
 domains = example.com
 config_file_version = 2
 services = nss,pam

 [domain/example.com]
 id_provider = proxy
 proxy_lib_name = files

 auth_provider = krb5
 krb5_server = kdc.example.com
 krb5_realm = EXAMPLE.COM
 krb5_map_user = jakub:jhrozek
 krb5_store_password_if_offline = True

 cache_credentials = True

All the interesting configuration is in the [domain] section. The identities are read from good old UNIX files by proxying nss_files through SSSD’s proxy provider. Authentication is performed using the krb5 provider, with a KDC server set to kdc.example.com and Kerberos realm EXAMPLE.COM

The krb5_map_user parameter represents new functionality. The parameter takes a form of username:primary and in this particular case maps a UNIX user jakub to a Kerberos principal which would in this case be jhrozek@EXAMPLE.COM.

On Fedora-22, care must be taken that the pam_sss.so module is called in the PAM stack even if pam_unix.so fails as the default behaviour for pam_unix.so is default=die – end the PAM conversation in case the user logging in exists in /etc/passwd and the password is incorrect. Which is our case, we’re logging in with UNIX user, we just want the authentication to be done by pam_sss.so.

Thanks to Sumit Bose who helped me figure out the PAM config, I used the following auth stack successfully:

auth     required pam_env.so
auth     [default=2 success=ok] pam_localuser.so
auth     sufficient pam_unix.so nullok try_first_pass
auth     [success=done ignore=ignore default=die] pam_sss.so use_first_pass
auth     requisite pam_succeed_if.so uid >= 1000 quiet_success
auth     sufficient pam_sss.so forward_pass
auth     required pam_deny.so

I also recorded a video shows the feature in action. The first login is with the UNIX password, so the ticket is not automatically acquired and the user must run kinit manually. The next login happens with the Kerberos password while the machine is connected to network, so the ticket is acquired on login. Finally, the last login simulates the case where the machine starts disconnected from the network, then connects. In this case, SSSD acquires the ticket on the background when it detects the networking parameters have changed and the KDC is available.

Nice, right? You can also get the video in a nicer resolution.

The Kerberos authentication provider has more features that can beat using kinit from the command line – automatized ticket renewal might be one of them. Please refer to the sssd-krb5 manual page or share your tips in the comments section!

Authenticating a docker container against host’s UNIX accounts

Recently, with the advent of Docker and similar technologies, there’s been an effort to containerize different kinds of setups that previously were running on a single machine or a set of tightly coupled machines. In case the components communicate over a network connection, the containerization might not be that hard, but what about cases where the components would talk over a strictly local interface, such as a UNIX socket?

A colleague of mine asked me to help setup up a container to authenticate against the host’s UNIX accounts using the PAM API the other day. It turned out to be doable, but maybe not obvious to anyone not familiar with some of the more esoteric SSSD options, so I’d like to write down the instructions in a blog post.

In our setup, there was a pam service, sss_test, that previously would run on the host and authenticate against accounts either locally stored on the hosts in /etc/passwd and /etc/shadow. The same setup could in principle be used to authenticate against a remote database using SSSD, just the host SSSD settings would be different.

So how does this work with containers? One possibility, especially with a remote database such as IPA or AD would be to run an SSSD instance in every container and authenticate to the remote store. But that doesn’t help us with cases where we’d like to authenticate against the local database stored on the host, since it’s not exposed outside the host. Moreover, putting SSSD into each container might also mean we’d need to put a keytab in each container, then each container would open its separate connection to the remote server..it just gets tedious. But let’s focus on the local accounts..

The trick we’ll use is bind-mounting. It’s possible to mount part of host’s filesystem into the container’s filesystem and we’ll leverage this to bind-mount the UNIX sockets SSSD communicates over into the container. This will allow the SSSD client side libraries to authenticate against the SSSD running on the host. Then, we’ll set up SSSD on the host to authenticate aganst the hosts’s UNIX files with the proxy back end.

This is the traditional schema:
application -> libpam -> pam_authenticate -> pam_unix.so -> /etc/passwd

If we add SSSD to the mix, it becomes:
application -> libpam -> pam_authenticate -> pam_sss.so -> SSSD -> pam_unix.so -> /etc/passwd

Let’s configure the host and the container, step by step.

  1. Host config
    1. Install packages
    2. yum -y install sssd-common sssd-proxy

    3. create a PAM service for the container.
    4. The name doesn’t matter, it just needs to be referenced from sssd.conf later.
      # cat /etc/pam.d/sss_proxy
      auth required pam_unix.so
      account required pam_unix.so
      password required pam_unix.so
      session required pam_unix.so

    5. create SSSD config file, /etc/sssd/sssd.conf
    6. Please note that the permissions must be 0600 and the file must be owned by root.root.
      # cat /etc/sssd/sssd.conf
      [sssd]
      services = nss, pam
      config_file_version = 2
      domains = proxy
      [nss]
      [pam]
      [domain/proxy]
      id_provider = proxy
      # The proxy provider will look into /etc/passwd for user info
      proxy_lib_name = files
      # The proxy provider will authenticate against /etc/pam.d/sss_proxy
      proxy_pam_target = sss_proxy

    7. start sssd
    8. # systemctl start sssd

    9. verify a user can be retrieved with sssd
    10. $ getent passwd -s sss localuser

  2. Container setup
  3. It’s important to bind-mount the /var/lib/sss/pipes directory from the host to the container since the SSSD UNIX sockets are located there.

    # docker run -i -t -v=/var/lib/sss/pipes/:/var/lib/sss/pipes/:rw --name sssd-cli fedora:latest /bin/sh

  4. Container config
  5. The container runs the PAM-aware application that authenticates against the host. I used a simple program in C that comes from the SSSD source. It pretty much just runs pam_authenticate() against a service called sss_test. Your application might need a different service, but then you just need to adjust the filename in /etc/pam.d/ in the container. All the steps below should be executed on the container itself.

    1. install only the sss client libraries
    2. # yum -y install sssd-client

    3. make sure sss is configured for passwd and group databases in /etc/nsswitch.conf
    4. # grep sss /etc/nsswitch.conf

    5. configure the PAM service that the application uses to call into SSSD

    6. # cat /etc/pam.d/sss_test
      auth required pam_sss.so
      account required pam_sss.so
      password required pam_sss.so
      session required pam_sss.so

    7. authenticate using your PAM application.
    8. In my case that was
      # ./pam_test_client auth localtest
      action: auth
      user: localtest
      testing pam_authenticate
      Password:
      pam_authenticate: Success

  6. Profit!

It would be possible to authenticate against any database, just by changing what the SSSD on the host authenticates against. There’s several gotchas, though, especially should you require that only certain containers are allowed to retrieve users from certain domains. Multitenancy doesn’t really work well, because we don’t have a good mechanism to retrieve the identity of the container.

Anatomy of SSSD user lookup

This blog post describes how a user lookup request is handled in SSSD. It should help you understand how the SSSD architecture looks like, how the data flows in SSSD and as a result help identify which part might not be functioning correctly on your system. It is aimed mostly at users and administrators – for developers, we have a separate document about SSSD internals on the SSSD wiki written by Yassir Elley. This document re-uses some of the info from the internals one.

We’ll look at the most common operation, looking up user info on a remote server. I won’t go into server-specific details, so most of the info should be equally true for LDAP, Active Directory or FreeIPA servers. There’s also more functionality in SSSD than looking up users, such as sudo or autofs integration, but they are out of scope of this post as well.

Before going into SSSD details, let’s do a really quick intro into what happens on the system in general when you request a user from a remote server. Let’s say the admin configured SSSD and tests the configuration by requesting the admin user:

$ getent passwd admin

When user information is requested about a user (with getent, id or similar), typically one of the functions of the Name Service Switch, such as getpwnam() or initgroups() in glibc is called. There’s lots of information about the Name Service Switch in the libc manual, but for our purposes, it’s enough to know that libc opens and reads the config file /etc/nsswitch.conf to find out which modules should be contacted in which order. The module that all of us have on our Linux machines is files which can read user info from /etc/passwd and user info from /etc/groups. There also exists an ldap module that would read the info directly from an LDAP server and of course an sss module that talks to SSSD. So how does that work?

The first thing to keep in mind is that, unlike nss_ldap or pam_ldap, the SSSD is not just a module that is loaded in the context of the application, but rather a deamon that the modules communicate with. Almost no logic is implemented in the modules, all the functionality happens in the deamon. A user-visible effect during debugging is that using strace is not too helpful as it would only show if the request made it to the SSSD. For debugging the rest, the SSSD debug logs should be used.

Earlier I said that SSSD is a deamon. That’s really not too precise, SSSD is actually a set of deamons that communicate with one another. There are three kinds of SSSD processes. One is the sssd process itself. Its purpose is to read the config file after startup and spawn the other processes according to the config file. Then there are responder or front end processes that listen to queries from the applications, like the query that would come from the getent command. If the responder process needs to contact the remote service for data, it talks to the last SSSD process type, which is the data provider or back end process. This architecture allows for a pluggable setup where there are different back end processes talking to different remote servers, while all these remote servers can be accessed from a range of applications or subsystems by the same lookup code in the responders.

Each process is represented by a section in the sssd.conf config file. The main sssd process is represented by the [sssd] section. The front end processes are defined on the services line in the [sssd] section and each can be configured in a section named after the service. And finally, the back end processes are those configured in the [domain] sections. Each process also logs into its own logfile.

Let’s continue with the getent passwd admin example. To illustrate the flow, there is a diagram that the text follows. The full arrows represent local IO operation (like opening a file), the empty arrows represent local IPC over UNIX sockets and the dotted arrow represents a network IO.

sssd-lookup

The user issued the getent command which calls libc’s getpwnam (diagram step 1), then the libc opens the nss_sss module as per nsswitch.conf and passes in the request. First, the nss_sss memory-mapped cache is consulted, that’s step 2 on the diagram. If the data is present in the cache, it is just returned without even contacting the SSSD, which is extremely fast. Otherwise, the request is passed to the SSSD’s responder process (step 3), in particular sssd_nss. The request first looks into the SSSD on-disk cache (step 4). If the data is present in the cache and valid, the nss responder reads the data from the cache and returns them to the application.

If the data is not present in the cache at all or if it’s expired, the sssd_nss request queries the appropriate back end process (step 5) and waits for reply. The back end process connects to the remote server, runs the search (step 6) and stores the resulting data into the cache (step 7). When the search request is finished, the provider process signals back to the responder process that the cache is updated (step 8). At that point, the front-end responder process checks the cache again. If there’s any data in the cache after the back end has updated it, the data is returned to the application – even in cases when the back end failed to update the cache for some reason, it’s better to return stale data than none. Of course, if no data is found in the cache after the back end has finished, an empty result is returned back. This final cache check is represented by step 9 in the diagram.

When I said the back end “runs a search” against the server, I really simplified the matter a lot. The search can involve many different steps, such as resolving the server to connect to, authenticating to the server, performing the search itself and storing the resulting data into the database. Some of the steps might even require a helper process, for instance authenticating against a remote server using a keytab is done in a heper process called ldap_child that logs into its own logfile called /var/log/sssd/ldap_child.log.

Given most steps happen in the back end itself, then most often, the problem or misconfiguration lies in the back end part. But it is still very important to know the overall architecture and be able to identify if and how the request made it to the back end at all. In the next part, we’ll apply this new information to perform a small case study and we will repair a buggy sssd setup.

Troubleshooting a failing SSSD user lookup.

With the SSSD architecture in mind, we can try a case study. Consider we have an IPA client, but no users, not even the admin show up:

$ getent passwd admin
$ echo $?
2

The admin user was not found! Given our knowledge of the architecture, let’s first see if the system is configured to query sssd for user information at all:

$ grep passwd /etc/nsswitch.conf
passwd: files sss

It is. Then the request was passed on to the nss responder process, since the only other possibility is a successful return from the memory cache. We need to raise the debug_level in the [nss] section like this:

[nss]
debug_level = 7

and restart sssd:

# systemctl restart sssd

Then we’ll request the admin user again and inspect the NSS logs:

[sssd[nss]] [accept_fd_handler] (0x0400): Client connected!
[sssd[nss]] [sss_cmd_get_version] (0x0200): Received client version [1].
[sssd[nss]] [sss_cmd_get_version] (0x0200): Offered version [1].
[sssd[nss]] [nss_cmd_getbynam] (0x0400): Running command [17] with input [admin].
[sssd[nss]] [sss_parse_name_for_domains] (0x0200): name 'admin' matched without domain, user is admin
[sssd[nss]] [nss_cmd_getbynam] (0x0100): Requesting info for [admin] from []
[sssd[nss]] [nss_cmd_getpwnam_search] (0x0100): Requesting info for [admin@ipa.example.com]
[sssd[nss]] [sss_dp_issue_request] (0x0400): Issuing request for [0x4266f9:1:admin@ipa.example.com]
[sssd[nss]] [sss_dp_get_account_msg] (0x0400): Creating request for [ipa.example.com][4097][1][name=admin]
[sssd[nss]] [sss_dp_internal_get_send] (0x0400): Entering request [0x4266f9:1:admin@ipa.example.com]
[sssd[nss]] [sss_dp_get_reply] (0x1000): Got reply from Data Provider - DP error code: 1 errno: 11 error message: Fast reply - offline
[sssd[nss]] [nss_cmd_getby_dp_callback] (0x0040): Unable to get information from Data Provider
Error: 1, 11, Fast reply - offline
Will try to return what we have in cache

Well, apparently the request for the admin user was received and passed on to the back end process, but the back end replied that it switched to offline mode..that means we need to also enable debugging in the domain part and continue investigation there. We need to add debug_level to the [domain] section and restart sssd again. Then run the getent command and inspect the file called /var/log/sssd/sssd_ipa.example.com starting with the time that corresponds to the NSS responder sending the data (as indicated by sss_dp_issue_request in the nss log). In the domain log we see:

[sssd[be[ipa.example.com]]] [fo_resolve_service_done] (0x0020): Failed to resolve server 'master.ipa.example.com:389': Domain name not found
[sssd[be[ipa.example.com]]] [set_server_common_status] (0x0100): Marking server 'master.ipa.example.com:389' as 'not working'
[sssd[be[ipa.example.com]]] [be_resolve_server_process] (0x0080): Couldn't resolve server (master.ipa.example.com:389), resolver returned (11)
[sssd[be[ipa.example.com]]] [be_resolve_server_process] (0x1000): Trying with the next one!
[sssd[be[ipa.example.com]]] [fo_resolve_service_send] (0x0100): Trying to resolve service 'IPA'
[sssd[be[ipa.example.com]]] [get_server_status] (0x1000): Status of server 'master.ipa.example.com:389' is 'not working'
[sssd[be[ipa.example.com]]] [get_port_status] (0x1000): Port status of port 0 for server '(no name)' is 'not working'
[sssd[be[ipa.example.com]]] [get_server_status] (0x1000): Status of server 'master.ipa.example.com:389' is 'not working'
[sssd[be[ipa.example.com]]] [fo_resolve_service_send] (0x0020): No available servers for service 'IPA'
[sssd[be[ipa.example.com]]] [be_resolve_server_done] (0x1000): Server resolution failed: 5
[sssd[be[ipa.example.com]]] [sdap_id_op_connect_done] (0x0020): Failed to connect, going offline (5 [Input/output error])
[sssd[be[ipa.example.com]]] [be_ptask_create] (0x0400): Periodic task [Check if online (periodic)] was created
[sssd[be[ipa.example.com]]] [be_ptask_schedule] (0x0400): Task [Check if online (periodic)]: scheduling task 70 seconds from now [1426087775]
[sssd[be[ipa.example.com]]] [be_run_offline_cb] (0x0080): Going offline. Running callbacks.

OK, that gets us somewhere. Indeed, our /etc/resolv.conf file was ponting to a bad nameserver. And indeed, after fixing the resolver settings and restarting SSSD, everything seems to be working:

$ getent passwd admin
admin:*:1546600000:1546600000:Administrator:/home/admin:/bin/bash

Awesome, we were able to repair a broken SSSD setup!

How does SSSD interact with tools like kinit?

Many SSSD users know that SSSD supports fail over from one server to another for authentication with services like su or ssh and even autodiscovers the Kerberos servers using DNS records.

But occasionally users would ask – OK, so SSSD lets me log in with another server but I also need to use kinit manually. Does kinit use the same server SSSD used? If so, how does kinit know which KDC SSSD uses?

The SSSD actually has a plugin that is able to tell what KDC or kadmin server to use for a particular realm. When SSSD discovers a Kerberos server, it puts the IP address of that server into a file stored under the /var/lib/sss/pubconf directory. The file that stores the KDC is called kdcinfo.$REALM and the kpasswd file is called kpasswd.$REALM. When SSSD switches to another Kerberos server during a fail over operation, a new IP address is generated in these files. Also, if SSSD goes offline completely, these files are removed, so that tools using libkrb5 only rely on other means of configuration, such as the krb5.conf file.

As noted above, the kdcinfo files are only refreshed during SSSD operation, like user login. This poses a disadvantage for systems that don’t perform many operations using the PAM stack, because the server that SSSD discovered might go offline without SSSD triggering a fail over operation. For these environments, it’s better to disable the kdcinfo files altogether by setting the krb5_use_kdcinfo option to False and relying on krb5.conf completely. We plan on improving the kdcinfo plugin in future upstream versions so that it plays better with these kind of setups.

The SSSD kdcinfo plugin even has a man page!

Fake DNS replies in unit tests using resolv_wrapper

If your unit tests require custom DNS queries, there are some options you might want to take, like adding records to the local /etc/hosts file. But that might not be possible for tests where you don't have root access (for instance, in build systems) and moreover you can't set any other records except A or AAAA. You can also run a full DNS server and set it into your resolv.conf file, but that normally requires root privileges, too and tampers with the usual setup of the test host. What would be ideal is a way to force the test into a mock DNS environment without affecting the live environment on the host system.

As Andreas Schneider pointed out earlier, it is time for another wrapper – so together with Andreas, we wrote resolv_wrapper! This post will show you how can resolv_wrapper help your testing.

Similar to the other wrappers, the resolv_wrapper provides a preloadable version of library calls. In this case it's res_init, res_query, res_search and res_close. These libresolv (or libc, depending on platform) library calls form the basis of DNS resolution routines like gethostbyname and can also be used to resolve less common DNS queries, such as SRV or SOA. In general, a unit test leveraging resolv_wrapper needs to set up its environment (more on that later), preload the libresolv_wrapper.so library using LD_PRELOAD and that's it.

If your test environment has its own DNS server (such as Samba or FreeIPA have), resolv_wrapper allows you to redirect DNS traffic to that server by pointing the test to a resolv.conf file that contains IP address of your DNS server:

echo "search test.example.com" > /tmp/testresolv.conf
echo "nameserver 127.0.0.1" >> /tmp/testresolv.conf
LD_PRELOAD=libresolv_wrapper.so RESOLV_WRAPPER_CONF=/tmp/testresolv.conf ./dns_unit_test

That would make your dns_unit_test perform all DNS queries through your DNS server running at 127.0.0.1, while your system would be still intact and using the original resolv.conf entries. In some other cases, you might want to test DNS resolution, but maybe you don't want to set up a full DNS server just for the test For this use-case, resolv_wrapper provides the ability to fake DNS replies using a hosts-like text file. Consider a unit test, where you want to make sure that kinit can discover a Kerberos KDC with SRV records. Start by defining the the hosts-like file:

echo "SRV _kerberos._tcp.example.com kdc.example.com 88" > /tmp/fakehosts
echo "A   kdc.example.com 127.0.0.10" >> /tmp/fakehosts

Then export this hosts file using the RESOLV_WRAPPER_HOSTS environment variable and preload the resolv_wrapper as illustrated before:

LD_PRELOAD=libresolv_wrapper.so RESOLV_WRAPPER_HOSTS=/tmp/fakehosts ./kinit_unit_test

If something is going wrong, resolv_wrapper allows the user to enable debugging when the RESOLV_WRAPPER_DEBUGLEVEL is set to a numerical value. The highest allowed value, that enabled low-level tracing is 4.

Let's show a complete example with a simple C program that tries to resolve an A record of kdc.example.com. We'll start with this C source file:

#include &ltstdlib.h&gt
#include &ltunistd.h&gt
#include &ltstring.h&gt
#include &ltstdio.h&gt

#include &ltnetinet/in.h&gt
#include &ltarpa/nameser.h&gt
#include &ltarpa/inet.h&gt
#include &ltresolv.h&gt

int main(void)
{
        int rv;
        struct __res_state dnsstate;
        unsigned char answer[256];
        char addr[64] = { 0 } ;
        ns_msg handle;
        ns_rr rr;

        memset(&dnsstate, 0, sizeof(struct __res_state));
        res_ninit(&dnsstate);
        res_nquery(&dnsstate, "kdc.example.com", ns_c_in, ns_t_a, answer, sizeof(answer));

        ns_initparse(answer, sizeof(answer), &handle);
        ns_parserr(&handle, ns_s_an, 0, &rr), 0;
        inet_ntop(AF_INET, ns_rr_rdata(rr), addr, sizeof(addr));
        puts(addr);

        return 0;
}

Please note I omitted all error checking to keep the code short.

Compile the file and link it with libresolv:

gcc rwrap_example.c -lresolv -o rwrap_example

And now you can just run the example binary along with resolv_wrapper, using the RESOLV_WRAPPER_DEBUGLEVEL to see the progress:

LD_PRELOAD=libresolv_wrapper.so RESOLV_WRAPPER_HOSTS=/tmp/fakehosts RESOLV_WRAPPER_DEBUGLEVEL=4 ./rwrap_example
RWRAP_TRACE(1970) - _rwrap_load_lib_function: Loaded __res_ninit from libc
RWRAP_TRACE(1970) - rwrap_res_nquery: Resolve the domain name [kdc.example.com] - class=1, type=1
RWRAP_TRACE(1970) - rwrap_res_nquery:         nameserver: 10.11.5.19
RWRAP_TRACE(1970) - rwrap_res_nquery:         nameserver: 10.5.30.160
RWRAP_TRACE(1970) - rwrap_res_nquery:         nameserver: 192.168.1.254
RWRAP_TRACE(1970) - rwrap_res_fake_hosts: Searching in fake hosts file /tmp/fakehosts
RWRAP_TRACE(1970) - rwrap_res_fake_hosts: Successfully faked answer for [kdc.example.com]
RWRAP_TRACE(1970) - rwrap_res_nquery: The returned response length is: 0
127.0.0.10

And that's pretty much it!

The resolv_wrapper lives at the cwrap.org site along with the other wrappers and has its own dedicated page. You can grab the source code from git.samba.org. The git tree includes a RST-formatted documentation file, with even more details and example. We're also working on making resolv_wrapper usable on other platforms than Linux, although there are still some bugs here and there.

Add sudo rules to Active Directory and access them with SSSD

Centralizing sudo rules in a centralized identity store such as FreeIPA is usually a good choice for your environment as opposed to copying the sudoers files around – the administrator has one place to edit the sudo rules and the rule set is always up to date. Replication mitigates most of the single-point-of-failure woes and by using modern clients like the SSSD, the rules can also be cached on the client side, making the client resilient against network outages.

What if your identity store is Active Directory though? In this post, I'll show you how to load sudo rules to an AD server and how to configure SSSD to retrieve and cache the rules. A prerequisite is a running AD instance and a Linux client enrolled to the AD instance using tools like realmd or adcli. In this post, I'll use dc=DOMAINNAME,dc=LOCAL as the Windows domain name.

The first step is to load the sudo schema into the AD server. The schema describes the objects sudo uses and their attributes and is not part of standard AD installations. In Fedora, the file describing the schema is part of the SUDO RPM and is located at /usr/share/doc/sudo/schema.ActiveDirectory. You can copy the file to your AD server or download it from the Internet directly.

Next, lauch the Windows command line and load the schema to AD's LDAP server using the ldifde utility:

ldifde -i -f schema.ActiveDirectory -c dc=X dc=DOMAINNAME,dc=LOCAL

Before creating the rule, let's also crate an LDAP container that would store the rules. It's not a good idea to mix sudo rules into the same OU that already stores other objects, like users – a separate OU makes management easier and allows to set more fine-grained permissions. You can create the sudoers OU in "ADSI Edit" quite easily by right-clicking the top-level container (dc=DOMAINNAME,dc=LOCAL), selecting "New->Object". In the dialog that opens, select "organizationalUnit", click "Next" and finally name the new OU "sudoers". If you select a different name or a different OU altogether, you'll have to set a custom ldap_sudo_search_base in sssd.conf, the default is ou=sudoers,$BASE_DN".

Now, let's add the rule itself. For illustration purposes, we'll allow the user called 'jdoe' to execute less on all Linux clients in the enterprise.

In my test, I used "ADSI Edit" again. Just right-click the SUDO container, select "New->Object" and then you should see sudoRole in the list of objectClasses. Create the rule based on the syntax described in the sudoers.ldap man page; as an example, I created a rule that allows the user called "jdoe" to run less, for instance to be able to inspect system log files.

dn: CN=lessrule,OU=sudoers,DC=DOMAINNAME,DC=LOCAL
objectClass: top
objectClass: sudoRole
cn: lessrule
distinguishedName: CN=lessrule,OU=sudoers,DC=DOMAINNAME,DC=LOCAL
name: lessrule
sudoHost: ALL
sudoCommand: /usr/bin/less
sudoUser: jdoe

The username of the user who is allowed to execute the rule is stored in the sudoUser attribute. Please note that the username must be stored non-qualified, which is different from the usual username@DOMAIN (or DOM\username) syntax used in Windows.For a more detailed description of how the sudo rules in LDAP work, refer to the sudoers.ldap manual page.

The client configuration involves minor modifications to two configuration files. First, edit /etc/nsswitch.conf and append 'sss' to the 'sudoers:' database configuration:

sudoers: files sss

If the sudoers database was not present in nsswitch.conf at all, just add the line as above. This modification would allow SSSD to communicate with the sssd with the libsss_sudo library.
Finally, open the /etc/sssd/sssd.conf file and edit the [sssd] section to include the sudo service:

services = nss, pam, sudo

Then just restart sssd and the setup is done! For testing, log in as the user in question ("jdoe" here) and run:

sudo -l

You should be able to see something like this in the output:
User jdoe may run the following commands on adclient:
(lcl) /usr/bin/less

That's it! Now you can use your AD server as an centralized sudo rules storage and the rules are cached and available offline with the SSSD.

COPR repos with newer SSSD versions for RHEL-5 and RHEL-6

Two interesting COPR repos with SSSD packages were made available recently.

One was prepared by Stephen Gallagher and contains SSSD 1.9.x for RHEL-5. I created the other one with SSSD 1.11 built for RHEL-6. I'd love to see test reports for the RHEL-6 repo, as we are considering upgrading to 1.11 in RHEL-6.6.

For more details on the repos see the announcements on sssd-devel about both Stephen's and mine repos.