pam_hbac: A PAM module to enforce IPA access control rules

Written by Jakub Hrozek and Pavel Reichl

FreeIPA is an open source identity management software. One of the nice features is the ability to limit which users can log in to which servers using Host Based Access Control (HBAC). Previously, only SSSD was able to parse and evaluate the HBAC rules – however, some users run operating systems that either do not support SSSD at all, or SSSD can only be configured in a way that doesn’t include the HBAC engine.

To provide HBAC support for these platforms, we wrote a new PAM module called pam_hbac. The first release of pam_hbac just happened and in this blog post we would like to introduce the project and show how it works.

pam_hbac connects to an IPA server, downloads the HBAC access control rules, evaluates them and decides whether access should be allowed or denied. Even though pam_hbac is a project on its own, it uses the same code to evaluate the HBAC rules as SSSD does, so the resulting access control decision will be the same for the same rules provided the input is the same as well.

Using pam_hbac should come with a disclaimer – if your operating system supports SSSD and you can use its IPA id_provider, please use SSSD instead of pam_hbac. SSSD is maintained by a large team of developers, it is included in distributions with commercial support available and has several advantages over pam_hbac, including offline caching or Kerberos authentication using the host keytab by default.

Who should use pam_hbac

There are legitimate use-cases where using a standalone PAM module like pam_hbac is required, though. These include:

  • if your IPA client runs an OS that doesn’t support SSSD at all like Solaris or its derivatives
  • if your IPA client runs an OS that does support SSSD but not its IPA provider. At the moment, this is the case with Amazon’s Linux distribution and also FreeBSD.
  • if your IPA client runs an OS that supports the IPA provider, but the IPA provider lacks the support for users from trusted Active Directory domains. This is the case for clients running RHEL-5 and leveraging the “compat” LDAP tree provided by the slapi-nis plugin of IPA.

At the moment, pam_hbac runs on Linux, FreeBSD and Solaris (tested on Oracle Solaris 11 and Omnios)

Obtaining and installing pam_hbac

Development of pam_hbac happens on github. There you can clone the git repository and compile pam_hbac yourself. Because the required dependencies or configure flags differ for each platform a bit, please refer to per-platform README files in the doc directory. As an example, there is a FreeBSD-specific README file that contains step-by-step instructions for building and installing pam_hbac on FreeBSD.

For RHEL-5 and RHEL-6, we have also provided a COPR repository with prebuilt binaries. On these platforms, you can just drop the .repo file into /etc/yum.repos.d/ and then yum install pam_hbac.

We would certainly welcome contributors who would like to provide prebuilt binaries for different platforms!

Examples

In the rest of the blog posts, we illustrate two examples of pam_hbac in action. One would be using pam_hbac on FreeBSD to grant access to a subset of IPA users and another one would be to use pam_hbac on CentOS-5 to restrict access to a single Active Directory group in a setup with trusts. For both examples, we use the same environment – the IPA domain is called IPA.TEST and is managed by an IPA server called unidirect.ipa.test. Our clients are called freebsd.ipa.test and centos5.ipa.test.

Example 1: using pam_hbac to provide access control for IPA users on FreeBSD

Even though the recent FreeBSD releases do ship SSSD, it is not built with the IPA provider by default (only through extra flags) and therefore HBAC enforcement might not be available easily. However, we can configure SSSD with the LDAP id_provider or just nss-pam-ldapd on FreeBSD and use pam_hbac for access control separately.

Our goal is to make it possible only for the bsduser to log in to the FreeBSD client machine and nobody else. Start the configuration by making sure you can resolve and authenticate the IPA users. Once that is done, we can configure pam_hbac to provide access control for the FreeBSD machine. Without access control configured, any user should be able to log in:

% su - bsduser
Password:
$ id
uid=1207000884(bsduser) gid=1207000884(bsduser) groups=1207000884(bsduser)
% ^D
% su - tuser
Password:
$ id
uid=1207000883(tuser) gid=1207000883(tuser) groups=1207000883(tuser)

The next step is to install and configure and pam_hbac. Because at the moment there are no prebuilt binary packages for FreeBSD, you’ll need to compile the module from source. The steps are documented in the FreeBSD README in pam_hbac’s git repo . After configuring the source tree, building and installing pam_hbac, you’ll end up with the module installed to /usr/local/lib/pam_hbac.so.

Because much of the information that pam_hbac reads is only accessible to an authenticated user, we need to create a special bind user that pam_hbac will authenticate as. To do so, prepare an LDIF file with the following contents:

dn: uid=hbac,cn=sysaccounts,cn=etc,$DC
objectClass: account
objectClass: simplesecurityobject
objectClass: top
uid: hbac
userPassword: $PASSWORD

Replace the $PASSWORD value with the desired password of the bind user and $DC with the base DN of your IPA server. Then add this LDIF to the IPA server:

ipaserver $ ldapadd -ZZ -H ldap://$IPA_HOSTNAME -D"cn=Directory Manager" -W < hbac_sysuser.ldif

Now we can create the configuration file for pam_hbac. The configuration options are documented in the pam_hbac.conf manpage, but in general it’s enough to point pam_hbac to the IPA server and specify the bind user and its credentials. The config file for pam_hbac on FreeBSD is located at /usr/local/etc/pam_hbac.conf:

[root@freebsd ~]# cat /usr/local/etc/pam_hbac.conf
URI = ldap://unidirect.ipa.test
BASE = dc=ipa,dc=test
BIND_DN = uid=hbac,cn=sysaccounts,cn=etc,dc=ipa,dc=test
BIND_PW = Secret123
SSL_PATH = /usr/local/etc/ipa.crt

Next, we add pam_hbac to the PAM configuration so that it enforces access control during the PAM account phase. Because pam_hbac only handles the account phase, we only add a single line to the account stack of /etc/pam.d/system to make it look like this:

account required pam_login_access.so
account required /usr/local/lib/pam_ldap.so no_warn ignore_authinfo_unavail ignore_unknown_user
account required /usr/local/lib/pam_hbac.so ignore_authinfo_unavail ignore_unknown_user
account required pam_unix.so

Finally, we can disable the allow_all rule on the server and instead only allow access to bsduser to the freebsd.ipa.test machine. Please don’t forget to add other rules in your test environment so that you can at least access your IPA masters!

ipaserver $ ipa hbacrule-add freebsd-bsd-user
ipaserver $ ipa hbacrule-add-host --hosts=freebsd.ipa.test freebsd-bsd-user
ipaserver $ ipa hbacrule-add-user --users=bsduser freebsd-bsd-user
ipaserver $ ipa hbacrule-mod --servicecat=all freebsd-bsd-user
ipaserver $ ipa hbacrule-disable allow_all

Time to test pam_hbac! First, we can make sure bsduser is still able to log in:

% su - bsduser
Password:
$ id
uid=1207000884(bsduser) gid=1207000884(bsduser) groups=1207000884(bsduser)
$ ^D

OK, now for a negative test, see if tuser is denied access:

% su - tuser
Password:
su: Sorry

Great! Looking to /var/log/auth.log reveals that it was indeed the account control module that denied access:

May 25 13:26:37 su: pam_acct_mgmt: permission denied

Example 2: using pam_hbac to provide access control for AD users on CentOS-5

One important use-case for pam_hbac is to provide access control for setups that resolve users from a trusted AD domain using the ‘legacy client’ setup in which a CentOS-5 machine is set up with id_provider=ldap pointing to the IPA server’s compat tree. Please note that if your IPA domain doesn’t have a trust relationship established with an AD domain, you can already use HBAC provided by SSSD and you don’t need pam_hbac at all in that case.

In our setup, let’s have an AD group linux_admins. Our goal will be to grant access to the CentOS-5 machine to members of linux_admins and nobody else. First, make sure your CentOS-5 client is able to resolve and authenticate AD users and the user is a member of the linux_admins group. You can use the output of ipa-advise config-redhat-sssd-before-1-9 as a starting point. Once the CentOS-5 client is set up, you’ll be able to resolve the user, id should report the group and you should be able to authenticate as that user. Because HBAC rules can only be linked to IPA POSIX groups, we also need to add the AD group as a member of an IPA external group which in turn needs to be added to an IPA POSIX group:

ipaserver $ ipa group-add --external linux_admins_ext
ipaserver $ ipa group-add-member --groups=linux_admins_ext ipa_linux_admins
ipaserver $ ipa group-add-member --external=linux_admins@win.trust.test linux_admins_ext

Try logging in:

$ su - linuxop@win.trust.test
Password:
-sh-3.2$ id
uid=300403108(linuxop@win.trust.test) gid=300403108(linuxop@win.trust.test) groups=300400513(domain users@win.trust.test),300403108(linuxop@win.trust.test),300403109(linux_admins@win.trust.test),1207000858(ipa_linux_admins) context=user_u:system_r:unconfined_t

OK, the id output reports the user is a member of the ipa_linux_admins group, so we can proceed with setting up the HBAC rules.

In order to set up the HBAC rules correctly, it’s important to understand how AD users are authenticated when using the compat tree – the SSSD client on the CentOS-5 machine does an LDAP bind using the user’s password against the IPA compat tree. This password bind is intercepted by the slapi-nis plugin running on the IPA server which in turn authenticates against the system-auth service on the IPA server itself. Therefore, it’s important that all users who should be allowed to authenticate against the compat tree are allowed access to the system-auth service on the IPA server. More details about the authentication against the compat tree can be found in the

Let’s add the system-auth rule first, together with its HBAC service that allows access to everyone to the IPA server itself using the system-auth PAM service:

ipaserver $ ipa hbacsvc-add system-auth
ipaserver $ ipa hbacrule-add system-auth-everyone
ipaserver $ ipa hbacrule-add-host --hosts=unidirect.ipa.test system-auth-everyone
ipaserver $ ipa hbacrule-mod --usercat=all system-auth-everyone

The resulting HBAC rule would look like this:

ipaserver $ ipa hbacrule-show system-auth-everyone
Rule name: system-auth-everyone
User category: all
Enabled: TRUE
Hosts: unidirect.ipa.test
Services: system-auth

To avoid allowing all access, disable the allow_all rule

ipaserver$ ipa hbacrule-disable allow_all

As a pre-flight check, you can disable the system-auth-everyone rule, then your user should be denied access and the server-side journal should show something like:

pam_sss(system-auth:account): Access denied for user linuxop@win.trust.test: 6 (Permission denied)

Enabling the rule would allow access again. Of course, don’t forget to allow access to your IPA servers and clients with other services and for other users as appropriate!

As the final step on the server, we can define the HBAC rule for our particular CentOS-5 machine. While the access was already checked against the system-auth rule on the server, that rule cannot discriminate between different hosts and applies to all authentication requests coming from the slapi-nis plugin.

Please note that the a bind user must be configured, refer to the BSD example for details.

The rule will permit all members of ipa_linux_admins group to access all PAM services on the host called centos5.ipa.test:

ipaserver $ ipa hbacrule-add centos5-ipa-linux-admins
ipaserver $ ipa hbacrule-add-host --hosts=centos5.ipa.test centos5-ipa-linux-admins
ipaserver $ ipa hbacrule-mod --servicecat=all centos5-ipa-linux-admins
ipaserver $ ipa hbacrule-add-user --groups=ipa_linux_admins centos5-ipa-linux-admins

Now we’re ready to configure the CentOS-5 client. Make sure pam_hbac is installed first – you can use our COPR repository for that. Just drop the repo file to /etc/yum.repos.d and then yum install pam_hbac.

The configuration file for CentOS-5 machine is quite similar to the one we used on BSD earlier:

URI = ldap://unidirect.ipa.test
BASE = dc=ipa,dc=test
BIND_DN = uid=hbac,cn=sysaccounts,cn=etc,dc=ipa,dc=test
BIND_PW = Secret123

The file is located at /etc/pam_hbac.conf.

Next, add pam_hbac to the /etc/pam.d/system-auth file on the CentOS-5 machine to enable HBAC rules enforcement. This snippet shows the whole account stack on my CentOS-5 machine:

account required pam_unix.so
account sufficient pam_succeed_if.so uid < 500 quiet
account [default=bad success=ok user_unknown=ignore] pam_sss.so
account sufficient pam_localuser.so
account [default=bad success=ok user_unknown=ignore] pam_hbac.so
account required pam_permit.so

You can see I also added the pam_localuser.so module just before the line with pam_hbac.so. Adding the pam_localuser.so module ensures that pam_hbac wouldn’t be called for local users defined in /etc/passwd – we only want the HBAC policies to apply to IPA and AD users.

It’s time to check the rule. As a positive check, we log in with the linuxop user:

$ su - linuxop@win.trust.test
Password:
su: warning: cannot change directory to /home/win.trust.test/linuxop: No such file or directory
-sh-3.2$

As a negative test, we can try logging in as the AD administrator perhaps:

$ su - administrator@win.trust.test
Password:
su: incorrect password

And indeed, /var/log/secure would tell us it was pam_hbac that denied access:

May 25 22:48:35 centos5 su: pam_hbac(su-l:account): returning [6]: Permission denied

Awesome, now even your CentOS-5 server enforces HBAC rules!

Troubleshooting pam_hbac

In case something doesn’t work as expected, there are several ways to debug pam_hbac and the HBAC access control in general. The first step should be to check with the ipa command line tools and the hbactest plugin. For example, this is how we’d test the bsduser‘s access to the freebsd.ipa.test machine:

ipaserver $ ipa hbactest --user=bsduser --service=su --host=freebsd.ipa.test
--------------------
Access granted: True
--------------------
Matched rules: freebsd-bsd-user
Not matched rules: centos5-ipa-linux-admins
Not matched rules: ph_test_trule
Not matched rules: system-auth-everyone

The pam_hbac module by default only logs failures. If you want to see more verbose output, add the debug parameter to the PAM service file configuration. All logging, including debug logs is done using the standard pam_syslog() PAM calls, so the location really depends on your operating system. But as an illustration, this is the tail of the debug output for the AD user case when the allowed user logs in:

May 25 22:48:47 centos5 su: pam_hbac(su-l:account): ALLOWED by rule [centos5-ipa-linux-admins].
May 25 22:48:47 centos5 su: pam_hbac(su-l:account): hbac_evaluate() >]
May 25 22:48:47 centos5 su: pam_hbac(su-l:account): Allowing access
May 25 22:48:47 centos5 su: pam_hbac(su-l:account): returning [0]: Success

The full debug output breaks down all the rules into their components and shows what matched and what did not.

Conclusion

This blog post described the first version of pam_hbac. We will continue the development to add more supported platforms – in the next version, we would like to add support for IBM AIX and Apple OS-X. There are also several bugs and minor enhancements we would like to add. Feel free to file an issue on github if there is something you would like to see improved or something doesn’t work for you!

Advertisements

Anatomy of SSSD user lookup

This blog post describes how a user lookup request is handled in SSSD. It should help you understand how the SSSD architecture looks like, how the data flows in SSSD and as a result help identify which part might not be functioning correctly on your system. It is aimed mostly at users and administrators – for developers, we have a separate document about SSSD internals on the SSSD wiki written by Yassir Elley. This document re-uses some of the info from the internals one.

We’ll look at the most common operation, looking up user info on a remote server. I won’t go into server-specific details, so most of the info should be equally true for LDAP, Active Directory or FreeIPA servers. There’s also more functionality in SSSD than looking up users, such as sudo or autofs integration, but they are out of scope of this post as well.

Before going into SSSD details, let’s do a really quick intro into what happens on the system in general when you request a user from a remote server. Let’s say the admin configured SSSD and tests the configuration by requesting the admin user:

$ getent passwd admin

When user information is requested about a user (with getent, id or similar), typically one of the functions of the Name Service Switch, such as getpwnam() or initgroups() in glibc is called. There’s lots of information about the Name Service Switch in the libc manual, but for our purposes, it’s enough to know that libc opens and reads the config file /etc/nsswitch.conf to find out which modules should be contacted in which order. The module that all of us have on our Linux machines is files which can read user info from /etc/passwd and user info from /etc/groups. There also exists an ldap module that would read the info directly from an LDAP server and of course an sss module that talks to SSSD. So how does that work?

The first thing to keep in mind is that, unlike nss_ldap or pam_ldap, the SSSD is not just a module that is loaded in the context of the application, but rather a deamon that the modules communicate with. Almost no logic is implemented in the modules, all the functionality happens in the deamon. A user-visible effect during debugging is that using strace is not too helpful as it would only show if the request made it to the SSSD. For debugging the rest, the SSSD debug logs should be used.

Earlier I said that SSSD is a deamon. That’s really not too precise, SSSD is actually a set of deamons that communicate with one another. There are three kinds of SSSD processes. One is the sssd process itself. Its purpose is to read the config file after startup and spawn the other processes according to the config file. Then there are responder or front end processes that listen to queries from the applications, like the query that would come from the getent command. If the responder process needs to contact the remote service for data, it talks to the last SSSD process type, which is the data provider or back end process. This architecture allows for a pluggable setup where there are different back end processes talking to different remote servers, while all these remote servers can be accessed from a range of applications or subsystems by the same lookup code in the responders.

Each process is represented by a section in the sssd.conf config file. The main sssd process is represented by the [sssd] section. The front end processes are defined on the services line in the [sssd] section and each can be configured in a section named after the service. And finally, the back end processes are those configured in the [domain] sections. Each process also logs into its own logfile.

Let’s continue with the getent passwd admin example. To illustrate the flow, there is a diagram that the text follows. The full arrows represent local IO operation (like opening a file), the empty arrows represent local IPC over UNIX sockets and the dotted arrow represents a network IO.

sssd-lookup

The user issued the getent command which calls libc’s getpwnam (diagram step 1), then the libc opens the nss_sss module as per nsswitch.conf and passes in the request. First, the nss_sss memory-mapped cache is consulted, that’s step 2 on the diagram. If the data is present in the cache, it is just returned without even contacting the SSSD, which is extremely fast. Otherwise, the request is passed to the SSSD’s responder process (step 3), in particular sssd_nss. The request first looks into the SSSD on-disk cache (step 4). If the data is present in the cache and valid, the nss responder reads the data from the cache and returns them to the application.

If the data is not present in the cache at all or if it’s expired, the sssd_nss request queries the appropriate back end process (step 5) and waits for reply. The back end process connects to the remote server, runs the search (step 6) and stores the resulting data into the cache (step 7). When the search request is finished, the provider process signals back to the responder process that the cache is updated (step 8). At that point, the front-end responder process checks the cache again. If there’s any data in the cache after the back end has updated it, the data is returned to the application – even in cases when the back end failed to update the cache for some reason, it’s better to return stale data than none. Of course, if no data is found in the cache after the back end has finished, an empty result is returned back. This final cache check is represented by step 9 in the diagram.

When I said the back end “runs a search” against the server, I really simplified the matter a lot. The search can involve many different steps, such as resolving the server to connect to, authenticating to the server, performing the search itself and storing the resulting data into the database. Some of the steps might even require a helper process, for instance authenticating against a remote server using a keytab is done in a heper process called ldap_child that logs into its own logfile called /var/log/sssd/ldap_child.log.

Given most steps happen in the back end itself, then most often, the problem or misconfiguration lies in the back end part. But it is still very important to know the overall architecture and be able to identify if and how the request made it to the back end at all. In the next part, we’ll apply this new information to perform a small case study and we will repair a buggy sssd setup.

Troubleshooting a failing SSSD user lookup.

With the SSSD architecture in mind, we can try a case study. Consider we have an IPA client, but no users, not even the admin show up:

$ getent passwd admin
$ echo $?
2

The admin user was not found! Given our knowledge of the architecture, let’s first see if the system is configured to query sssd for user information at all:

$ grep passwd /etc/nsswitch.conf
passwd: files sss

It is. Then the request was passed on to the nss responder process, since the only other possibility is a successful return from the memory cache. We need to raise the debug_level in the [nss] section like this:

[nss]
debug_level = 7

and restart sssd:

# systemctl restart sssd

Then we’ll request the admin user again and inspect the NSS logs:

[sssd[nss]] [accept_fd_handler] (0x0400): Client connected!
[sssd[nss]] [sss_cmd_get_version] (0x0200): Received client version [1].
[sssd[nss]] [sss_cmd_get_version] (0x0200): Offered version [1].
[sssd[nss]] [nss_cmd_getbynam] (0x0400): Running command [17] with input [admin].
[sssd[nss]] [sss_parse_name_for_domains] (0x0200): name 'admin' matched without domain, user is admin
[sssd[nss]] [nss_cmd_getbynam] (0x0100): Requesting info for [admin] from []
[sssd[nss]] [nss_cmd_getpwnam_search] (0x0100): Requesting info for [admin@ipa.example.com]
[sssd[nss]] [sss_dp_issue_request] (0x0400): Issuing request for [0x4266f9:1:admin@ipa.example.com]
[sssd[nss]] [sss_dp_get_account_msg] (0x0400): Creating request for [ipa.example.com][4097][1][name=admin]
[sssd[nss]] [sss_dp_internal_get_send] (0x0400): Entering request [0x4266f9:1:admin@ipa.example.com]
[sssd[nss]] [sss_dp_get_reply] (0x1000): Got reply from Data Provider - DP error code: 1 errno: 11 error message: Fast reply - offline
[sssd[nss]] [nss_cmd_getby_dp_callback] (0x0040): Unable to get information from Data Provider
Error: 1, 11, Fast reply - offline
Will try to return what we have in cache

Well, apparently the request for the admin user was received and passed on to the back end process, but the back end replied that it switched to offline mode..that means we need to also enable debugging in the domain part and continue investigation there. We need to add debug_level to the [domain] section and restart sssd again. Then run the getent command and inspect the file called /var/log/sssd/sssd_ipa.example.com starting with the time that corresponds to the NSS responder sending the data (as indicated by sss_dp_issue_request in the nss log). In the domain log we see:

[sssd[be[ipa.example.com]]] [fo_resolve_service_done] (0x0020): Failed to resolve server 'master.ipa.example.com:389': Domain name not found
[sssd[be[ipa.example.com]]] [set_server_common_status] (0x0100): Marking server 'master.ipa.example.com:389' as 'not working'
[sssd[be[ipa.example.com]]] [be_resolve_server_process] (0x0080): Couldn't resolve server (master.ipa.example.com:389), resolver returned (11)
[sssd[be[ipa.example.com]]] [be_resolve_server_process] (0x1000): Trying with the next one!
[sssd[be[ipa.example.com]]] [fo_resolve_service_send] (0x0100): Trying to resolve service 'IPA'
[sssd[be[ipa.example.com]]] [get_server_status] (0x1000): Status of server 'master.ipa.example.com:389' is 'not working'
[sssd[be[ipa.example.com]]] [get_port_status] (0x1000): Port status of port 0 for server '(no name)' is 'not working'
[sssd[be[ipa.example.com]]] [get_server_status] (0x1000): Status of server 'master.ipa.example.com:389' is 'not working'
[sssd[be[ipa.example.com]]] [fo_resolve_service_send] (0x0020): No available servers for service 'IPA'
[sssd[be[ipa.example.com]]] [be_resolve_server_done] (0x1000): Server resolution failed: 5
[sssd[be[ipa.example.com]]] [sdap_id_op_connect_done] (0x0020): Failed to connect, going offline (5 [Input/output error])
[sssd[be[ipa.example.com]]] [be_ptask_create] (0x0400): Periodic task [Check if online (periodic)] was created
[sssd[be[ipa.example.com]]] [be_ptask_schedule] (0x0400): Task [Check if online (periodic)]: scheduling task 70 seconds from now [1426087775]
[sssd[be[ipa.example.com]]] [be_run_offline_cb] (0x0080): Going offline. Running callbacks.

OK, that gets us somewhere. Indeed, our /etc/resolv.conf file was ponting to a bad nameserver. And indeed, after fixing the resolver settings and restarting SSSD, everything seems to be working:

$ getent passwd admin
admin:*:1546600000:1546600000:Administrator:/home/admin:/bin/bash

Awesome, we were able to repair a broken SSSD setup!

How does SSSD interact with tools like kinit?

Many SSSD users know that SSSD supports fail over from one server to another for authentication with services like su or ssh and even autodiscovers the Kerberos servers using DNS records.

But occasionally users would ask – OK, so SSSD lets me log in with another server but I also need to use kinit manually. Does kinit use the same server SSSD used? If so, how does kinit know which KDC SSSD uses?

The SSSD actually has a plugin that is able to tell what KDC or kadmin server to use for a particular realm. When SSSD discovers a Kerberos server, it puts the IP address of that server into a file stored under the /var/lib/sss/pubconf directory. The file that stores the KDC is called kdcinfo.$REALM and the kpasswd file is called kpasswd.$REALM. When SSSD switches to another Kerberos server during a fail over operation, a new IP address is generated in these files. Also, if SSSD goes offline completely, these files are removed, so that tools using libkrb5 only rely on other means of configuration, such as the krb5.conf file.

As noted above, the kdcinfo files are only refreshed during SSSD operation, like user login. This poses a disadvantage for systems that don’t perform many operations using the PAM stack, because the server that SSSD discovered might go offline without SSSD triggering a fail over operation. For these environments, it’s better to disable the kdcinfo files altogether by setting the krb5_use_kdcinfo option to False and relying on krb5.conf completely. We plan on improving the kdcinfo plugin in future upstream versions so that it plays better with these kind of setups.

The SSSD kdcinfo plugin even has a man page!

Fake DNS replies in unit tests using resolv_wrapper

If your unit tests require custom DNS queries, there are some options you might want to take, like adding records to the local /etc/hosts file. But that might not be possible for tests where you don't have root access (for instance, in build systems) and moreover you can't set any other records except A or AAAA. You can also run a full DNS server and set it into your resolv.conf file, but that normally requires root privileges, too and tampers with the usual setup of the test host. What would be ideal is a way to force the test into a mock DNS environment without affecting the live environment on the host system.

As Andreas Schneider pointed out earlier, it is time for another wrapper – so together with Andreas, we wrote resolv_wrapper! This post will show you how can resolv_wrapper help your testing.

Similar to the other wrappers, the resolv_wrapper provides a preloadable version of library calls. In this case it's res_init, res_query, res_search and res_close. These libresolv (or libc, depending on platform) library calls form the basis of DNS resolution routines like gethostbyname and can also be used to resolve less common DNS queries, such as SRV or SOA. In general, a unit test leveraging resolv_wrapper needs to set up its environment (more on that later), preload the libresolv_wrapper.so library using LD_PRELOAD and that's it.

If your test environment has its own DNS server (such as Samba or FreeIPA have), resolv_wrapper allows you to redirect DNS traffic to that server by pointing the test to a resolv.conf file that contains IP address of your DNS server:

echo "search test.example.com" > /tmp/testresolv.conf
echo "nameserver 127.0.0.1" >> /tmp/testresolv.conf
LD_PRELOAD=libresolv_wrapper.so RESOLV_WRAPPER_CONF=/tmp/testresolv.conf ./dns_unit_test

That would make your dns_unit_test perform all DNS queries through your DNS server running at 127.0.0.1, while your system would be still intact and using the original resolv.conf entries. In some other cases, you might want to test DNS resolution, but maybe you don't want to set up a full DNS server just for the test For this use-case, resolv_wrapper provides the ability to fake DNS replies using a hosts-like text file. Consider a unit test, where you want to make sure that kinit can discover a Kerberos KDC with SRV records. Start by defining the the hosts-like file:

echo "SRV _kerberos._tcp.example.com kdc.example.com 88" > /tmp/fakehosts
echo "A   kdc.example.com 127.0.0.10" >> /tmp/fakehosts

Then export this hosts file using the RESOLV_WRAPPER_HOSTS environment variable and preload the resolv_wrapper as illustrated before:

LD_PRELOAD=libresolv_wrapper.so RESOLV_WRAPPER_HOSTS=/tmp/fakehosts ./kinit_unit_test

If something is going wrong, resolv_wrapper allows the user to enable debugging when the RESOLV_WRAPPER_DEBUGLEVEL is set to a numerical value. The highest allowed value, that enabled low-level tracing is 4.

Let's show a complete example with a simple C program that tries to resolve an A record of kdc.example.com. We'll start with this C source file:

#include &ltstdlib.h&gt
#include &ltunistd.h&gt
#include &ltstring.h&gt
#include &ltstdio.h&gt

#include &ltnetinet/in.h&gt
#include &ltarpa/nameser.h&gt
#include &ltarpa/inet.h&gt
#include &ltresolv.h&gt

int main(void)
{
        int rv;
        struct __res_state dnsstate;
        unsigned char answer[256];
        char addr[64] = { 0 } ;
        ns_msg handle;
        ns_rr rr;

        memset(&dnsstate, 0, sizeof(struct __res_state));
        res_ninit(&dnsstate);
        res_nquery(&dnsstate, "kdc.example.com", ns_c_in, ns_t_a, answer, sizeof(answer));

        ns_initparse(answer, sizeof(answer), &handle);
        ns_parserr(&handle, ns_s_an, 0, &rr), 0;
        inet_ntop(AF_INET, ns_rr_rdata(rr), addr, sizeof(addr));
        puts(addr);

        return 0;
}

Please note I omitted all error checking to keep the code short.

Compile the file and link it with libresolv:

gcc rwrap_example.c -lresolv -o rwrap_example

And now you can just run the example binary along with resolv_wrapper, using the RESOLV_WRAPPER_DEBUGLEVEL to see the progress:

LD_PRELOAD=libresolv_wrapper.so RESOLV_WRAPPER_HOSTS=/tmp/fakehosts RESOLV_WRAPPER_DEBUGLEVEL=4 ./rwrap_example
RWRAP_TRACE(1970) - _rwrap_load_lib_function: Loaded __res_ninit from libc
RWRAP_TRACE(1970) - rwrap_res_nquery: Resolve the domain name [kdc.example.com] - class=1, type=1
RWRAP_TRACE(1970) - rwrap_res_nquery:         nameserver: 10.11.5.19
RWRAP_TRACE(1970) - rwrap_res_nquery:         nameserver: 10.5.30.160
RWRAP_TRACE(1970) - rwrap_res_nquery:         nameserver: 192.168.1.254
RWRAP_TRACE(1970) - rwrap_res_fake_hosts: Searching in fake hosts file /tmp/fakehosts
RWRAP_TRACE(1970) - rwrap_res_fake_hosts: Successfully faked answer for [kdc.example.com]
RWRAP_TRACE(1970) - rwrap_res_nquery: The returned response length is: 0
127.0.0.10

And that's pretty much it!

The resolv_wrapper lives at the cwrap.org site along with the other wrappers and has its own dedicated page. You can grab the source code from git.samba.org. The git tree includes a RST-formatted documentation file, with even more details and example. We're also working on making resolv_wrapper usable on other platforms than Linux, although there are still some bugs here and there.

Add sudo rules to Active Directory and access them with SSSD

Centralizing sudo rules in a centralized identity store such as FreeIPA is usually a good choice for your environment as opposed to copying the sudoers files around – the administrator has one place to edit the sudo rules and the rule set is always up to date. Replication mitigates most of the single-point-of-failure woes and by using modern clients like the SSSD, the rules can also be cached on the client side, making the client resilient against network outages.

What if your identity store is Active Directory though? In this post, I'll show you how to load sudo rules to an AD server and how to configure SSSD to retrieve and cache the rules. A prerequisite is a running AD instance and a Linux client enrolled to the AD instance using tools like realmd or adcli. In this post, I'll use dc=DOMAINNAME,dc=LOCAL as the Windows domain name.

The first step is to load the sudo schema into the AD server. The schema describes the objects sudo uses and their attributes and is not part of standard AD installations. In Fedora, the file describing the schema is part of the SUDO RPM and is located at /usr/share/doc/sudo/schema.ActiveDirectory. You can copy the file to your AD server or download it from the Internet directly.

Next, lauch the Windows command line and load the schema to AD's LDAP server using the ldifde utility:

ldifde -i -f schema.ActiveDirectory -c dc=X dc=DOMAINNAME,dc=LOCAL

Before creating the rule, let's also crate an LDAP container that would store the rules. It's not a good idea to mix sudo rules into the same OU that already stores other objects, like users – a separate OU makes management easier and allows to set more fine-grained permissions. You can create the sudoers OU in "ADSI Edit" quite easily by right-clicking the top-level container (dc=DOMAINNAME,dc=LOCAL), selecting "New->Object". In the dialog that opens, select "organizationalUnit", click "Next" and finally name the new OU "sudoers". If you select a different name or a different OU altogether, you'll have to set a custom ldap_sudo_search_base in sssd.conf, the default is ou=sudoers,$BASE_DN".

Now, let's add the rule itself. For illustration purposes, we'll allow the user called 'jdoe' to execute less on all Linux clients in the enterprise.

In my test, I used "ADSI Edit" again. Just right-click the SUDO container, select "New->Object" and then you should see sudoRole in the list of objectClasses. Create the rule based on the syntax described in the sudoers.ldap man page; as an example, I created a rule that allows the user called "jdoe" to run less, for instance to be able to inspect system log files.

dn: CN=lessrule,OU=sudoers,DC=DOMAINNAME,DC=LOCAL
objectClass: top
objectClass: sudoRole
cn: lessrule
distinguishedName: CN=lessrule,OU=sudoers,DC=DOMAINNAME,DC=LOCAL
name: lessrule
sudoHost: ALL
sudoCommand: /usr/bin/less
sudoUser: jdoe

The username of the user who is allowed to execute the rule is stored in the sudoUser attribute. Please note that the username must be stored non-qualified, which is different from the usual username@DOMAIN (or DOM\username) syntax used in Windows.For a more detailed description of how the sudo rules in LDAP work, refer to the sudoers.ldap manual page.

The client configuration involves minor modifications to two configuration files. First, edit /etc/nsswitch.conf and append 'sss' to the 'sudoers:' database configuration:

sudoers: files sss

If the sudoers database was not present in nsswitch.conf at all, just add the line as above. This modification would allow SSSD to communicate with the sssd with the libsss_sudo library.
Finally, open the /etc/sssd/sssd.conf file and edit the [sssd] section to include the sudo service:

services = nss, pam, sudo

Then just restart sssd and the setup is done! For testing, log in as the user in question ("jdoe" here) and run:

sudo -l

You should be able to see something like this in the output:
User jdoe may run the following commands on adclient:
(lcl) /usr/bin/less

That's it! Now you can use your AD server as an centralized sudo rules storage and the rules are cached and available offline with the SSSD.

COPR repos with newer SSSD versions for RHEL-5 and RHEL-6

Two interesting COPR repos with SSSD packages were made available recently.

One was prepared by Stephen Gallagher and contains SSSD 1.9.x for RHEL-5. I created the other one with SSSD 1.11 built for RHEL-6. I'd love to see test reports for the RHEL-6 repo, as we are considering upgrading to 1.11 in RHEL-6.6.

For more details on the repos see the announcements on sssd-devel about both Stephen's and mine repos.

Enrolling an Active Directory RHEL-6 client machine using adcli

If you're adding a modern Linux client to an Active Directory domain, you really should be using realmd. It's easy to use, secure and does the right thing by default.

If you haven't heard about realmd already, check out the documentation. In a nutshell, realmd makes the client enrollment as easy as:


# realm join

However, realmd depends on some software that is not available on stable platforms used in production, like RHEL-6 and its derivatives. Still, it's possible to use some of the components realmd builds on separately and have a reasonably user-friendly experience. In this blog post, I’ll show you how, using a package called adcli, that is usually just a building block of realmd.

My test AD server domain is called win.example.com and the server that runs the domain is called server.win.example.com. For the test, I've used a mostly default CentOS 6.5 VM.

Typically, you'll want to point your Linux machine to the AD server for DNS:


# cat /etc/resolv.conf
nameserver 192.168.122.70
# host 192.168.122.70
70.122.168.192.in-addr.arpa domain name pointer server.win.example.com.

Start the setup by enabling the EPEL repository and installing the 'adcli'
package:


# yum install adcli

You can type just 'adcli' to get an overview of what commands adcli supports. We're interested in joining the client to the AD domain in order to be able to log in as users from Active Directory.

Now you should be able to find your domain already:


# adcli info win.example.com
[domain]
domain-name = WIN.EXAMPLE.COM
domain-short = WIN
domain-forest = WIN.EXAMPLE.COM
domain-controller = SERVER.WIN.EXAMPLE.COM
domain-controller-site = Default-First-Site-Name
domain-controller-flags = pdc gc ldap ds kdc timeserv closest writable good-timeserv full-secret ads-web
domain-controller-usable = yes
domain-controllers = SERVER.WIN.EXAMPLE.COM
[computer]
computer-site = Default-First-Site-Name

As you can see, adcli was able to discover quite a few details about my test domain, so it’s time to join the client:


# adcli join win.example.com
Password for Administrator@WIN.EXAMPLE.COM:

You’ll be prompted for the Administrator password by default, but it’s possible to specify another user with the -U option. See the adcli man page for full list of details.

The join operation creates a keytab the machine will authenticate with. When you inspect the with klist -kt, you should see several entries that contain you client hostname in some form. Here are the keytab contents on my test system:


# klist -k | head
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
4 RHEL6$@WIN.EXAMPLE.COM
4 RHEL6$@WIN.EXAMPLE.COM
4 RHEL6$@WIN.EXAMPLE.COM
4 RHEL6$@WIN.EXAMPLE.COM
4 RHEL6$@WIN.EXAMPLE.COM
4 HOST/RHEL6@WIN.EXAMPLE.COM
4 HOST/RHEL6@WIN.EXAMPLE.COM

It’s recommeded to also configure /etc/krb5.conf to use the AD domain:


[libdefaults]
default_realm = WIN.EXAMPLE.COM
dns_lookup_realm = true
dns_lookup_kdc = true
ticket_lifetime = 24h
renew_lifetime = 7d
forwardable = true

[realms]
WIN.EXAMPLE.COM = {
kdc = server.win.example.com
admin_server = server.win.example.com
}

[domain_realm]
.win.example.com = WIN.EXAMPLE.COM
win.example.com = WIN.EXAMPLE.COM

The final step is setting up the SSSD (or Winbind if you like) to actually make use of the keytab to resolve users. I’ll show how to use the AD back end of SSSD as an example. Make sure sssd and authconfig are installed:


# yum install authconfig sssd

Unfortunately, authconfig in RHEL-6 doesn’t support configuring the AD back end directly, so you’ll have to do a bit of manual configuration. We can still use authconfig to set up the Name Service Switch and PAM stacks:


# authconfig --enablesssd --enablesssdauth --update

Now you should see ‘sss’ being present in /etc/nsswitch.conf and the pam stack configuration:


# grep sss /etc/nsswitch.conf
passwd: files sss
shadow: files sss
group: files sss
services: files sss
netgroup: files sss

The final step is to configure the SSSD itself. Open /etc/sssd/sssd.conf and define a single domain:


[sssd]
services = nss, pam, ssh, autofs
config_file_version = 2
domains = WIN.EXAMPLE.COM

[domain/WIN.EXAMPLE.COM]
id_provider = ad
# Uncomment if service discovery is not working
# ad_server = server.win.example.com

Start the SSSD and make sure it’s up after reboots:


# service sssd start
# chkconfig sssd on

You should now be able to log in as an AD user just fine:


su - administrator@win.example.com
Password:
-sh-4.1$ id
uid=388000500(administrator) gid=388000513(domain users) groups=388000513(domain users),388000512(domain admins),388000518(schema admins),388000519(enterprise admins),388000520(group policy creator owners),388000572(denied rodc password replication group),388001123(supergroup) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

For more information on the SSSD ad provider, see the SSSD wiki page or just sssd-ad the manual page.

Why is id so slow with SSSD?

Every once in a while, when debugging an SSSD performance problem for a user or a customer, I see that even experienced users tend to measure login performance by running id(1). That’s not really the best thing to do, for one reason – running the plain id command does much more than what happens during login, and of course, at a cost.

Typically, the login program such as ssh or gdm, needs to perform a couple of tasks aside verifying the user’s credentials. These include finding out if the user exists, what shell and home directory does he have and what groups the user is a member of, so that the user can access the files he should be allowed to (and vice versa). These tasks boil down to two corresponding glibc calls – getpwnam to find the details about the user, and getgrouplist to retrieve the list of groups he is a member of.

Because these two library functions are used so often, we take special care in the SSSD to make sure that we use any optimization that is available, such as the transitive memberOf attribute when IPA is used or the tokenGroup attribute when the AD provider is configured. In order to measure what the performance of Name Service Switch calls the login program does is like, you can call "id -G $username" from the command line.

Notice the extra "-G" switch. That really makes a bit of a difference, because the getgrouplist operation returns a list of numerical IDs the user is a member of. That’s usually good enough for the login programs to set the groups for the user who logs in but not really friendly for the admin inspecting the output of the id command.

In contrast to "id -G $username", "id $username" does one operation that sounds trivial but can be extremely expensive – resolves the group GIDs to group names. While that sounds like a really easy operation, it involves calling getgrgid for each of the GIDs returned by getgroupslist. And there comes the slowdown, the getgrgid operation is kind of an all-or-nothing call. It retrieves not only the information about the group itself, such as its name, but also all information about the members of the group, including all the users. This can get quite expensive, consider a university setup where each student was a member of group “students” that consisted of all students on the university.

A legitimate question might be whether its possible to restrict the amount of information the get-by-GID call retrieves. And the answer is both yes and no – the POSIX interface doesn’t allow any such query directly, but the SSSD offers several means to speed up the overall processing. One quite recent addition is the ignore_group_members configuration option that was contributed by Paul Henson. Setting this option to True causes all groups to effectively appear empty, avoiding the need to download the members. Keep in mind that with many server implementations, the members might also include other nested groups which causes the whole operation to recurse.

The SSSD Active Directory provider, part 2

The previous post gave a high level overview of what new features are present in the AD provider starting with 1.9 and especially in the latest versions. This post will illustrate these AD specific features in more detail. Because the AD backend is still undergoing rapid development, most features are accompanied with the version it appeared in.

Faster logins

One of very common complaints about using the LDAP provider with SSSD was that logins are too slow. Typically this was the case with very large and nested group memberships the user was a member of, as the SSSD previously crawled the LDAP directory, looking up the groups.

The AD provider is able to take advantage of a special attribute present in Active Directory called tokenGroups to read all the groups is a member of in a single call. This performance enhancement can reduce the number of LDAP calls needed to find the group memberships to a single one, drastically improving the login time.

The tokenGroups attribute is only leveraged if the SSSD maps the ID values from SIDs, not when POSIX attributes are used in the older versions, up to 1.11.3. With 1.11.3 or later, the tokenGroups attribute is leveraged even when POSIX attributes are used instead of automatic mapping.

Dynamic DNS updates

Clients enrolled to an Active Directory domain may be allowed to update their DNS records stored in AD dynamically. At the same time, Active Directory servers support DNS aging and scavenging, which means that stale DNS records might be removed from AD after a period of inactivity.

The AD provider supports both scenarios described above by default. The AD provider will attempt to update the DNS record every time the AD provider goes online (typically after startup) and periodically every 24 hours to keep the records from being scavenged.

Dynamic NetBIOS name discovery

When referring to a user from an Active Directory domain, typically the domain is part of the identifier. In contrast to UNIX users that are normally just referred to with the username:
id root

The AD users can be referred to either by fully qualifiying the name with the AD domain:
id Administrator@ad.example.com

Or just using the NetBIOS name of the AD domain:
id AD\\Administrator

In most deployments, the NetBIOS name is just the first part of the full domain name, but not always. The NetBIOS name can be customized on the AD side by the administrator.The AD provider of SSSD is able to recognize both formats and autodetect the right NetBIOS name as well.

DNS site discovery

Large Active Directory environments often span across multiple locations in multiple geographies. In Active Directory, these physical locations are represented as "sites". Each client belongs to a site based on the network subnet it resides in. For best performance, it is important that the clients are able to find the closest site and use the other domain controllers only as a fallback.

The support for discovering the closes site was added in SSSD 1.10 and is enabled by default.

Support for trusted domains in the same forest

Starting with version 1.10, the SSSD is able to dynamically find the trusted domains in the same forest and provide both authentication and identity information for users coming from the trusted domains. The SSSD retrieves identity information from the Global Catalog, so it's important that the users and all needed attributes are replicated to the Global Catalog. This includes even POSIX attributes such as home directory, login shell and most importantly UIDs and GIDs if not using ID mapping.

Prior to 1.10, it was somewhat possible to configure the SSSD to fetch identity data from trusted domain, but the administrator had to represent each domain with a separate [domain] stanza in the config file. Each domain stanza had to be fully configured as a separate identity source including search bases and host names. Moreover, groups could only contain members from the same domain. The native support also requires no configuration at all, the trusted domains are discovered on the fly.

Support for enterprise logins

Some users in AD might use a different Kerberos Principal suffix than the default one. This feature is on by default in the AD provider and was introduced in SSSD 1.10. This feature is also required to support logins of users from trusted domains.

A simplified LDAP access control mechanism

Starting with upstream version 1.11.2, there is a simplified way to express the access control using an LDAP filter with the AD backend. The administrator can now only specify access_provider=ad and then use the
access filter with an option aptly called ad_access_filter.

The following example illustrated restricting access to users whose name begins with 'jo' and have a valid home directory attribute:
access_provider = ad
ad_access_filter = (&(sAMAccountName=jo*)(unixHomeDirectory=*))

In addition to checking whether the user matches the filter, the AD access filter also checks the account validity. Expired accounts are not permitted even if they matched the filter. Previously, the administrator had to configure the LDAP access filter and specify all the options manually.

The example used above would the look much more involved:
access_provider = ldap
ldap_access_order = filter, expire
ldap_account_expire_policy = ad
ldap_access_filter = (&(sAMAccountName=jo*)(unixHomeDirectory=*))
ldap_sasl_mech = GSSAPI
ldap_sasl_authid = CLIENT_SHORTNAME$@EXAMPLE.COM
ldap_schema = ad

Still, for most deployments, the simple access provider is the best choice for the ease of configuration, especially when it comes to group membership.

The SSSD Active Directory provider

This post intends to introduce a feature of SSSD, that, despite being around since the release of 1.9, is still not used as often as it should – the Active Directory backend.

Even though 1.9 was released more than a year ago, I still see many deployments configuring the pure LDAP backend when configuring an AD client machine. I'll try to explain the advantages of the AD backend compared to the LDAP backend, but in short, you should always use the AD backend when configuring SSSD with an AD server.

Below is a summary of the biggest advantages of the AD provider in my opinion. A more detailed description of the new features will follow in a next blog post.

The enrollment using realmd is easier and more secure

This is technically not a feature of the AD backend, but it's still worth noting.

There are several ways to enroll a Linux client machine to AD – generate a keytab on Windows, use Samba, etc. All of them require some amount of knowledge and manual tweaking – refer to the SSSD wiki page
for details. In contrast, realmd is a great tool written by a friendly upstream that reduces all that effort into a single line of shell command:

# realm join ad.example.com

You'd be queried for AD Administrator credentials by default, but any user authorized to join a client to the realm can be used. Realmd also supports one-time passwords and more. After the join finishes, the client machine will run the SSSD with the AD provider configured by default, but winbind is also available, if you prefer that option. Realmd is included in the last couple of Fedora releases, starting with Fedora 18. If you are running an older distribution, such as RHEL6, I'd advise to use the underlying client tool called adcli, which is available from EPEL.

The configuration is simpler

Even if you opt for manual configuration or have a client already joined to a domain, the simplified configuration might be of interest. While AD can be treated just as an LDAP/Kerberos combo, several configuration options need to be tailored in order to match what is stored on the server
side.

For example, user objects in AD are of objectclass user. In contrast, the LDAP provider defaults to posixAccount. Active Directory is also case insensitive, which requires the use of the "case_sensitive" option. The AD provider already comes with all the defaults set out of the box, so the previously complex configuration can be simplified to:


[domain/ad.example.com]
id_provider = ad

The example above assumes that UIDs and GIDs are mapped automatically by the SSSD and the AD servers are autodiscovered from DNS.

AD specific features included

In comparison to using the generic LDAP and Kerberos providers, the AD provider allows the client machine to use several features unique to the AD backend. In particular:

  • logins are faster as the AD provider can leverage the special tokenGroups feature
  • the client machine is able to update or refresh its DNS records
  • the NetBIOS domain name can be autodiscovered and used in both lookups and output format (getent passwd AD\\Administrator now works)
  • clients are able to automatically discover the closest AD server to connect to using the 'sites' feature of AD
  • the AD provider automatically discovers trusted domains in the same forest, allowing all users from the same forest to log in to the machine
  • expressing access control with an LDAP filter was made much simpler with a new configuration option
  • custom UPN suffixes, also known as Enterprise Principals are supported by default

The next post will illustrate these AD specific features in more detail.