pam_hbac: A PAM module to enforce IPA access control rules

Written by Jakub Hrozek and Pavel Reichl

FreeIPA is an open source identity management software. One of the nice features is the ability to limit which users can log in to which servers using Host Based Access Control (HBAC). Previously, only SSSD was able to parse and evaluate the HBAC rules – however, some users run operating systems that either do not support SSSD at all, or SSSD can only be configured in a way that doesn’t include the HBAC engine.

To provide HBAC support for these platforms, we wrote a new PAM module called pam_hbac. The first release of pam_hbac just happened and in this blog post we would like to introduce the project and show how it works.

pam_hbac connects to an IPA server, downloads the HBAC access control rules, evaluates them and decides whether access should be allowed or denied. Even though pam_hbac is a project on its own, it uses the same code to evaluate the HBAC rules as SSSD does, so the resulting access control decision will be the same for the same rules provided the input is the same as well.

Using pam_hbac should come with a disclaimer – if your operating system supports SSSD and you can use its IPA id_provider, please use SSSD instead of pam_hbac. SSSD is maintained by a large team of developers, it is included in distributions with commercial support available and has several advantages over pam_hbac, including offline caching or Kerberos authentication using the host keytab by default.

Who should use pam_hbac

There are legitimate use-cases where using a standalone PAM module like pam_hbac is required, though. These include:

  • if your IPA client runs an OS that doesn’t support SSSD at all like Solaris or its derivatives
  • if your IPA client runs an OS that does support SSSD but not its IPA provider. At the moment, this is the case with Amazon’s Linux distribution and also FreeBSD.
  • if your IPA client runs an OS that supports the IPA provider, but the IPA provider lacks the support for users from trusted Active Directory domains. This is the case for clients running RHEL-5 and leveraging the “compat” LDAP tree provided by the slapi-nis plugin of IPA.

At the moment, pam_hbac runs on Linux, FreeBSD and Solaris (tested on Oracle Solaris 11 and Omnios)

Obtaining and installing pam_hbac

Development of pam_hbac happens on github. There you can clone the git repository and compile pam_hbac yourself. Because the required dependencies or configure flags differ for each platform a bit, please refer to per-platform README files in the doc directory. As an example, there is a FreeBSD-specific README file that contains step-by-step instructions for building and installing pam_hbac on FreeBSD.

For RHEL-5 and RHEL-6, we have also provided a COPR repository with prebuilt binaries. On these platforms, you can just drop the .repo file into /etc/yum.repos.d/ and then yum install pam_hbac.

We would certainly welcome contributors who would like to provide prebuilt binaries for different platforms!

Examples

In the rest of the blog posts, we illustrate two examples of pam_hbac in action. One would be using pam_hbac on FreeBSD to grant access to a subset of IPA users and another one would be to use pam_hbac on CentOS-5 to restrict access to a single Active Directory group in a setup with trusts. For both examples, we use the same environment – the IPA domain is called IPA.TEST and is managed by an IPA server called unidirect.ipa.test. Our clients are called freebsd.ipa.test and centos5.ipa.test.

Example 1: using pam_hbac to provide access control for IPA users on FreeBSD

Even though the recent FreeBSD releases do ship SSSD, it is not built with the IPA provider by default (only through extra flags) and therefore HBAC enforcement might not be available easily. However, we can configure SSSD with the LDAP id_provider or just nss-pam-ldapd on FreeBSD and use pam_hbac for access control separately.

Our goal is to make it possible only for the bsduser to log in to the FreeBSD client machine and nobody else. Start the configuration by making sure you can resolve and authenticate the IPA users. Once that is done, we can configure pam_hbac to provide access control for the FreeBSD machine. Without access control configured, any user should be able to log in:

% su - bsduser
Password:
$ id
uid=1207000884(bsduser) gid=1207000884(bsduser) groups=1207000884(bsduser)
% ^D
% su - tuser
Password:
$ id
uid=1207000883(tuser) gid=1207000883(tuser) groups=1207000883(tuser)

The next step is to install and configure and pam_hbac. Because at the moment there are no prebuilt binary packages for FreeBSD, you’ll need to compile the module from source. The steps are documented in the FreeBSD README in pam_hbac’s git repo . After configuring the source tree, building and installing pam_hbac, you’ll end up with the module installed to /usr/local/lib/pam_hbac.so.

Because much of the information that pam_hbac reads is only accessible to an authenticated user, we need to create a special bind user that pam_hbac will authenticate as. To do so, prepare an LDIF file with the following contents:

dn: uid=hbac,cn=sysaccounts,cn=etc,$DC
objectClass: account
objectClass: simplesecurityobject
objectClass: top
uid: hbac
userPassword: $PASSWORD

Replace the $PASSWORD value with the desired password of the bind user and $DC with the base DN of your IPA server. Then add this LDIF to the IPA server:

ipaserver $ ldapadd -ZZ -H ldap://$IPA_HOSTNAME -D"cn=Directory Manager" -W < hbac_sysuser.ldif

Now we can create the configuration file for pam_hbac. The configuration options are documented in the pam_hbac.conf manpage, but in general it’s enough to point pam_hbac to the IPA server and specify the bind user and its credentials. The config file for pam_hbac on FreeBSD is located at /usr/local/etc/pam_hbac.conf:

[root@freebsd ~]# cat /usr/local/etc/pam_hbac.conf
URI = ldap://unidirect.ipa.test
BASE = dc=ipa,dc=test
BIND_DN = uid=hbac,cn=sysaccounts,cn=etc,dc=ipa,dc=test
BIND_PW = Secret123
SSL_PATH = /usr/local/etc/ipa.crt

Next, we add pam_hbac to the PAM configuration so that it enforces access control during the PAM account phase. Because pam_hbac only handles the account phase, we only add a single line to the account stack of /etc/pam.d/system to make it look like this:

account required pam_login_access.so
account required /usr/local/lib/pam_ldap.so no_warn ignore_authinfo_unavail ignore_unknown_user
account required /usr/local/lib/pam_hbac.so ignore_authinfo_unavail ignore_unknown_user
account required pam_unix.so

Finally, we can disable the allow_all rule on the server and instead only allow access to bsduser to the freebsd.ipa.test machine. Please don’t forget to add other rules in your test environment so that you can at least access your IPA masters!

ipaserver $ ipa hbacrule-add freebsd-bsd-user
ipaserver $ ipa hbacrule-add-host --hosts=freebsd.ipa.test freebsd-bsd-user
ipaserver $ ipa hbacrule-add-user --users=bsduser freebsd-bsd-user
ipaserver $ ipa hbacrule-mod --servicecat=all freebsd-bsd-user
ipaserver $ ipa hbacrule-disable allow_all

Time to test pam_hbac! First, we can make sure bsduser is still able to log in:

% su - bsduser
Password:
$ id
uid=1207000884(bsduser) gid=1207000884(bsduser) groups=1207000884(bsduser)
$ ^D

OK, now for a negative test, see if tuser is denied access:

% su - tuser
Password:
su: Sorry

Great! Looking to /var/log/auth.log reveals that it was indeed the account control module that denied access:

May 25 13:26:37 su: pam_acct_mgmt: permission denied

Example 2: using pam_hbac to provide access control for AD users on CentOS-5

One important use-case for pam_hbac is to provide access control for setups that resolve users from a trusted AD domain using the ‘legacy client’ setup in which a CentOS-5 machine is set up with id_provider=ldap pointing to the IPA server’s compat tree. Please note that if your IPA domain doesn’t have a trust relationship established with an AD domain, you can already use HBAC provided by SSSD and you don’t need pam_hbac at all in that case.

In our setup, let’s have an AD group linux_admins. Our goal will be to grant access to the CentOS-5 machine to members of linux_admins and nobody else. First, make sure your CentOS-5 client is able to resolve and authenticate AD users and the user is a member of the linux_admins group. You can use the output of ipa-advise config-redhat-sssd-before-1-9 as a starting point. Once the CentOS-5 client is set up, you’ll be able to resolve the user, id should report the group and you should be able to authenticate as that user. Because HBAC rules can only be linked to IPA POSIX groups, we also need to add the AD group as a member of an IPA external group which in turn needs to be added to an IPA POSIX group:

ipaserver $ ipa group-add --external linux_admins_ext
ipaserver $ ipa group-add-member --groups=linux_admins_ext ipa_linux_admins
ipaserver $ ipa group-add-member --external=linux_admins@win.trust.test linux_admins_ext

Try logging in:

$ su - linuxop@win.trust.test
Password:
-sh-3.2$ id
uid=300403108(linuxop@win.trust.test) gid=300403108(linuxop@win.trust.test) groups=300400513(domain users@win.trust.test),300403108(linuxop@win.trust.test),300403109(linux_admins@win.trust.test),1207000858(ipa_linux_admins) context=user_u:system_r:unconfined_t

OK, the id output reports the user is a member of the ipa_linux_admins group, so we can proceed with setting up the HBAC rules.

In order to set up the HBAC rules correctly, it’s important to understand how AD users are authenticated when using the compat tree – the SSSD client on the CentOS-5 machine does an LDAP bind using the user’s password against the IPA compat tree. This password bind is intercepted by the slapi-nis plugin running on the IPA server which in turn authenticates against the system-auth service on the IPA server itself. Therefore, it’s important that all users who should be allowed to authenticate against the compat tree are allowed access to the system-auth service on the IPA server. More details about the authentication against the compat tree can be found in the

Let’s add the system-auth rule first, together with its HBAC service that allows access to everyone to the IPA server itself using the system-auth PAM service:

ipaserver $ ipa hbacsvc-add system-auth
ipaserver $ ipa hbacrule-add system-auth-everyone
ipaserver $ ipa hbacrule-add-host --hosts=unidirect.ipa.test system-auth-everyone
ipaserver $ ipa hbacrule-mod --usercat=all system-auth-everyone

The resulting HBAC rule would look like this:

ipaserver $ ipa hbacrule-show system-auth-everyone
Rule name: system-auth-everyone
User category: all
Enabled: TRUE
Hosts: unidirect.ipa.test
Services: system-auth

To avoid allowing all access, disable the allow_all rule

ipaserver$ ipa hbacrule-disable allow_all

As a pre-flight check, you can disable the system-auth-everyone rule, then your user should be denied access and the server-side journal should show something like:

pam_sss(system-auth:account): Access denied for user linuxop@win.trust.test: 6 (Permission denied)

Enabling the rule would allow access again. Of course, don’t forget to allow access to your IPA servers and clients with other services and for other users as appropriate!

As the final step on the server, we can define the HBAC rule for our particular CentOS-5 machine. While the access was already checked against the system-auth rule on the server, that rule cannot discriminate between different hosts and applies to all authentication requests coming from the slapi-nis plugin.

Please note that the a bind user must be configured, refer to the BSD example for details.

The rule will permit all members of ipa_linux_admins group to access all PAM services on the host called centos5.ipa.test:

ipaserver $ ipa hbacrule-add centos5-ipa-linux-admins
ipaserver $ ipa hbacrule-add-host --hosts=centos5.ipa.test centos5-ipa-linux-admins
ipaserver $ ipa hbacrule-mod --servicecat=all centos5-ipa-linux-admins
ipaserver $ ipa hbacrule-add-user --groups=ipa_linux_admins centos5-ipa-linux-admins

Now we’re ready to configure the CentOS-5 client. Make sure pam_hbac is installed first – you can use our COPR repository for that. Just drop the repo file to /etc/yum.repos.d and then yum install pam_hbac.

The configuration file for CentOS-5 machine is quite similar to the one we used on BSD earlier:

URI = ldap://unidirect.ipa.test
BASE = dc=ipa,dc=test
BIND_DN = uid=hbac,cn=sysaccounts,cn=etc,dc=ipa,dc=test
BIND_PW = Secret123

The file is located at /etc/pam_hbac.conf.

Next, add pam_hbac to the /etc/pam.d/system-auth file on the CentOS-5 machine to enable HBAC rules enforcement. This snippet shows the whole account stack on my CentOS-5 machine:

account required pam_unix.so
account sufficient pam_succeed_if.so uid < 500 quiet
account [default=bad success=ok user_unknown=ignore] pam_sss.so
account sufficient pam_localuser.so
account [default=bad success=ok user_unknown=ignore] pam_hbac.so
account required pam_permit.so

You can see I also added the pam_localuser.so module just before the line with pam_hbac.so. Adding the pam_localuser.so module ensures that pam_hbac wouldn’t be called for local users defined in /etc/passwd – we only want the HBAC policies to apply to IPA and AD users.

It’s time to check the rule. As a positive check, we log in with the linuxop user:

$ su - linuxop@win.trust.test
Password:
su: warning: cannot change directory to /home/win.trust.test/linuxop: No such file or directory
-sh-3.2$

As a negative test, we can try logging in as the AD administrator perhaps:

$ su - administrator@win.trust.test
Password:
su: incorrect password

And indeed, /var/log/secure would tell us it was pam_hbac that denied access:

May 25 22:48:35 centos5 su: pam_hbac(su-l:account): returning [6]: Permission denied

Awesome, now even your CentOS-5 server enforces HBAC rules!

Troubleshooting pam_hbac

In case something doesn’t work as expected, there are several ways to debug pam_hbac and the HBAC access control in general. The first step should be to check with the ipa command line tools and the hbactest plugin. For example, this is how we’d test the bsduser‘s access to the freebsd.ipa.test machine:

ipaserver $ ipa hbactest --user=bsduser --service=su --host=freebsd.ipa.test
--------------------
Access granted: True
--------------------
Matched rules: freebsd-bsd-user
Not matched rules: centos5-ipa-linux-admins
Not matched rules: ph_test_trule
Not matched rules: system-auth-everyone

The pam_hbac module by default only logs failures. If you want to see more verbose output, add the debug parameter to the PAM service file configuration. All logging, including debug logs is done using the standard pam_syslog() PAM calls, so the location really depends on your operating system. But as an illustration, this is the tail of the debug output for the AD user case when the allowed user logs in:

May 25 22:48:47 centos5 su: pam_hbac(su-l:account): ALLOWED by rule [centos5-ipa-linux-admins].
May 25 22:48:47 centos5 su: pam_hbac(su-l:account): hbac_evaluate() >]
May 25 22:48:47 centos5 su: pam_hbac(su-l:account): Allowing access
May 25 22:48:47 centos5 su: pam_hbac(su-l:account): returning [0]: Success

The full debug output breaks down all the rules into their components and shows what matched and what did not.

Conclusion

This blog post described the first version of pam_hbac. We will continue the development to add more supported platforms – in the next version, we would like to add support for IBM AIX and Apple OS-X. There are also several bugs and minor enhancements we would like to add. Feel free to file an issue on github if there is something you would like to see improved or something doesn’t work for you!

Creating large AD test environments from the Linux command line

I’ve been recently working on making SSSD perform better in large environments, that contain thousands of users or groups. And because two of the types of setup I wanted to test were SSSD directly enrolled into an Active Directory domain and SSSD as a client of an IPA server that trusts an Active Directory domain, I needed to create a large Active Directory environment for testing.

Of course I didn’t want to create the users in the AD Users and Groups GUI snap-in one by one. And I preferred to create the users and groups from the Linux command line, because that’s what I’m familiar with. After some tinkering, I realized the adcli command already does exactly what I need to with its “create-user”, “create-group” and “add-member” commands.

For example, creating an active directory user is as simple as:

 $ adcli create-user --domain=your.ad.domain adusername

To create a group, you’d call:

 $ adcli create-group --domain=your.ad.domain adgroupname

And finally add a user to the group:

 $ adcli add-member --domain=your.ad.domain adgroupname adusername

By default, adcli would ask you for the AD Administrator, but using a Kerberos ccache is as simple as adding the “-c” option.

Using these three commands I wrote three simple shell scripts that helped me create a thousand users, a thousand groups and add members to these groups in one go. The scripts are available at: https://github.com/jhrozek/adcli_scripts

Of course, adcli does much more than just creating user or group object in Active Directory. adcli is a powerful command-line tool for performing actions on an Active Directory domain. You probably used it already without realizing if you ever added an Active Directory client with realmd, because realmd uses adcli internally. In particular, it would use the “adcli join” command.

On RHEL-6, adcli is available already in EPEL and will be available in RHEL-6.8 as a regular RHEL package. In RHEL-7 and supported Fedora distributions, both realmd and adcli are already available in repositories.

SSSD local overrides

Written by Pavel Březina and Jakub Hrozek

In most cases, using the SSSD is all about connecting a client machine to a central user database, like FreeIPA or Active Directory precisely because you want all users on all machines across the domain to have exactly the same properties. However, even though it would be best to centralize all the things, there will always be exceptions. There can be an odd legacy server where a particular user has a UID set to a value that predates the migration to a central domain controller. Or you might administer a machine where a user stores her files in a different home directory than any other machines.

If the client is a member of an FreeIPA domain, you can just define an “ID view” and define the custom values centrally on the IPA server. Using ID views is the recommended way of setting any overriden attributes – however, not all environments use a FreeIPA server..

To solve those use-cases, the SSSD provides a command-line tool that allows the administrator to set one or more POSIX attributes to a different value on that particular system. The tool is called sss_override and is part of the sssd-tools package since version 1.13.1. Running sss_override --help would present the user with a list of available sub-commands. We’ll show some examples of using them later in this text. Please note that the local overrides can not be used in combination with IPA provider. Instead, the server side overrides can be defined directly through IPA web or command line interface.

One very important aspect to keep in mind is that the overrides are stored in the cache, along with the data from the directory server. This conflicts with a very commonly used practice among admins who often remove the database and try to start from a clean cache when they encounter any problems with sssd. Therefore it is necessary to manually export the override data prior deleting the cache and import them again later, otherwise the override data would be lost!

Creating user and group overrides

Let’s consider our domain has a user “tuser” that resolves with his UID and GID as defined on the server:
$ getent passwd tuser
tuser:*:1190000015:1190000015:test user:/home/tuser:/bin/sh
But on this particular host, we need this user to have a different IDs – maybe the user already owned some files before this host was added as a member to the domain. We can call the sss_override tool and assign him a different ID:
$ sudo sss_override user-add tuser -u 1234
SSSD needs to be restarted for the changes to take effect.
As the info message tells us, currently SSSD needs to be restarted when the first override is added in order for the overriden values to take effect. All subsequent overrides will take effect immediately.
$ sudo systemctl restart sssd
Now, let’s request the user again:
$ getent passwd tuser
tuser:*:1234:1190000015:test user:/home/tuser:/bin/sh
And the changes are visible now! Keep in mind that user-add always replaces the whole local override, so if we wanted to override this user’s name, too, the previous UID change would be lost:
$ sudo sss_override user-add tuser -n xuser
$ getent passwd xuser
xuser:*:1190000015:1190000015:test user:/home/tuser:/bin/sh
As you can see, only the login name was overriden, but the UID was reverted to the original value. This is because the -add command replaces the override completely. The correct way to override both values would be to specify both switches at the same time:
$ sudo sss_override user-add tuser -n xuser -u 1234
It’s possible to also override group attributes – at the moment, overriding group’s name or GID is supported. A very common use is to override a group ID, so let’s take a look:
$ id tuser
uid=1190000015(tuser) gid=1190000015(tuser) groups=1190000015(tuser),1190000016(tgroup)
$ sudo sss_override group-add tgroup -g 5555
$ id tuser
uid=1190000015(tuser) gid=1190000015(tuser) groups=1190000015(tuser),5555(tgroup)

Import and export

As explained earlier, the overrides are stored in the sssd cache itself. To provide some recovery mechanism for admins who sometimes need to purge the cache in order to reset the environment into clean state, we added import and export commands to the sss_override tool. With the example above, we can export the override for tuser with:
$ sudo sss_override user-export /var/log/sssd/user_overrides
This creates a passwd-like file that contains the override data:
$ sudo cat /var/log/sssd/user_overrides
tuser@ipaldap:xuser:1234::::
 In case we decide to remove the cache, we can later add the override data back with:
$ sudo sss_override user-import  /var/log/sssd/user_overrides


Listing

To see what overrides are actually configured on the system, you can use sss_override commands user-find, user-show, group-find and group-show.

The command sss_override user-find allows to find all overrides from all configured sssd domains:
$ sudo sss_override user-find
tuser@ipaldap:xuser:1234::::
admin@anotherdomain:administrator:4321::::
We can also direct the user-find command to a particular domain:
$ sudo sss_override user-find -d ipaldap
tuser@ipaldap:xuser:1234::::
 Or, using the user-show command, even a particular user:
$ sss_override user-show tuser
tuser@ipaldap:xuser:1234::::

Conclusion

The local overrides are a powerful mechanism to deal with irregularities in an otherwise centralized environment. That said, using a completely centralized solution, like IPA’s idviews feature avoids issues with keeping the overrides in the cache itself as well as provides a unified view of all overrides in the UI.

As an incremental improvement, the SSSD team is looking into ways of making the import and export more automated so that the danger of removing the database is mitigated by the SSSD tools.

Performance tuning SSSD for large IPA-AD trust deployments

Written by Alexander Bokovoy and Jakub Hrozek

This blog post describes several sssd.conf options that are available for performance tuning of SSSD, especially focusing on deployment of an IPA server with trust established with an AD server. Some of the options are useful for other scenarios as well, but it should be noted that diverting from the defaults is something that needs understanding of the implication the change has. This post is also written with SSSD version 1.12 and 1.13 in mind – later versions might have different tunables available or might not need these at all as we continuously improve performance.

The anatomy of a trusted identity lookup

The typical use-case this post is trying to address is “a login to my IPA client using AD users and his AD credentials is way too slow”. Later in the article, we’ll be setting several options in the IPA server’s sssd.conf. That might be confusing if you’re not familiar with the architecture of the IPA-AD trust lookups and authentication data flow – so let’s illustrate it briefly.

The two basic requests type the client can perform are identity lookups and authentication. For IPA users, the identity lookups are normal LDAP searches against the IPA server and authentication is performed using an internal kinit-like tool also connecting to the IPA server. Trusted AD users work differently — the identity lookups from the client machines connect to the IPA server using an LDAP extended operation. The IPA server’s extdom plugin queries the SSSD instance running on the server for the AD user information, the SSSD instance runs a search against the Active Directory server using the LDAP protocol and passes the data back to the IPA client. For password-based authentication requests, the IPA clients connect directly to the AD servers using Kerberos.

Keep in mind that because the correct set of groups must be set during authentication time, the authentication request must also perform an identity lookup, typically the initgroups operation, before the authentication. The following diagram illustrates the data flow:

ipa-passwordless-auth

The diagram illustrates the data flow in case a Windows user logs in using his Kerberos credentials (passwordless) to an IPA client:

  1. The windows user opens an SSH client (like putty) and logs in to an IPA client
  2. The SSH daemon initiates processing of the Kerberos ticket
  3. The Kerberos tickets of Active Directory users have a blob attached to them, called MS-PAC (a privilege attribute certificate, see https://msdn.microsoft.com/en-us/library/cc237917.aspx for details of the specification). The MS-PAC (or just PAC going forward) contains the complete and precise list of group memberships at the time user logged in into the Active Directory domain, signed by the domain controller. It is actually the mechanism Windows clients use to set list of the groups the user is a member of. If there is a PAC present, then a special libkrb5 plugin invokes the sssd_pac responder.
  4. The sssd_pac responder parses the group memberships from the PAC blob during login. This processing needs to resolve internal identifiers of groups in Active Directory (SIDs) to their names and POSIX IDs because Linux cannot yet deal with SIDs natively. For each SID identifier that is not stored in cache yet, the sssd_pac process asks the sssd_be process to translate the SID into a name and/or GID.
  5. The sssd_be process calls an LDAP extended operation towards the IPA server, asking for information about a particular SID.
  6. The extended operation is received by the IPA server’s Directory server plugin, that in turn queries the SSSD instance running on the IPA server through the standard NSS interface, which contacts the sssd_nss process on the IPA server
  7. If the information about this object’s SID is not present in the cache on the IPA server itself either, then the sssd_be process is asked to resolve this SID and store the object that corresponds to the SID into the SSSD cache on the server
  8. The sssd_be process on the server searches the Active Directory server’s LDAP store for the corresponding LDAP objects and stores them into the cache. When the object is stored, the flow reverses – the sssd_be process on the server tells the sssd_nss process the cache is up-to-date. The sssd_nss process returns data to the DS plugin on the server, which in turn returns data in the extdom-extop operation reply to the client.
  9. Finally, all the groups from the PAC object are processed. When the SSH daemon on the client opens the session for the user, it would call the initgroups() function to set up group memberships. When this function’s equivalent reaches the sssd_nss responder, all the data would be cached from processing the PAC blob and returned from the SSSD cache.

Please note that for simplicity, this diagram omits additional actions that might happen during login, such as HBAC access control or setting the SELinux label for the user.

If password-based authentication is used, the data flow is quite similar, except the sssd_pac responder is not involved until later in the process when the ticket is acquired on the IPA client itself. With identity lookup (for example id aduser@ad_domain), the PAC responder is not invoked at all, but all lookups are driven completely by the sssd_nss responder. But even here, the general flow between IPA client and IPA server is the same.

Also keep in mind that retrieving members of trusted groups or retrieving user groups prior to login requires RHEL-7.1 or RHEL-6.7 packages on the client side. Additionally, the IPA server must be running RHEL-7.1 or newer at the same time.

Not all IPA masters are capable to resolve AD users this way. In FreeIPA < 4.2, only IPA masters where ipa-adtrust-install command was executed can retrieve information from Active Directory domain controllers. Such IPA masters were also used for establishing actuall cross-forest trust to Active Directory and AD domain controllers were talking back to IPA masters as part of that flow. With FreeIPA 4.2 a concept of ‘AD trust agents’ was introduced that allows other IPA masters to resolve AD users and groups without the need to configure them as a full domain controller.

The data flow also has implications on debugging. If it’s not possible to fetch data about a trusted user on the IPA client (ie getent passwd aduser@ad_domain doesn’t display the user), first make sure it’s possible to run the command on the IPA server. The error messages on the client would normally manifest in functions that contain s2n in the name, such as:

[ipa_s2n_exop_done] (0x0040): ldap_extended_operation result: Operations error(1), Failed to handle the request.

or:

[ipa_s2n_get_user_done] (0x0040): s2n exop request failed.

The most data-intensive operation is usually retrieving the information about user’s groups on the server side. If the group objects are not cached yet, that operation can amount to downloading a large amount of data, storing it in server’s sssd cache, transferring the data to clients and storing them again in client’s sssd cache. Subsequent lookups for the same LDAP objects, even from another client instance would then just read the objects from the server side sssd’s cache, eliminating the server side costs, but the very first search might be costly in a huge environment.

Available tunables

Now that we know how the flow works, we can start optimizing it. Because the heavy lifting of the lookups is done on the server side, the changes in server’s sssd.conf also have the biggest impact on the lookups performance.

Server options

The following options should be added to the /etc/sssd/sssd.conf file on the IPA masters.

ignore_group_members
Normally the most data-intensive operation is downloading the groups including their members. Usually, we are interested in what groups a user is a member of (id aduser@ad_domain) as the initial step rather than what members do specific groups include (getent group adgroup@ad_domain). Setting the ignore_group_members option to True makes all groups appear as empty, thus downloading only information about the group objects themselves and not their members, providing a significant performance boost. Please note that id aduser@ad_domain would still return all the correct groups!

  • Minimal version: For users from IPA domain, this option was introduced in sssd-1.10. However, it’s only possible to use it for trusted domain users via the subdomain_inherit option (see below).
  • Recommended value: ignore_group_members=True unless your environment requires the output of getent group to also contain members
  • Pros: getgrnam/getgrgid calls are significantly faster
  • Cons: getgrnam/getgrgid calls only return the group information, not the members
ldap_purge_cache_timeout
SSSD 1.12.x runs a cache cleanup operation on startup and then periodically. The cache cleanup operation removes cached entries that were not used for a while to make sure the cache doesn’t grow too large. But since SSSD can also operate in offline mode, we must be very conservative about what is removed, otherwise we might lock out user’s access to a file. At the same time, searching the cache and examining the entry could take a lot of CPU time. Therefore, the cache cleanup operation was of limited use and was disabled by default in 1.13. You can disable it in previous versions with a config change as well.

  • Minimal SSSD version: All supported SSSD versions have this option.
  • Recommended value: ldap_purge_cache_timeout = 0
  • Pros: the cache cleanup operation was not particularly useful as a periodic task, but took a long time to execute
  • Cons: the cache can grow in size.
subdomain_inherit
The options above can be passed to the trusted AD domains’ configuration. At the moment, the only supported method is using the subdomain_inherit option in the sssd.conf‘s domain section. Any of the two options’ names from above can be listed as a value of subdomain_inherit and they will apply to both the main (IPA) domain as well as the AD subdomain. In the future, we would prefer to add support for sub-sections in sssd.conf, but we’re not there yet.. The complete list of options that can be inherited by a subdomain is listed in the sssd.conf manual page.

  • Miminal version: Upstream 1.12.5. However, this option has been backported to both RHEL-6.7 and also RHEL-7.1 updates.
  • Pros-Cons: N/A, this is just a control option to extend influence of the options above to domains from a trusted AD forest.

Mount the cache in tmpfs

Quite a bit of the time spent processing the request is writing LDAP objects to the cache. Because the cache maintains full ACID properties, it does disk syncs with every internal SSSD transaction, which causes data to be written to disk. On the plus side, this makes sure the cache is always available in case of a network outage and will be usable after a machine crash, but on the other hand, writing the data takes time. It’s possible to mount the cache into a ramdisk, elliminating the disk IO cost by adding the following to /etc/fstab as a single line:

tmpfs /var/lib/sss/db/ tmpfs size=300M,mode=0700,noauto,rootcontext=system_u:object_r:sssd_var_lib_t:s0 0 0

Then mount the directory and restart the sssd afterwards:

# mount /var/lib/sss/db/
# systemctl restart sssd

Please tune the size parameter according to your IPA and AD directory size. As a rule of thumb, you can use 100MBs per 10000 LDAP entries.

Doing this change on the IPA server is a bit safer than IPA clients, because the SSSD instance on the server will never lose connectivity to the IPA server, so the cache can always be rebuilt. But in case the cache was lost after a reboot and the AD server was not reachable due to a network error or a similar condition, the node would not be able to fall back to cached data about AD users.

  • Pros: I/O operations on the cache are much faster.
  • Cons: The cache does not persist across reboots. Practically, this means the cache must be rebuilt after machine reboot, but also that cached password are lost after a reboot.

All in all, we’re looking at adding the following changes to the server side’s sssd.conf:

[domain/domname]
subdomain_inherit = ignore_group_members, ldap_purge_cache_timeout
ignore_group_members = True
ldap_purge_cache_timeout = 0

and optionally an fstab change to remount the database directory in a
ramdisk.

Client options

The following options should be added to /etc/sssd/sssd.conf on the IPA clients.

pam_id_timeout
On a Linux system, user group membership is set for processes during login time. Therefore, during PAM conversation, SSSD has to prefer precision over speed and contact the server for accurate information. However, a single login can span over multiple PAM requests as PAM processing is split into several stages – for example, there might be a request for authentication and a separate request for account check (HBAC). It’s not beneficial to contact the server separately for both requests, therefore we set a very short timeout for the PAM responder during which the requests will be answered from in-memory cache. The default value of 5 seconds might not be enough in cases where complex group memberships are populated on server and client side. The recommended value of this option is as long as a single un-cached login takes.

  • Recommended value: pam_id_timeout = login_time_in_seconds
  • Pros: The remote server would only be contacted once during a login session
  • Cons: If the group memberships change rapidly on the server side, SSSD might still only use the cached values
krb5_auth_timeout
In case the Kerberos ticket has a PAC blob attached to it (see above) and password authentication is used, the krb5_child processes the PAC blob to help establish the group memberships. This processing might take longer than the time the krb5_child process is allowed to run. Therefore, for environments where users are members of a large number of groups, the krb5_auth_timeout value should be increased to allow the groups to be processed. In future SSSD versions, we aim at making the processing much faster, so the default 6 seconds timeout would suffice. Contacting the PAC responder can also be avoided completely by disabling the krb5_validate option, however disabling that option has security implications as we can’t any longer verify the TGT has not been spoofed with a MITM attack.

  • Recommended value: krb5_auth_timeout = login_time_in_seconds
  • Pros: The PAC responder has enough time to process user’s group memberships
  • Cons: Detecting legitimate offline situations might take too long
Mount the cache to tmpfs
Please see the server side section. On the client side, there is an additional caveat – the risk the client would reboot and then lose connectivity to the server is higher. Please do not set this option on roaming clients where you rely on offline logins!

To sum up the client side login changes, we’re looking at these additions to the config file:

[pam]
pam_id_timeout = N

[domain/domname]
krb5_auth_timeout = N
# Disabling the validation is dangerous!!
# krb5_validate = false

Don’t forget to restart the SSSD instance for the new settings to take effect!

Get rid of manually calling kinit with SSSD’s help

For quite some time, my laptop doesn’t have a UNIX /etc/passwd user for the account I normally use at all. Instead, I’ve used SSSD exclusively, logging in with our Red Hat corporate account. That allows me to use some cool SSSD features, most notably the krb5_store_password_if_offline option.

When that option is enabled and you log in while SSSD is offline (typically when a laptop was just started and is not connected to corporate VPN yet) using your cached password, SSSD stores the password you use to the kernel keyring. Then, when SSSD figures out that the servers are reachable, it attempts to kinit using that password on your behalf.

Because SSSD can detect when you connected to the VPN by listening to libnl events or watching resolv.conf for changes, the routine of connecting to Kerberos simplifies rapidly to:

  • open laptop
  • login using Kerberos password
  • connect to VPN

That’s it. kinit happens on the background and Kerberos ticket is acquired without any user intervention. Now you can just start Firefox and start accessing HTML pages or start Thunderbird and read your mail. No more starting an extra terminal and typing kinit or starting a separate Gnome dialog.

However, until now, this setup required that you also use an account from a centralized store that contains the Kerberos principal as one of its attributes or at least has the same user name as the first part of the Kerberos principal (called primary, up to the ‘@’ sign). And that’s a deal breaker for many people who are used to a certain local user name for ages, irrespective of their company’s Kerberos principal.

SSSD 1.12.5 that was released recently includes a very nice new feature, that allows you to work around that and map a local UNIX user to a particular Kerberos principal. The setup includes a fairly simple sssd.conf file:

 [sssd]
 domains = example.com
 config_file_version = 2
 services = nss,pam

 [domain/example.com]
 id_provider = proxy
 proxy_lib_name = files

 auth_provider = krb5
 krb5_server = kdc.example.com
 krb5_realm = EXAMPLE.COM
 krb5_map_user = jakub:jhrozek
 krb5_store_password_if_offline = True

 cache_credentials = True

All the interesting configuration is in the [domain] section. The identities are read from good old UNIX files by proxying nss_files through SSSD’s proxy provider. Authentication is performed using the krb5 provider, with a KDC server set to kdc.example.com and Kerberos realm EXAMPLE.COM

The krb5_map_user parameter represents new functionality. The parameter takes a form of username:primary and in this particular case maps a UNIX user jakub to a Kerberos principal which would in this case be jhrozek@EXAMPLE.COM.

On Fedora-22, care must be taken that the pam_sss.so module is called in the PAM stack even if pam_unix.so fails as the default behaviour for pam_unix.so is default=die – end the PAM conversation in case the user logging in exists in /etc/passwd and the password is incorrect. Which is our case, we’re logging in with UNIX user, we just want the authentication to be done by pam_sss.so.

Thanks to Sumit Bose who helped me figure out the PAM config, I used the following auth stack successfully:

auth     pam_env.so
auth     [default=2 success=ok] pam_localuser.so
auth     sufficient pam_unix.so nullok try_first_pass
auth     [success=done ignore=ignore default=die] pam_sss.so use_first_pass
auth     requisite pam_succeed_if.so uid >= 1000 quiet_success
auth     sufficient pam_sss.so forward_pass
auth     required pam_deny.so

I also recorded a video shows the feature in action. The first login is with the UNIX password, so the ticket is not automatically acquired and the user must run kinit manually. The next login happens with the Kerberos password while the machine is connected to network, so the ticket is acquired on login. Finally, the last login simulates the case where the machine starts disconnected from the network, then connects. In this case, SSSD acquires the ticket on the background when it detects the networking parameters have changed and the KDC is available.

Nice, right? You can also get the video in a nicer resolution.

The Kerberos authentication provider has more features that can beat using kinit from the command line – automatized ticket renewal might be one of them. Please refer to the sssd-krb5 manual page or share your tips in the comments section!

Authenticating a docker container against host’s UNIX accounts

Recently, with the advent of Docker and similar technologies, there’s been an effort to containerize different kinds of setups that previously were running on a single machine or a set of tightly coupled machines. In case the components communicate over a network connection, the containerization might not be that hard, but what about cases where the components would talk over a strictly local interface, such as a UNIX socket?

A colleague of mine asked me to help setup up a container to authenticate against the host’s UNIX accounts using the PAM API the other day. It turned out to be doable, but maybe not obvious to anyone not familiar with some of the more esoteric SSSD options, so I’d like to write down the instructions in a blog post.

In our setup, there was a pam service, sss_test, that previously would run on the host and authenticate against accounts either locally stored on the hosts in /etc/passwd and /etc/shadow. The same setup could in principle be used to authenticate against a remote database using SSSD, just the host SSSD settings would be different.

So how does this work with containers? One possibility, especially with a remote database such as IPA or AD would be to run an SSSD instance in every container and authenticate to the remote store. But that doesn’t help us with cases where we’d like to authenticate against the local database stored on the host, since it’s not exposed outside the host. Moreover, putting SSSD into each container might also mean we’d need to put a keytab in each container, then each container would open its separate connection to the remote server..it just gets tedious. But let’s focus on the local accounts..

The trick we’ll use is bind-mounting. It’s possible to mount part of host’s filesystem into the container’s filesystem and we’ll leverage this to bind-mount the UNIX sockets SSSD communicates over into the container. This will allow the SSSD client side libraries to authenticate against the SSSD running on the host. Then, we’ll set up SSSD on the host to authenticate aganst the hosts’s UNIX files with the proxy back end.

This is the traditional schema:
application -> libpam -> pam_authenticate -> pam_unix.so -> /etc/passwd

If we add SSSD to the mix, it becomes:
application -> libpam -> pam_authenticate -> pam_sss.so -> SSSD -> pam_unix.so -> /etc/passwd

Let’s configure the host and the container, step by step.

  1. Host config
    1. Install packages
    2. yum -y install sssd-common sssd-proxy

    3. create a PAM service for the container.
    4. The name doesn’t matter, it just needs to be referenced from sssd.conf later.
      # cat /etc/pam.d/sss_proxy
      auth required pam_unix.so
      account required pam_unix.so
      password required pam_unix.so
      session required pam_unix.so

    5. create SSSD config file, /etc/sssd/sssd.conf
    6. Please note that the permissions must be 0600 and the file must be owned by root.root.
      # cat /etc/sssd/sssd.conf
      [sssd]
      services = nss, pam
      config_file_version = 2
      domains = proxy
      [nss]
      [pam]
      [domain/proxy]
      id_provider = proxy
      # The proxy provider will look into /etc/passwd for user info
      proxy_lib_name = files
      # The proxy provider will authenticate against /etc/pam.d/sss_proxy
      proxy_pam_target = sss_proxy

    7. start sssd
    8. # systemctl start sssd

    9. verify a user can be retrieved with sssd
    10. $ getent passwd -s sss localuser

  2. Container setup
  3. It’s important to bind-mount the /var/lib/sss/pipes directory from the host to the container since the SSSD UNIX sockets are located there.

    # docker run -i -t -v=/var/lib/sss/pipes/:/var/lib/sss/pipes/:rw --name sssd-cli fedora:latest /bin/sh

  4. Container config
  5. The container runs the PAM-aware application that authenticates against the host. I used a simple program in C that comes from the SSSD source. It pretty much just runs pam_authenticate() against a service called sss_test. Your application might need a different service, but then you just need to adjust the filename in /etc/pam.d/ in the container. All the steps below should be executed on the container itself.

    1. install only the sss client libraries
    2. # yum -y install sssd-client

    3. make sure sss is configured for passwd and group databases in /etc/nsswitch.conf
    4. # grep sss /etc/nsswitch.conf

    5. configure the PAM service that the application uses to call into SSSD

    6. # cat /etc/pam.d/sss_test
      auth required pam_sss.so
      account required pam_sss.so
      password required pam_sss.so
      session required pam_sss.so

    7. authenticate using your PAM application.
    8. In my case that was
      # ./pam_test_client auth localtest
      action: auth
      user: localtest
      testing pam_authenticate
      Password:
      pam_authenticate: Success

  6. Profit!

It would be possible to authenticate against any database, just by changing what the SSSD on the host authenticates against. There’s several gotchas, though, especially should you require that only certain containers are allowed to retrieve users from certain domains. Multitenancy doesn’t really work well, because we don’t have a good mechanism to retrieve the identity of the container.

Anatomy of SSSD user lookup

This blog post describes how a user lookup request is handled in SSSD. It should help you understand how the SSSD architecture looks like, how the data flows in SSSD and as a result help identify which part might not be functioning correctly on your system. It is aimed mostly at users and administrators – for developers, we have a separate document about SSSD internals on the SSSD wiki written by Yassir Elley. This document re-uses some of the info from the internals one.

We’ll look at the most common operation, looking up user info on a remote server. I won’t go into server-specific details, so most of the info should be equally true for LDAP, Active Directory or FreeIPA servers. There’s also more functionality in SSSD than looking up users, such as sudo or autofs integration, but they are out of scope of this post as well.

Before going into SSSD details, let’s do a really quick intro into what happens on the system in general when you request a user from a remote server. Let’s say the admin configured SSSD and tests the configuration by requesting the admin user:

$ getent passwd admin

When user information is requested about a user (with getent, id or similar), typically one of the functions of the Name Service Switch, such as getpwnam() or initgroups() in glibc is called. There’s lots of information about the Name Service Switch in the libc manual, but for our purposes, it’s enough to know that libc opens and reads the config file /etc/nsswitch.conf to find out which modules should be contacted in which order. The module that all of us have on our Linux machines is files which can read user info from /etc/passwd and user info from /etc/groups. There also exists an ldap module that would read the info directly from an LDAP server and of course an sss module that talks to SSSD. So how does that work?

The first thing to keep in mind is that, unlike nss_ldap or pam_ldap, the SSSD is not just a module that is loaded in the context of the application, but rather a deamon that the modules communicate with. Almost no logic is implemented in the modules, all the functionality happens in the deamon. A user-visible effect during debugging is that using strace is not too helpful as it would only show if the request made it to the SSSD. For debugging the rest, the SSSD debug logs should be used.

Earlier I said that SSSD is a deamon. That’s really not too precise, SSSD is actually a set of deamons that communicate with one another. There are three kinds of SSSD processes. One is the sssd process itself. Its purpose is to read the config file after startup and spawn the other processes according to the config file. Then there are responder or front end processes that listen to queries from the applications, like the query that would come from the getent command. If the responder process needs to contact the remote service for data, it talks to the last SSSD process type, which is the data provider or back end process. This architecture allows for a pluggable setup where there are different back end processes talking to different remote servers, while all these remote servers can be accessed from a range of applications or subsystems by the same lookup code in the responders.

Each process is represented by a section in the sssd.conf config file. The main sssd process is represented by the [sssd] section. The front end processes are defined on the services line in the [sssd] section and each can be configured in a section named after the service. And finally, the back end processes are those configured in the [domain] sections. Each process also logs into its own logfile.

Let’s continue with the getent passwd admin example. To illustrate the flow, there is a diagram that the text follows. The full arrows represent local IO operation (like opening a file), the empty arrows represent local IPC over UNIX sockets and the dotted arrow represents a network IO.

sssd-lookup

The user issued the getent command which calls libc’s getpwnam (diagram step 1), then the libc opens the nss_sss module as per nsswitch.conf and passes in the request. First, the nss_sss memory-mapped cache is consulted, that’s step 2 on the diagram. If the data is present in the cache, it is just returned without even contacting the SSSD, which is extremely fast. Otherwise, the request is passed to the SSSD’s responder process (step 3), in particular sssd_nss. The request first looks into the SSSD on-disk cache (step 4). If the data is present in the cache and valid, the nss responder reads the data from the cache and returns them to the application.

If the data is not present in the cache at all or if it’s expired, the sssd_nss request queries the appropriate back end process (step 5) and waits for reply. The back end process connects to the remote server, runs the search (step 6) and stores the resulting data into the cache (step 7). When the search request is finished, the provider process signals back to the responder process that the cache is updated (step 8). At that point, the front-end responder process checks the cache again. If there’s any data in the cache after the back end has updated it, the data is returned to the application – even in cases when the back end failed to update the cache for some reason, it’s better to return stale data than none. Of course, if no data is found in the cache after the back end has finished, an empty result is returned back. This final cache check is represented by step 9 in the diagram.

When I said the back end “runs a search” against the server, I really simplified the matter a lot. The search can involve many different steps, such as resolving the server to connect to, authenticating to the server, performing the search itself and storing the resulting data into the database. Some of the steps might even require a helper process, for instance authenticating against a remote server using a keytab is done in a heper process called ldap_child that logs into its own logfile called /var/log/sssd/ldap_child.log.

Given most steps happen in the back end itself, then most often, the problem or misconfiguration lies in the back end part. But it is still very important to know the overall architecture and be able to identify if and how the request made it to the back end at all. In the next part, we’ll apply this new information to perform a small case study and we will repair a buggy sssd setup.

Troubleshooting a failing SSSD user lookup.

With the SSSD architecture in mind, we can try a case study. Consider we have an IPA client, but no users, not even the admin show up:

$ getent passwd admin
$ echo $?
2

The admin user was not found! Given our knowledge of the architecture, let’s first see if the system is configured to query sssd for user information at all:

$ grep passwd /etc/nsswitch.conf
passwd: files sss

It is. Then the request was passed on to the nss responder process, since the only other possibility is a successful return from the memory cache. We need to raise the debug_level in the [nss] section like this:

[nss]
debug_level = 7

and restart sssd:

# systemctl restart sssd

Then we’ll request the admin user again and inspect the NSS logs:

[sssd[nss]] [accept_fd_handler] (0x0400): Client connected!
[sssd[nss]] [sss_cmd_get_version] (0x0200): Received client version [1].
[sssd[nss]] [sss_cmd_get_version] (0x0200): Offered version [1].
[sssd[nss]] [nss_cmd_getbynam] (0x0400): Running command [17] with input [admin].
[sssd[nss]] [sss_parse_name_for_domains] (0x0200): name 'admin' matched without domain, user is admin
[sssd[nss]] [nss_cmd_getbynam] (0x0100): Requesting info for [admin] from []
[sssd[nss]] [nss_cmd_getpwnam_search] (0x0100): Requesting info for [admin@ipa.example.com]
[sssd[nss]] [sss_dp_issue_request] (0x0400): Issuing request for [0x4266f9:1:admin@ipa.example.com]
[sssd[nss]] [sss_dp_get_account_msg] (0x0400): Creating request for [ipa.example.com][4097][1][name=admin]
[sssd[nss]] [sss_dp_internal_get_send] (0x0400): Entering request [0x4266f9:1:admin@ipa.example.com]
[sssd[nss]] [sss_dp_get_reply] (0x1000): Got reply from Data Provider - DP error code: 1 errno: 11 error message: Fast reply - offline
[sssd[nss]] [nss_cmd_getby_dp_callback] (0x0040): Unable to get information from Data Provider
Error: 1, 11, Fast reply - offline
Will try to return what we have in cache

Well, apparently the request for the admin user was received and passed on to the back end process, but the back end replied that it switched to offline mode..that means we need to also enable debugging in the domain part and continue investigation there. We need to add debug_level to the [domain] section and restart sssd again. Then run the getent command and inspect the file called /var/log/sssd/sssd_ipa.example.com starting with the time that corresponds to the NSS responder sending the data (as indicated by sss_dp_issue_request in the nss log). In the domain log we see:

[sssd[be[ipa.example.com]]] [fo_resolve_service_done] (0x0020): Failed to resolve server 'master.ipa.example.com:389': Domain name not found
[sssd[be[ipa.example.com]]] [set_server_common_status] (0x0100): Marking server 'master.ipa.example.com:389' as 'not working'
[sssd[be[ipa.example.com]]] [be_resolve_server_process] (0x0080): Couldn't resolve server (master.ipa.example.com:389), resolver returned (11)
[sssd[be[ipa.example.com]]] [be_resolve_server_process] (0x1000): Trying with the next one!
[sssd[be[ipa.example.com]]] [fo_resolve_service_send] (0x0100): Trying to resolve service 'IPA'
[sssd[be[ipa.example.com]]] [get_server_status] (0x1000): Status of server 'master.ipa.example.com:389' is 'not working'
[sssd[be[ipa.example.com]]] [get_port_status] (0x1000): Port status of port 0 for server '(no name)' is 'not working'
[sssd[be[ipa.example.com]]] [get_server_status] (0x1000): Status of server 'master.ipa.example.com:389' is 'not working'
[sssd[be[ipa.example.com]]] [fo_resolve_service_send] (0x0020): No available servers for service 'IPA'
[sssd[be[ipa.example.com]]] [be_resolve_server_done] (0x1000): Server resolution failed: 5
[sssd[be[ipa.example.com]]] [sdap_id_op_connect_done] (0x0020): Failed to connect, going offline (5 [Input/output error])
[sssd[be[ipa.example.com]]] [be_ptask_create] (0x0400): Periodic task [Check if online (periodic)] was created
[sssd[be[ipa.example.com]]] [be_ptask_schedule] (0x0400): Task [Check if online (periodic)]: scheduling task 70 seconds from now [1426087775]
[sssd[be[ipa.example.com]]] [be_run_offline_cb] (0x0080): Going offline. Running callbacks.

OK, that gets us somewhere. Indeed, our /etc/resolv.conf file was ponting to a bad nameserver. And indeed, after fixing the resolver settings and restarting SSSD, everything seems to be working:

$ getent passwd admin
admin:*:1546600000:1546600000:Administrator:/home/admin:/bin/bash

Awesome, we were able to repair a broken SSSD setup!