Prevent CVE exploits in your Kubernetes cluster with seccomp and SELinux profiles

The other day I was reading a cool post by Snyk in which the author exploits a known CVE in ImageMagick to get a reverse shell and I started wondering what would it take to write policies that would prevent this attack.

This would also be a nice demonstration a project I was contributing to recently
called the Security Profiles Operator (SPO) which enables you to easily distribute and use security profiles in a Kubernetes cluster. While it was possible to use profiles like seccomp or SELinux in Kubernetes clusters for some time, the distribution and usage part is something new that SPO enables which would be a manual and tedious process otherwise.

Note that the security policies are always a security benefit and should be used for any pods, but they are even more important for privileged pods. Your cluster should ideally not run any privileged pods and if it does, they should be constrained. This is unfortunately not the case in many Kubernetes distributions.

The original blog by Snyk post comes with a Github demo repository which I forked and put all the manifests in. You can follow the original repo to play around with the exploits locally using docker or podman and you can use the fork to follow this blog post along.

You’ll need:

• a kubernetes cluster, ideally with admin access to try all the scenarios including the one that deploys SPO. I used a vanilla 1.20 cluster running Fedora 33 as cluster nodes running on libvirt-powered VMs. If your cluster is running another Linux distribution, you might not be able to test the SELinux profiles, but the seccomp part should be testable from everywhere.
• another machine reachable from the cluster to test the reverse shell. For the demo, I used another VM just to really test that the exploit can reach outside the cluster, but you can even use another cluster node.
• a local clone of my goof-container-breaking-in demo repo

Try out the exploit in Kubernetes

Let’s first verify we can exploit an image running in a cluster to make sure we have something tangible to defend against. Because the exploit starts a reverse shell, we’re going to prepare the listening endpoint that will just sit there and wait for the vulnerable pod to be attacked, which would initiate the connection from the container and start the reverse shell. Clone the demo repo and make sure the payload includes the IP address of this listening machine.

As said before, I was testing with a local VM, running Fedora. Log in to the machine and make sure netcat is instaled:
$ sudo dnf -y install nc

Then, start the listener on a port that matches the port set in the exploit payload:
$ nc -l 3131
Type some command (ls is good enough), the command will just hang at the moment until you run the exploit in the vulnerable pod.

Now it’s time to run the vulnerable payload in the cluster. From the checkout of the repo, run:
$ kubectl create -f k8s/goof.yaml
to create the pod. For the sake of simplicity, we’re deploying the app in the same namespace as the operator.

The pod is not very complex:

---
apiVersion: v1
kind: Namespace
metadata:
  labels:
    app: security-profiles-operator
  name: security-profiles-operator
---
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    app: goof
  name: goof
  namespace: security-profiles-operator
spec:
  containers:
  - image: quay.io/jhrozek/goof-container:latest
    name: goof
    ports:
    - containerPort: 3112

---

I didn’t bother setting up services for the demo, but instead just exposed the container of the pod with port-forward:
$ kubectl port-forward pod/goof -nsecurity-profiles-operator 8080:3112

Navigate to http://localhost:8080/public using your browser and you should see the image resizing app. Click Browse, select the payload (exploits/nc-reverse-shell.jpg) and click Resize to submit it. The exploit now does its thing and runs our code, which fetches nc and connects to our listener VM. At this point, or a couple seconds later, we should see the “ls” command returning output.

The reverse shell might seem as a toy, but it’s actually useful to perform lateral movement in the cluster. Depending on how the pod and the cluster are (mis)configured, there are interesting things you can achieve. Try some other commands in the shell to get a feel of what you can do (hint: a service account’s token is a good start).

We’ve established that the image can be exploited. Delete the pod now:
$ kubectl delete pods goof -nsecurity-profiles-operator

Preventing the attack

So how do you prevent mischiefs like this one? There are several things you can do and you should actually employ a combination of them, for defense in depth:

• detect vulnerable images and prevent them from being deployed. The original blog post from Snyk talks about this. Of course this presumes that you know that the image is vulnerable. In case someone decides to exploit an unknown vulnerability or a configuration issue, this won’t help.
• detect and prevent the reverse shells. Sysdig had a nice blog post about that some time ago although I haven’t tried it myself.
• restrict the app to only do what it’s supposed to do. Even if the app is exploited, the potential to do damage is limited. This can be done using several techniques, including limiting the container capabilities or using security profiles like seccomp or SELinux.
• I guess many more…

In the rest of the post, we’ll concentrate on restricting the app to only do that it’s supposed to do using both seccomp and SELinux and as importantly, making sure those policies can be used as first-class cluster objects so that we can list the policies installed in the cluster, view their status, see which workloads use which policy and so on.

The seccomp and SELinux policies are really complementary. While seccomp restricts the syscall surface of an application, SELinux allows you to restrict what kind of objects can the app interact with based on labels on both processes and files. For example, if your app needs to write somewhere, you’ll permit the usage of the write syscall, but using SELinux, you’ll be able to express that the app can only write to files labeled with a certain label and no others. Same applies to e.g. devices that might need to be mounted to privileged containers, using SELinux you can restrict the app to only communicate with the devices labeled wth the appropriate label.

Deploy SPO

Follow the official documentation of SPO to install it don’t forget to install cert-manager first.

The central part of the SPO configuration is the “spod” CRD, which will eventually show you that SPO is up and running:

$ kubectl get spod -nsecurity-profiles-operator
NAME STATE
spod RUNNING

Prevent the attack with a seccomp profile

Let’s first start with the seccomp profile as more people are probably familiar with seccomp and virtually any Kubernetes cluster allows you to deploy seccomp profiles.

Of course, we need to generate the policy first. The ideal way is the generate the profile in the environment that is very close to where your app is actually running – and the SPO offers a feature that does just that, through the ProfileRecording CRD. This feature requires CRI-O 1.21.0 or newer and in addition your worker nodes must have the oci-seccomp-bpf-hook package installed. Because this is not the case in all available clusters, we’ll show both the ProfileRecording and how to generate the profiles locally as well,

Finally, if you’re not interested in generating the policies, feel free to use the one in the repo (k8s/seccomp-policy.yaml) and just skip to the section where we use the policy.

Generate the seccomp profile in-cluster

To generate the policy, create a ProfileRecording CR that matches the goof app we’re running:

apiVersion: security-profiles-operator.x-k8s.io/v1alpha1
kind: ProfileRecording
metadata:
   # The name of the Recording is the same as the resulting seccomProfile CR
   name: goof
   namespace: security-profiles-operator
spec:
   kind: SeccompProfile
   podSelector:
      matchLabels:
         app: goof

Next, create the app again:
$ kubectl create -f k8s/goof.yaml
and verify that the pod is annotated with "io.containers.trace-syscall/goof". This annotation tells the CRI-O runtime that it should record the syscalls the container does.

Now forward the ports again:
$ kubectl port-forward pod/goof -nsecurity-profiles-operator 8080:3112
navigate to the app as earlier and convert an ordinary image, not the exploit. Once you’re done, remove the goof pod:
$ kubectl delete pods goof -nsecurity-profiles-operator

In a bit, a policy should be automagically defined in your cluster and you’ll be able to see it with:

$ kubectl get sp goof -nsecurity-profiles-operator
NAME STATUS AGE
goof Installed 10m

including the details:
$ kubectl get sp goof -nsecurity-profiles-operator -oyaml

Now you can remove the profilerecording object:
$ kubectl delete profilerecordings goof -nsecurity-profiles-operator

Generate the seccomp policy locally

To make the article usable also for people who can’t run the new version of CRI-O or install the oci-seccomp-bpf-hook in their clusters, let’s also generate the policy locally. I generated the profile locally with podman using an OCI seccomp BPF hook as described here.

Generating the profile locally is of course not ideal – you’ll ideally want to generate the profile in the same or very similar environment to the one you’ll be running the app in as the profile needs to include both the syscalls your app is doing as well as what the container runtime is doing. I had to adjust the podman-generated policy a bit to include the setuid() call in my case.

So to generate the profile, let’s start the app container locally, but tell podman to record the syscalls using the seccomp bpf hook. That can be done using the io.containers.trace-syscall annotation:
$ sudo podman run --annotation io.containers.trace-syscall="of:/tmp/goof-seccomp-normal.json" --rm -p 3112:3112 --name goof quay.io/jhrozek/goof-container

Navigate to http://localhost:3113/public and convert a normal image to get the baseline seccomp policy. Let’s do a similar thing again, just recording into a different file and this time let’s feed the app the exploit (exploits/nc-reverse-shell.jpg) again:
$ sudo podman run --annotation io.containers.trace-syscall="of:/tmp/goof-seccomp-rsh.json" --rm -p 3112:3112 --name goof quay.io/jhrozek/goof-container

Note that we run the containers using "sudo podman". This is needed to be allowed to record the policies.

Now if we compare the two policies:
$ diff <(jq -c '.syscalls[].names[]' < /tmp/goof-seccomp-normal.json) <(jq -c '.syscalls[].names[]' < /tmp/goof-seccomp-rsh.json)

we see that one generated using the exploit contains several more syscalls:

2a3 > "alarm"
10a12
> "connect"
18a21,23
> "fchmodat"
> "fcntl"
> "fork"
25a31,32
> "getpeername"
> "getpgrp"
28a36
> "getrusage"
29a38,39
> "getsockopt"
> "gettid"
40a51
> "newfstatat"
41a53
> "pipe"
45a58
> "prlimit64"
47a61
> "recvfrom"
62a77,78
> "statfs"
> "symlink"
66a83,85
> "unlink"
> "unlinkat"
> "utimensat"
68a88

Presumably connect is what is used to establish the connection outside and fchmodat was used to make the netcat executable after the exploit downloaded netcat. So this is already looking promising!

Now you’d convert the policy to the SeccompProfile object definition. To save you from the work, you can just head to the goof github repo and use the seccomp-profile-binding.yaml file:
$ kubectl create -f k8s/seccomp-profile-binding.yaml
Verify that the profile was created, also peek at the securityprofilenodestatuses objects to see the status of the policy per nodes:
$ kubectl get securityprofilenodestatuses,seccompprofiles -nsecurity-profiles-operator
All the policy statuses should say “Installed”.

Test that the seccomp profile prevents the attack

Now it’s time to put our policy to use! We could either reference our policy directly in the pod manifest, but that’s not very scalable. What we probably want instead is to make sure all workloads with a certain image have a certain policy applied. This is when the ProfileBinding CRD comes to play. The binding
object binds an image to a profile reference:

apiVersion: security-profiles-operator.x-k8s.io/v1alpha1
kind: ProfileBinding
metadata:
   name: goof-binding
   namespace: security-profiles-operator
spec:
   profileRef:
      kind: SeccompProfile
      name: goof
   image: quay.io/jhrozek/goof-container:latest

Now, let’s deploy the vulnerable pod again and forward the ports:
$ kubectl create -f k8s/goof.yaml

Verify that the policy was applied – check the pod manifest in the
cluster:
$ kubectl get pods goof -nsecurity-profiles-operator -ojsonpath='{.metadata.annotations}' | jq
and you should see the pod annotated with the seccomp policy on-disk path:
"container.seccomp.security.alpha.kubernetes.io/goof": "localhost/operator/security-profiles-operator/goof.json"
Similarly, the SeccompProfile object keeps tracks of the workloads that use the profile so that you can’t remove a profile in use by accident:
$ kubectl sp goof -nsecurity-profiles-operator -ojsonpath='{.status.activeWorkloads}'
["security-profiles-operator/goof"]

Finally, let’s run the exploit again:
$ kubectl port-forward pod/goof 8080:3112
Navigate to localhost:8080, click browse, select the payload (exploits/nc-reverse-shell.jpg) and click Resize to submit it.

…and nothing! The attack was prevented by the seccomp profile and your company won’t be on the front page of any IT newspaper tomorrow for all the wrong reasons.

Let’s clean up after our experiments so that we can move on to the next one with SELinux.

$ kubectl delete pods goof -nsecurity-profiles-operator
$ kubectl delete profilebinding --all -nsecurity-profiles-operator
$ kubectl delete sp -nsecurity-profiles-operator --all

Prevent the attack with SELinux profile

In some distributions, like OpenShift, SELinux is used by default to make sure that pods are really isolated and contained from each other. There are some really good blog posts by Dan Walsh or my team mate Ozz.

This post has already gotten way longer than I anticipated, so I’ll just show how to create and use a policy without going too deep into details about how SELinux works in Kubernetes. If you peek at Ozz’s post, you might notice that the workflow includes generating a policy locally and then distributing the policy file to each of the cluster nodes. Ouch, this doesn’t sound very scalable and easy. Do you now need to ssh into the nodes and manage the policies yourself?

To address this concern, Ozz and I wrote a small project called selinuxd which is now used by SPO and handles installation and removal of the policies for you. This means that with SPO’s and selinuxd’s help, installing an SELinux policy is as easy as kubectl create. I’m tactfully omitting the part where you need to create your own policy: at the moment, we still don’t have a good answer except generate the policy locally, but we have ideas for future 🙂

First, let’s enable SELinux in SPO:
$ kubectl patch spod spod -p '{"spec":{"enableSelinux": true}}' --type=merge
This triggers rollout of the DaemonSet called “spod” which might take up to a couple of minutes (copying and using the selinux policies is slow..).

Generate the SELinux policy locally

As said earlier, there is no way to generate the policy in cluster at the moment,
although the feature is planned. So we’ll resort to generating the policy locally, with podman and udica. Run the container:
$ podman run --rm -p 3112:3112 --name goof quay.io/jhrozek/goof-container
and in another terminal, inspect the container and point udica at the JSON file that podman inspect generates:
$ podman inspect a0fa31b6a538 > /tmp/goof.json
$ sudo udica -j /tmp/goof.json goof-selinux
This will generate a .cil file that we will wrap in a SelinuxProfile CR. You can view the result in the repo (k8s/selinuxpolicy-goof.yaml).

Test that the policy prevents the attack

You can use the prepared policy in the repo (k8s/selinuxpolicy-goof.yaml). Create it as a usual k8s object:
$ kubectl create -f k8s/selinuxpolicy-goof.yaml

Installing the policy takes a bit of time during which selinuxd on the hosts does its job, so let’s wait until the policy switches to installed:

$ kubectl get selinuxprofile
NAME USAGE STATE
goof goof_security-profiles-operator.process Pending
$ kubectl wait --for=condition=ready selinuxprofile goof
selinuxprofile.security-profiles-operator.x-k8s.io/goof condition met
$ kubectl get selinuxprofile
NAME USAGE STATE
goof goof_security-profiles-operator.process Installed

Now that the policy is installed, we can take it into use. The policy name (called usage) is different from the profile object name, which can be confusing, but either way, the policy name can be found at .status.usage:
$ kubectl get selinuxprofile goof -ojsonpath='{.status.usage}'
goof_security-profiles-operator.process
We can put this string into the pod’s .spec.containers.securityContext.seLinuxOptions:

spec:
  containers:
  - image: quay.io/jhrozek/goof-container:latest
    name: goof
    ports:
    - containerPort: 3112
    securityContext:
      seLinuxOptions:
        type: goof_security-profiles-operator.process

The full pod manifest using the policy can be found at k8s/goof-selinux.yaml in the repo. So let’s start the pod:
$ kubectl create -f k8s/goof-selinux.yaml

Forward the ports, set up the listener and run the exploit again:
$ kubectl port-forward pod/goof -nsecurity-profiles-operator 8080:3112

Navigate to localhost:8080, click browse, select the payload (exploits/nc-reverse-shell.jpg) and click Resize to submit it.

…and nothing! If we go to the node and inspect its audit.log, we’ll see that the pod tried to do something that the policy didn’t allow it to:

type=AVC msg=audit(1621934085.650:196): avc: denied { name_connect } for pid=58536 comm="nc" dest=3131 scontext=system_u:system_r:goof_security-profiles-operator.process:s0:c285,c1001 tcontext=system_u:object_r:unreserved_port_t:s0 tcl>

Aha! This is the nc binary trying to connect to a port, which the policy does not allow. SELinux saved the day.

Conclusion

The Security Profiles Operator doesn’t bring any new hardening techniques on its own. Both SELinux and seccomp were supported by container runtimes and Kubernetes for some time. What SPO does bring is the ability to treat the security profiles as first-class objects including status, record workloads
to create the security profiles without having to analyze the program line by line and allows the administrator to bind the workloads to the profiles.

The SPO upstream had created documents with user stories and personas that hopefully illustrates better who is SPO aimed for.

As can be seen from the github issues, the SPO development is far from over. If you’re interested in the world of usable container security, please come over to the #security-profiles-operator channel on the
Kubernetes slack, try the operator and give feedback.

Advertisement

IPA sudo rules for local users

Central identity management solutions like IPA offer a very nice capability – in addition to storing users and groups on the server, also certain kinds of policies can be defined in one place and used by all the machines in the domain.

Sudo rules are one of the most common policies that administrators define for their domain in place of distributing or worse, hand-editing /etc/sudoers. But what if the administrator would take advantage of the central management of sudo rules, but apply these sudo rules to a user who is not part of the domain, but only exists in the traditional UNIX files, like /etc/passwd?

This blog post illustrates several strategies for achieving that.

Option 1: Move the user to IPA

This might be the easiest solution from the point of view of policy management that allows you to really keep everything in one place. But it might not be always possible to move the user. A common issue arises with UIDs and file ownership. If the UIDs are somewhat consistent on majority of hosts, perhaps it would be possible to use the ID views feature of IPA to present the UID a particular system expects. If it’s not feasible to move the user to the IPA domain, there are other options to consider.

Option 2: Use the files provider of SSSD

One of the core design principles of SSSD is that a user and its policies must both be served from SSSD and even from the same SSSD domain. This prevents unexpected results in case of overlapping user names between SSSD domains or between the data SSSD is handling and the rest of the system.

But at the same time, local users are by definition anchored in /etc/passwd. To be able to serve the local users (and more, for example better performance), SSSD ships a files data provider since its 1.15 release. At the moment (as of 1.16.1), the files release mostly mirrors the contents of /etc/passwd and /etc/groups and presents them via the SSSD interfaces. And that’s precisely what is needed to be able to serve the sudo rules as well.

Let’s consider an example. A client is enrolled in an IPA domain and there is a user called lcluser on the client. We want to let the user run sudo less to be able to display the contents of any file on the client.

Let’s start with adding the sudo rule on the IPA server. First, we add the less command itself:
$ ipa sudocmd-add --desc='For reading log files' /usr/bin/less

Then we add the rule and link the user:
$ ipa sudorule-add readfiles
$ ipa sudorule-add-user --users=lcluser


Note that we lcluser doesn’t exist in IPA, yet we were able to add the user with the regular sudorule-add-user command. When we inspect the sudo rule, we can see that IPA detected that the user is external to its directory and created a special attribute externalUser where the username was added to, unlike the IPA user admin, which is linked to its LDAP object.
$ ipa sudorule-show --all --raw readfiles
dn: ipaUniqueID=31051b50-30d5-11e8-b1a7-5254004e66c1,cn=sudorules,cn=sudo,dc=ipa,dc=test
cn: readfiles
ipaenabledflag: TRUE
hostcategory: all
externaluser: lcluser
memberuser: uid=admin,cn=users,cn=accounts,dc=ipa,dc=test
ipaUniqueID: 31051b50-30d5-11e8-b1a7-5254004e66c1
memberallowcmd: ipaUniqueID=179de976-30d5-11e8-8e98-5254004e66c1,cn=sudocmds,cn=sudo,dc=ipa,dc=test
objectClass: ipaassociation
objectClass: ipasudorule

Finally, let’s link the sudo rule with the host we want to allow the sudo rule at:
ipa sudorule-add-host readfiles --hosts ipa.client.test

Now, we can configure the IPA client. We being by defining a new SSSD domain that will mirror the
local users, but we also add a ‘sudo_provider’:
[domain/files]
id_provider = files
sudo_provider = ldap
ldap_uri = ldap://unidirect.ipa.test

The reason it is preferable to use the IPA sudo provider over the LDAP provider is that the LDAP provider relies on schema conversion on the IPA server from IPA’s own schema into the schema normally used when storing sudo rules in LDAP as described in man sudoers.ldap. This conversion is done by the slapi-nis plugin and can be somewhat costly in large environments.

Finally, we enable the domain in the [sssd] section:
[sssd]
services = nss, pam, ssh, sudo, ifp
domains = files, ipa.test

Now, we should be able to run sudo as “lcluser”:
$ su - lcluser
Password:
[lcluser@client ~]$ sudo -l
[sudo] password for lcluser:
Matching Defaults entries for lcluser on client:
!visiblepw, env_reset, env_keep="COLORS DISPLAY HOSTNAME HISTSIZE KDEDIR LS_COLORS", env_keep+="MAIL PS1 PS2 QTDIR USERNAME LANG LC_ADDRESS LC_CTYPE", env_keep+="LC_COLLATE
LC_IDENTIFICATION LC_MEASUREMENT LC_MESSAGES", env_keep+="LC_MONETARY LC_NAME LC_NUMERIC LC_PAPER LC_TELEPHONE", env_keep+="LC_TIME LC_ALL LANGUAGE LINGUAS _XKB_CHARSET
XAUTHORITY", secure_path=/sbin\:/bin\:/usr/sbin\:/usr/bin

User lcluser may run the following commands on client:
(root) /usr/bin/less

Woo, it works!

Option 3: Use the proxy SSSD provider proxying to files

The files provider was introduced to SSSD in the 1.15 version. If you are running an older SSSD version, this provider might not be available. Still, you can use SSSD, just with somewhat more convoluted configuration using id_provider=proxy. All the server-side steps can be taken from the previous paragraph, just the client-side configuration will look like this:

[domain/proxytofiles]
id_provider = proxy
proxy_lib_name = files
proxy_pam_target = none

sudo_provider = ldap
ldap_uri = ldap://unidirect.ipa.test

Using the files provider is preferable, because it actively watches the changes to /etc/passwd and group using inotify, so the provider receives any change done to the files instantly. The proxy provider reaches out to the files as if it were a generic data source. Also, as you can see, the configuration is not as nice, but it’s a valid option for older releases.

Option 4: Use the LDAP connector built into sudo

If using SSSD is not an option at all, because perhaps you are configuring a non-Linux client, you can still point sudo to the IPA LDAP server manually. I’m not going to go into details, because the setup is nicely described in “man sudoers.ldap”.

Unlike all the above options, this doesn’t give you any caching, though.

Restrict the set of groups the user is a member of with SSSD

Written by Jakub Hrozek and Sumit Bose

One of the frequent requests we receive for a future enhancement for the SSSD project is one where admins would like to restrict the list of groups users on a host are a member of. We currently don’t offer this functionality and in this blog post I would like to explain why it’s not a completely trivial thing to implement correctly (and therefore why it’s not implemented already..) and offer a workaround that can be used for many environments as long as the administrator understands its limitations.

The administrator typically wants to restrict the list of groups the uses are a member of on the host for either (or both) of these reasons:

• to limit access to resources beyond what the users are normally privileged to access

• to provide a performance boost where sssd wouldn’t even attempt to resolve groups that are not listed in the configured group list

The vast majority of administrators are actually asking to improve performance, especially in High Performance Computing environments where users are part of a huge domain, but the client machine is only interested in a subset of the groups that give access to some resource. At the same time, avoiding the group resolution completely is significantly less trivial than the first case where SSSD would resolve all the groups and then only present a subset to the system.

Resolving the list of groups the user is a member of is a complex task, which can query different resources or interfaces depending on how the user logs in or what kind of command the administrator invokes. In the simplest case, where SSSD is connected to a generic LDAP server and the admin calls the “id” utility, SSSD would search the LDAP directory for groups the user is a member of. This scenario is actually possible to restrict already (and we’ll show how later in the post), but there are more ways to resolve a user’s group memberships. For example, in an Active Directory domain, the group membership might come from the PAC data blob attached to the Kerberos ticket acquired during authentication in the form of a list of SIDs which must be resolved into either a POSIX ID or a name before to be filtered later – and this SID-to-name/SID-to-ID resolution would negate any performance benefit in most cases.

Similarly, when an IPA client is resolving groups for an AD user from a trusted directory, it would ask one of the IPA masters for a list of groups the user belongs to. These two examples hopefully illustrate that there are many code paths to consider to make this feature work as expected in all possible scenarios. While implementing this feature is something we have on our roadmap, it will take time to get there.

With that said, SSSD already provides a way to only include certain groups as long as only the generic LDAP lookups are used to resolve group memberships. As an example, we’ll illustrate how to restrict the client to only search and return two groups. We’ll start with a setup where SSSD is connected to an Active Directory domain and returns the full list of group memberships, for example:

$ id sssduser@win.trust.test
uid=679801116(sssduser@win.trust.test) gid=679800513(domain users@win.trust.test) groups=679800513(domain users@win.trust.test),679801117(global_group@win.trust.test),679801105(sudogroup@win.trust.test),679801118(universal_group@win.trust.test)

The client uses a fairly default configuration where the domain section of sssd.conf looks like this:

[domain/win.trust.test]
id_provider = ad
access_provider = ad
ad_domain = win.trust.test
krb5_realm = WIN.TRUST.TEST
realmd_tags = manages-system joined-with-adcli

Let’s say that in our environment, we only care about the “sudogroup” so that the user can elevate her privileges with sudo and the “Domain Users” group to access shared resources.

The first step is to make the client look up the groups the user is a member of using plain LDAP lookups instead of looking up the AD-specific tokenGroups attribute. Normally, if all groups are to be returned, using the tokenGroups attribute provides a significant performance benefit, because the list of all groups is a member of can be returned with a single BASE-scoped search of the user entry. However, the tokenGroups attribute is a multi-valued list of SIDs the user is a member of and as said earlier, all the SIDs would have to be resolved into group names anyway. Therefore we disable the tokengroups support in the [domain] section of the sssd.conf config file by adding this parameter:

ldap_use_tokengroups = false

Then, we need to instruct SSSD to only look for the two groups we care about on the client. We include the names of the two groups as an extension of the LDAP group search base, which can optionally also include the scope and the filter to search with:

ldap_group_search_base = CN=Users,DC=win,DC=trust,DC=test?sub?(|(cn=domain users)(cn=sudogroup))

Please note that the group search base is not required to be set unless in the typical case, because SSSD infers its value from the domain name of the AD domain it is joined to.

Finally, we restart the SSSD service:

pam_hbac: A PAM module to enforce IPA access control rules

Written by Jakub Hrozek and Pavel Reichl

FreeIPA is an open source identity management software. One of the nice features is the ability to limit which users can log in to which servers using Host Based Access Control (HBAC). Previously, only SSSD was able to parse and evaluate the HBAC rules – however, some users run operating systems that either do not support SSSD at all, or SSSD can only be configured in a way that doesn’t include the HBAC engine.

To provide HBAC support for these platforms, we wrote a new PAM module called pam_hbac. The first release of pam_hbac just happened and in this blog post we would like to introduce the project and show how it works.

pam_hbac connects to an IPA server, downloads the HBAC access control rules, evaluates them and decides whether access should be allowed or denied. Even though pam_hbac is a project on its own, it uses the same code to evaluate the HBAC rules as SSSD does, so the resulting access control decision will be the same for the same rules provided the input is the same as well.

Using pam_hbac should come with a disclaimer – if your operating system supports SSSD and you can use its IPA id_provider, please use SSSD instead of pam_hbac. SSSD is maintained by a large team of developers, it is included in distributions with commercial support available and has several advantages over pam_hbac, including offline caching or Kerberos authentication using the host keytab by default.

Who should use pam_hbac

There are legitimate use-cases where using a standalone PAM module like pam_hbac is required, though. These include:

• if your IPA client runs an OS that doesn’t support SSSD at all like Solaris or its derivatives
• if your IPA client runs an OS that does support SSSD but not its IPA provider. At the moment, this is the case with Amazon’s Linux distribution and also FreeBSD.
• if your IPA client runs an OS that supports the IPA provider, but the IPA provider lacks the support for users from trusted Active Directory domains. This is the case for clients running RHEL-5 and leveraging the “compat” LDAP tree provided by the slapi-nis plugin of IPA.

At the moment, pam_hbac runs on Linux, FreeBSD and Solaris (tested on Oracle Solaris 11 and Omnios)

Obtaining and installing pam_hbac

Development of pam_hbac happens on github. There you can clone the git repository and compile pam_hbac yourself. Because the required dependencies or configure flags differ for each platform a bit, please refer to per-platform README files in the doc directory. As an example, there is a FreeBSD-specific README file that contains step-by-step instructions for building and installing pam_hbac on FreeBSD.

For RHEL-5 and RHEL-6, we have also provided a COPR repository with prebuilt binaries. On these platforms, you can just drop the .repo file into /etc/yum.repos.d/ and then yum install pam_hbac.

We would certainly welcome contributors who would like to provide prebuilt binaries for different platforms!

Examples

In the rest of the blog posts, we illustrate two examples of pam_hbac in action. One would be using pam_hbac on FreeBSD to grant access to a subset of IPA users and another one would be to use pam_hbac on CentOS-5 to restrict access to a single Active Directory group in a setup with trusts. For both examples, we use the same environment – the IPA domain is called IPA.TEST and is managed by an IPA server called unidirect.ipa.test. Our clients are called freebsd.ipa.test and centos5.ipa.test.

Example 1: using pam_hbac to provide access control for IPA users on FreeBSD

Even though the recent FreeBSD releases do ship SSSD, it is not built with the IPA provider by default (only through extra flags) and therefore HBAC enforcement might not be available easily. However, we can configure SSSD with the LDAP id_provider or just nss-pam-ldapd on FreeBSD and use pam_hbac for access control separately.

Our goal is to make it possible only for the bsduser to log in to the FreeBSD client machine and nobody else. Start the configuration by making sure you can resolve and authenticate the IPA users. Once that is done, we can configure pam_hbac to provide access control for the FreeBSD machine. Without access control configured, any user should be able to log in:

% su - bsduser
Password:
$ id
uid=1207000884(bsduser) gid=1207000884(bsduser) groups=1207000884(bsduser)
% ^D
% su - tuser
Password:
$ id
uid=1207000883(tuser) gid=1207000883(tuser) groups=1207000883(tuser)


The next step is to install and configure and pam_hbac. Because at the moment there are no prebuilt binary packages for FreeBSD, you’ll need to compile the module from source. The steps are documented in the FreeBSD README in pam_hbac’s git repo . After configuring the source tree, building and installing pam_hbac, you’ll end up with the module installed to /usr/local/lib/pam_hbac.so.

Because much of the information that pam_hbac reads is only accessible to an authenticated user, we need to create a special bind user that pam_hbac will authenticate as. To do so, prepare an LDIF file with the following contents:

dn: uid=hbac,cn=sysaccounts,cn=etc,$DC
objectClass: account
objectClass: simplesecurityobject
objectClass: top
uid: hbac
userPassword: $PASSWORD


Replace the $PASSWORD value with the desired password of the bind user and $DC with the base DN of your IPA server. Then add this LDIF to the IPA server:

ipaserver $ ldapadd -ZZ -H ldap://$IPA_HOSTNAME -D"cn=Directory Manager" -W < hbac_sysuser.ldif


Now we can create the configuration file for pam_hbac. The configuration options are documented in the pam_hbac.conf manpage, but in general it’s enough to point pam_hbac to the IPA server and specify the bind user and its credentials. The config file for pam_hbac on FreeBSD is located at /usr/local/etc/pam_hbac.conf:

[root@freebsd ~]# cat /usr/local/etc/pam_hbac.conf
URI = ldap://unidirect.ipa.test
BASE = dc=ipa,dc=test
BIND_DN = uid=hbac,cn=sysaccounts,cn=etc,dc=ipa,dc=test
BIND_PW = Secret123
SSL_PATH = /usr/local/etc/ipa.crt


Next, we add pam_hbac to the PAM configuration so that it enforces access control during the PAM account phase. Because pam_hbac only handles the account phase, we only add a single line to the account stack of /etc/pam.d/system to make it look like this:

account required pam_login_access.so
account required /usr/local/lib/pam_ldap.so no_warn ignore_authinfo_unavail ignore_unknown_user
account required /usr/local/lib/pam_hbac.so ignore_authinfo_unavail ignore_unknown_user
account required pam_unix.so


Finally, we can disable the allow_all rule on the server and instead only allow access to bsduser to the freebsd.ipa.test machine. Please don’t forget to add other rules in your test environment so that you can at least access your IPA masters!

ipaserver $ ipa hbacrule-add freebsd-bsd-user
ipaserver $ ipa hbacrule-add-host --hosts=freebsd.ipa.test freebsd-bsd-user
ipaserver $ ipa hbacrule-add-user --users=bsduser freebsd-bsd-user
ipaserver $ ipa hbacrule-mod --servicecat=all freebsd-bsd-user
ipaserver $ ipa hbacrule-disable allow_all

Time to test pam_hbac! First, we can make sure bsduser is still able to log in:

% su - bsduser
Password:
$ id
uid=1207000884(bsduser) gid=1207000884(bsduser) groups=1207000884(bsduser)
$ ^D


OK, now for a negative test, see if tuser is denied access:

% su - tuser
Password:
su: Sorry


Great! Looking to /var/log/auth.log reveals that it was indeed the account control module that denied access:

May 25 13:26:37 su: pam_acct_mgmt: permission denied


Example 2: using pam_hbac to provide access control for AD users on CentOS-5

One important use-case for pam_hbac is to provide access control for setups that resolve users from a trusted AD domain using the ‘legacy client’ setup in which a CentOS-5 machine is set up with id_provider=ldap pointing to the IPA server’s compat tree. Please note that if your IPA domain doesn’t have a trust relationship established with an AD domain, you can already use HBAC provided by SSSD and you don’t need pam_hbac at all in that case.

In our setup, let’s have an AD group linux_admins. Our goal will be to grant access to the CentOS-5 machine to members of linux_admins and nobody else. First, make sure your CentOS-5 client is able to resolve and authenticate AD users and the user is a member of the linux_admins group. You can use the output of ipa-advise config-redhat-sssd-before-1-9 as a starting point. Once the CentOS-5 client is set up, you’ll be able to resolve the user, id should report the group and you should be able to authenticate as that user. Because HBAC rules can only be linked to IPA POSIX groups, we also need to add the AD group as a member of an IPA external group which in turn needs to be added to an IPA POSIX group:

ipaserver $ ipa group-add --external linux_admins_ext
ipaserver $ ipa group-add-member --groups=linux_admins_ext ipa_linux_admins
ipaserver $ ipa group-add-member --external=linux_admins@win.trust.test linux_admins_ext


Try logging in:

$ su - linuxop@win.trust.test
Password:
-sh-3.2$ id
uid=300403108(linuxop@win.trust.test) gid=300403108(linuxop@win.trust.test) groups=300400513(domain users@win.trust.test),300403108(linuxop@win.trust.test),300403109(linux_admins@win.trust.test),1207000858(ipa_linux_admins) context=user_u:system_r:unconfined_t

OK, the id output reports the user is a member of the ipa_linux_admins group, so we can proceed with setting up the HBAC rules.

In order to set up the HBAC rules correctly, it’s important to understand how AD users are authenticated when using the compat tree – the SSSD client on the CentOS-5 machine does an LDAP bind using the user’s password against the IPA compat tree. This password bind is intercepted by the slapi-nis plugin running on the IPA server which in turn authenticates against the system-auth service on the IPA server itself. Therefore, it’s important that all users who should be allowed to authenticate against the compat tree are allowed access to the system-auth service on the IPA server. More details about the authentication against the compat tree can be found in the

Let’s add the system-auth rule first, together with its HBAC service that allows access to everyone to the IPA server itself using the system-auth PAM service:

ipaserver $ ipa hbacsvc-add system-auth
ipaserver $ ipa hbacrule-add system-auth-everyone
ipaserver $ ipa hbacrule-add-host --hosts=unidirect.ipa.test system-auth-everyone
ipaserver $ ipa hbacrule-mod --usercat=all system-auth-everyone


The resulting HBAC rule would look like this:

ipaserver $ ipa hbacrule-show system-auth-everyone
Rule name: system-auth-everyone
User category: all
Enabled: TRUE
Hosts: unidirect.ipa.test
Services: system-auth


To avoid allowing all access, disable the allow_all rule

ipaserver$ ipa hbacrule-disable allow_all


As a pre-flight check, you can disable the system-auth-everyone rule, then your user should be denied access and the server-side journal should show something like:

pam_sss(system-auth:account): Access denied for user linuxop@win.trust.test: 6 (Permission denied)


Enabling the rule would allow access again. Of course, don’t forget to allow access to your IPA servers and clients with other services and for other users as appropriate!

As the final step on the server, we can define the HBAC rule for our particular CentOS-5 machine. While the access was already checked against the system-auth rule on the server, that rule cannot discriminate between different hosts and applies to all authentication requests coming from the slapi-nis plugin.

Please note that the a bind user must be configured, refer to the BSD example for details.

The rule will permit all members of ipa_linux_admins group to access all PAM services on the host called centos5.ipa.test:

ipaserver $ ipa hbacrule-add centos5-ipa-linux-admins
ipaserver $ ipa hbacrule-add-host --hosts=centos5.ipa.test centos5-ipa-linux-admins
ipaserver $ ipa hbacrule-mod --servicecat=all centos5-ipa-linux-admins
ipaserver $ ipa hbacrule-add-user --groups=ipa_linux_admins centos5-ipa-linux-admins


Now we’re ready to configure the CentOS-5 client. Make sure pam_hbac is installed first – you can use our COPR repository for that. Just drop the repo file to /etc/yum.repos.d and then yum install pam_hbac.

The configuration file for CentOS-5 machine is quite similar to the one we used on BSD earlier:

URI = ldap://unidirect.ipa.test
BASE = dc=ipa,dc=test
BIND_DN = uid=hbac,cn=sysaccounts,cn=etc,dc=ipa,dc=test
BIND_PW = Secret123

The file is located at /etc/pam_hbac.conf.

Next, add pam_hbac to the /etc/pam.d/system-auth file on the CentOS-5 machine to enable HBAC rules enforcement. This snippet shows the whole account stack on my CentOS-5 machine:

account required pam_unix.so
account sufficient pam_succeed_if.so uid < 500 quiet
account [default=bad success=ok user_unknown=ignore] pam_sss.so
account sufficient pam_localuser.so
account [default=bad success=ok user_unknown=ignore] pam_hbac.so
account required pam_permit.so


You can see I also added the pam_localuser.so module just before the line with pam_hbac.so. Adding the pam_localuser.so module ensures that pam_hbac wouldn’t be called for local users defined in /etc/passwd – we only want the HBAC policies to apply to IPA and AD users.

It’s time to check the rule. As a positive check, we log in with the linuxop user:

$ su - linuxop@win.trust.test
Password:
su: warning: cannot change directory to /home/win.trust.test/linuxop: No such file or directory
-sh-3.2$


As a negative test, we can try logging in as the AD administrator perhaps:

$ su - administrator@win.trust.test
Password:
su: incorrect password

And indeed, /var/log/secure would tell us it was pam_hbac that denied access:

May 25 22:48:35 centos5 su: pam_hbac(su-l:account): returning [6]: Permission denied


Awesome, now even your CentOS-5 server enforces HBAC rules!

Troubleshooting pam_hbac

In case something doesn’t work as expected, there are several ways to debug pam_hbac and the HBAC access control in general. The first step should be to check with the ipa command line tools and the hbactest plugin. For example, this is how we’d test the bsduser‘s access to the freebsd.ipa.test machine:

ipaserver $ ipa hbactest --user=bsduser --service=su --host=freebsd.ipa.test
--------------------
Access granted: True
--------------------
Matched rules: freebsd-bsd-user
Not matched rules: centos5-ipa-linux-admins
Not matched rules: ph_test_trule
Not matched rules: system-auth-everyone


The pam_hbac module by default only logs failures. If you want to see more verbose output, add the debug parameter to the PAM service file configuration. All logging, including debug logs is done using the standard pam_syslog() PAM calls, so the location really depends on your operating system. But as an illustration, this is the tail of the debug output for the AD user case when the allowed user logs in:

May 25 22:48:47 centos5 su: pam_hbac(su-l:account): ALLOWED by rule [centos5-ipa-linux-admins].
May 25 22:48:47 centos5 su: pam_hbac(su-l:account): hbac_evaluate() >]
May 25 22:48:47 centos5 su: pam_hbac(su-l:account): Allowing access
May 25 22:48:47 centos5 su: pam_hbac(su-l:account): returning [0]: Success


The full debug output breaks down all the rules into their components and shows what matched and what did not.

Conclusion

This blog post described the first version of pam_hbac. We will continue the development to add more supported platforms – in the next version, we would like to add support for IBM AIX and Apple OS-X. There are also several bugs and minor enhancements we would like to add. Feel free to file an issue on github if there is something you would like to see improved or something doesn’t work for you!

Creating large AD test environments from the Linux command line

I’ve been recently working on making SSSD perform better in large environments, that contain thousands of users or groups. And because two of the types of setup I wanted to test were SSSD directly enrolled into an Active Directory domain and SSSD as a client of an IPA server that trusts an Active Directory domain, I needed to create a large Active Directory environment for testing.

Of course I didn’t want to create the users in the AD Users and Groups GUI snap-in one by one. And I preferred to create the users and groups from the Linux command line, because that’s what I’m familiar with. After some tinkering, I realized the adcli command already does exactly what I need to with its “create-user”, “create-group” and “add-member” commands.

For example, creating an active directory user is as simple as:

 $ adcli create-user --domain=your.ad.domain adusername

To create a group, you’d call:

 $ adcli create-group --domain=your.ad.domain adgroupname

And finally add a user to the group:

 $ adcli add-member --domain=your.ad.domain adgroupname adusername

By default, adcli would ask you for the AD Administrator, but using a Kerberos ccache is as simple as adding the “-c” option.

Using these three commands I wrote three simple shell scripts that helped me create a thousand users, a thousand groups and add members to these groups in one go. The scripts are available at: https://github.com/jhrozek/adcli_scripts

Of course, adcli does much more than just creating user or group object in Active Directory. adcli is a powerful command-line tool for performing actions on an Active Directory domain. You probably used it already without realizing if you ever added an Active Directory client with realmd, because realmd uses adcli internally. In particular, it would use the “adcli join” command.

On RHEL-6, adcli is available already in EPEL and will be available in RHEL-6.8 as a regular RHEL package. In RHEL-7 and supported Fedora distributions, both realmd and adcli are already available in repositories.

SSSD local overrides

Written by Pavel Březina and Jakub Hrozek

In most cases, using the SSSD is all about connecting a client machine to a central user database, like FreeIPA or Active Directory precisely because you want all users on all machines across the domain to have exactly the same properties. However, even though it would be best to centralize all the things, there will always be exceptions. There can be an odd legacy server where a particular user has a UID set to a value that predates the migration to a central domain controller. Or you might administer a machine where a user stores her files in a different home directory than any other machines.

If the client is a member of an FreeIPA domain, you can just define an “ID view” and define the custom values centrally on the IPA server. Using ID views is the recommended way of setting any overriden attributes – however, not all environments use a FreeIPA server..

To solve those use-cases, the SSSD provides a command-line tool that allows the administrator to set one or more POSIX attributes to a different value on that particular system. The tool is called sss_override and is part of the sssd-tools package since version 1.13.1. Running sss_override --help would present the user with a list of available sub-commands. We’ll show some examples of using them later in this text. Please note that the local overrides can not be used in combination with IPA provider. Instead, the server side overrides can be defined directly through IPA web or command line interface.

One very important aspect to keep in mind is that the overrides are stored in the cache, along with the data from the directory server. This conflicts with a very commonly used practice among admins who often remove the database and try to start from a clean cache when they encounter any problems with sssd. Therefore it is necessary to manually export the override data prior deleting the cache and import them again later, otherwise the override data would be lost!

Creating user and group overrides

Let’s consider our domain has a user “tuser” that resolves with his UID and GID as defined on the server:

$ getent passwd tuser
tuser:*:1190000015:1190000015:test user:/home/tuser:/bin/sh

But on this particular host, we need this user to have a different IDs – maybe the user already owned some files before this host was added as a member to the domain. We can call the sss_override tool and assign him a different ID:

$ sudo sss_override user-add tuser -u 1234
SSSD needs to be restarted for the changes to take effect.

As the info message tells us, currently SSSD needs to be restarted when the first override is added in order for the overriden values to take effect. All subsequent overrides will take effect immediately.

$ sudo systemctl restart sssd

Now, let’s request the user again:

$ getent passwd tuser
tuser:*:1234:1190000015:test user:/home/tuser:/bin/sh

And the changes are visible now! Keep in mind that user-add always replaces the whole local override, so if we wanted to override this user’s name, too, the previous UID change would be lost:

$ sudo sss_override user-add tuser -n xuser
$ getent passwd xuser
xuser:*:1190000015:1190000015:test user:/home/tuser:/bin/sh

As you can see, only the login name was overriden, but the UID was reverted to the original value. This is because the -add command replaces the override completely. The correct way to override both values would be to specify both switches at the same time:

$ sudo sss_override user-add tuser -n xuser -u 1234

It’s possible to also override group attributes – at the moment, overriding group’s name or GID is supported. A very common use is to override a group ID, so let’s take a look:

$ id tuser
uid=1190000015(tuser) gid=1190000015(tuser) groups=1190000015(tuser),1190000016(tgroup)

$ sudo sss_override group-add tgroup -g 5555

$ id tuser
uid=1190000015(tuser) gid=1190000015(tuser) groups=1190000015(tuser),5555(tgroup)

Import and export

As explained earlier, the overrides are stored in the sssd cache itself. To provide some recovery mechanism for admins who sometimes need to purge the cache in order to reset the environment into clean state, we added import and export commands to the sss_override tool. With the example above, we can export the override for tuser with:

$ sudo sss_override user-export /var/log/sssd/user_overrides

This creates a passwd-like file that contains the override data:

$ sudo cat /var/log/sssd/user_overrides
tuser@ipaldap:xuser:1234::::

 In case we decide to remove the cache, we can later add the override data back with:

$ sudo sss_override user-import  /var/log/sssd/user_overrides

Listing

To see what overrides are actually configured on the system, you can use sss_override commands user-find, user-show, group-find and group-show.

The command sss_override user-find allows to find all overrides from all configured sssd domains:

$ sudo sss_override user-find
tuser@ipaldap:xuser:1234::::
admin@anotherdomain:administrator:4321::::

We can also direct the user-find command to a particular domain:

$ sudo sss_override user-find -d ipaldap
tuser@ipaldap:xuser:1234::::

 Or, using the user-show command, even a particular user:

$ sss_override user-show tuser
tuser@ipaldap:xuser:1234::::

Conclusion

The local overrides are a powerful mechanism to deal with irregularities in an otherwise centralized environment. That said, using a completely centralized solution, like IPA’s idviews feature avoids issues with keeping the overrides in the cache itself as well as provides a unified view of all overrides in the UI.

As an incremental improvement, the SSSD team is looking into ways of making the import and export more automated so that the danger of removing the database is mitigated by the SSSD tools.

Performance tuning SSSD for large IPA-AD trust deployments

Written by Alexander Bokovoy and Jakub Hrozek

This blog post describes several sssd.conf options that are available for performance tuning of SSSD, especially focusing on deployment of an IPA server with trust established with an AD server. Some of the options are useful for other scenarios as well, but it should be noted that diverting from the defaults is something that needs understanding of the implication the change has. This post is also written with SSSD version 1.12 and 1.13 in mind – later versions might have different tunables available or might not need these at all as we continuously improve performance.

The anatomy of a trusted identity lookup

The typical use-case this post is trying to address is “a login to my IPA client using AD users and his AD credentials is way too slow”. Later in the article, we’ll be setting several options in the IPA server’s sssd.conf. That might be confusing if you’re not familiar with the architecture of the IPA-AD trust lookups and authentication data flow – so let’s illustrate it briefly.

The two basic requests type the client can perform are identity lookups and authentication. For IPA users, the identity lookups are normal LDAP searches against the IPA server and authentication is performed using an internal kinit-like tool also connecting to the IPA server. Trusted AD users work differently — the identity lookups from the client machines connect to the IPA server using an LDAP extended operation. The IPA server’s extdom plugin queries the SSSD instance running on the server for the AD user information, the SSSD instance runs a search against the Active Directory server using the LDAP protocol and passes the data back to the IPA client. For password-based authentication requests, the IPA clients connect directly to the AD servers using Kerberos.

Keep in mind that because the correct set of groups must be set during authentication time, the authentication request must also perform an identity lookup, typically the initgroups operation, before the authentication. The following diagram illustrates the data flow:

The diagram illustrates the data flow in case a Windows user logs in using his Kerberos credentials (passwordless) to an IPA client:

1. The windows user opens an SSH client (like putty) and logs in to an IPA client
2. The SSH daemon initiates processing of the Kerberos ticket
3. The Kerberos tickets of Active Directory users have a blob attached to them, called MS-PAC (a privilege attribute certificate, see https://msdn.microsoft.com/en-us/library/cc237917.aspx for details of the specification). The MS-PAC (or just PAC going forward) contains the complete and precise list of group memberships at the time user logged in into the Active Directory domain, signed by the domain controller. It is actually the mechanism Windows clients use to set list of the groups the user is a member of. If there is a PAC present, then a special libkrb5 plugin invokes the sssd_pac responder.
4. The sssd_pac responder parses the group memberships from the PAC blob during login. This processing needs to resolve internal identifiers of groups in Active Directory (SIDs) to their names and POSIX IDs because Linux cannot yet deal with SIDs natively. For each SID identifier that is not stored in cache yet, the sssd_pac process asks the sssd_be process to translate the SID into a name and/or GID.
5. The sssd_be process calls an LDAP extended operation towards the IPA server, asking for information about a particular SID.
6. The extended operation is received by the IPA server’s Directory server plugin, that in turn queries the SSSD instance running on the IPA server through the standard NSS interface, which contacts the sssd_nss process on the IPA server
7. If the information about this object’s SID is not present in the cache on the IPA server itself either, then the sssd_be process is asked to resolve this SID and store the object that corresponds to the SID into the SSSD cache on the server
8. The sssd_be process on the server searches the Active Directory server’s LDAP store for the corresponding LDAP objects and stores them into the cache. When the object is stored, the flow reverses – the sssd_be process on the server tells the sssd_nss process the cache is up-to-date. The sssd_nss process returns data to the DS plugin on the server, which in turn returns data in the extdom-extop operation reply to the client.
9. Finally, all the groups from the PAC object are processed. When the SSH daemon on the client opens the session for the user, it would call the initgroups() function to set up group memberships. When this function’s equivalent reaches the sssd_nss responder, all the data would be cached from processing the PAC blob and returned from the SSSD cache.

Please note that for simplicity, this diagram omits additional actions that might happen during login, such as HBAC access control or setting the SELinux label for the user.

If password-based authentication is used, the data flow is quite similar, except the sssd_pac responder is not involved until later in the process when the ticket is acquired on the IPA client itself. With identity lookup (for example id aduser@ad_domain), the PAC responder is not invoked at all, but all lookups are driven completely by the sssd_nss responder. But even here, the general flow between IPA client and IPA server is the same.

Also keep in mind that retrieving members of trusted groups or retrieving user groups prior to login requires RHEL-7.1 or RHEL-6.7 packages on the client side. Additionally, the IPA server must be running RHEL-7.1 or newer at the same time.

Not all IPA masters are capable to resolve AD users this way. In FreeIPA < 4.2, only IPA masters where ipa-adtrust-install command was executed can retrieve information from Active Directory domain controllers. Such IPA masters were also used for establishing actuall cross-forest trust to Active Directory and AD domain controllers were talking back to IPA masters as part of that flow. With FreeIPA 4.2 a concept of ‘AD trust agents’ was introduced that allows other IPA masters to resolve AD users and groups without the need to configure them as a full domain controller.

The data flow also has implications on debugging. If it’s not possible to fetch data about a trusted user on the IPA client (ie getent passwd aduser@ad_domain doesn’t display the user), first make sure it’s possible to run the command on the IPA server. The error messages on the client would normally manifest in functions that contain s2n in the name, such as:

[ipa_s2n_exop_done] (0x0040): ldap_extended_operation result: Operations error(1), Failed to handle the request.

or:

[ipa_s2n_get_user_done] (0x0040): s2n exop request failed.


The most data-intensive operation is usually retrieving the information about user’s groups on the server side. If the group objects are not cached yet, that operation can amount to downloading a large amount of data, storing it in server’s sssd cache, transferring the data to clients and storing them again in client’s sssd cache. Subsequent lookups for the same LDAP objects, even from another client instance would then just read the objects from the server side sssd’s cache, eliminating the server side costs, but the very first search might be costly in a huge environment.

Available tunables

Now that we know how the flow works, we can start optimizing it. Because the heavy lifting of the lookups is done on the server side, the changes in server’s sssd.conf also have the biggest impact on the lookups performance.

Server options

The following options should be added to the /etc/sssd/sssd.conf file on the IPA masters.

ignore_group_members

Normally the most data-intensive operation is downloading the groups including their members. Usually, we are interested in what groups a user is a member of (id aduser@ad_domain) as the initial step rather than what members do specific groups include (getent group adgroup@ad_domain). Setting the ignore_group_members option to True makes all groups appear as empty, thus downloading only information about the group objects themselves and not their members, providing a significant performance boost. Please note that id aduser@ad_domain would still return all the correct groups!

• Minimal version: For users from IPA domain, this option was introduced in sssd-1.10. However, it’s only possible to use it for trusted domain users via the subdomain_inherit option (see below).
• Recommended value: ignore_group_members=True unless your environment requires the output of getent group to also contain members
• Pros: getgrnam/getgrgid calls are significantly faster
• Cons: getgrnam/getgrgid calls only return the group information, not the members

ldap_purge_cache_timeout

SSSD 1.12.x runs a cache cleanup operation on startup and then periodically. The cache cleanup operation removes cached entries that were not used for a while to make sure the cache doesn’t grow too large. But since SSSD can also operate in offline mode, we must be very conservative about what is removed, otherwise we might lock out user’s access to a file. At the same time, searching the cache and examining the entry could take a lot of CPU time. Therefore, the cache cleanup operation was of limited use and was disabled by default in 1.13. You can disable it in previous versions with a config change as well.

• Minimal SSSD version: All supported SSSD versions have this option.
• Recommended value: ldap_purge_cache_timeout = 0
• Pros: the cache cleanup operation was not particularly useful as a periodic task, but took a long time to execute
• Cons: the cache can grow in size.

subdomain_inherit

The options above can be passed to the trusted AD domains’ configuration. At the moment, the only supported method is using the subdomain_inherit option in the sssd.conf‘s domain section. Any of the two options’ names from above can be listed as a value of subdomain_inherit and they will apply to both the main (IPA) domain as well as the AD subdomain. In the future, we would prefer to add support for sub-sections in sssd.conf, but we’re not there yet.. The complete list of options that can be inherited by a subdomain is listed in the sssd.conf manual page.

• Miminal version: Upstream 1.12.5. However, this option has been backported to both RHEL-6.7 and also RHEL-7.1 updates.
• Pros-Cons: N/A, this is just a control option to extend influence of the options above to domains from a trusted AD forest.

Mount the cache in tmpfs

Quite a bit of the time spent processing the request is writing LDAP objects to the cache. Because the cache maintains full ACID properties, it does disk syncs with every internal SSSD transaction, which causes data to be written to disk. On the plus side, this makes sure the cache is always available in case of a network outage and will be usable after a machine crash, but on the other hand, writing the data takes time. It’s possible to mount the cache into a ramdisk, elliminating the disk IO cost by adding the following to /etc/fstab as a single line:

tmpfs /var/lib/sss/db/ tmpfs size=300M,mode=0700,rootcontext=system_u:object_r:sssd_var_lib_t:s0 0 0
Then mount the directory and restart the sssd afterwards:

# mount /var/lib/sss/db/
# systemctl restart sssd


Please tune the size parameter according to your IPA and AD directory size. As a rule of thumb, you can use 100MBs per 10000 LDAP entries.

Doing this change on the IPA server is a bit safer than IPA clients, because the SSSD instance on the server will never lose connectivity to the IPA server, so the cache can always be rebuilt. But in case the cache was lost after a reboot and the AD server was not reachable due to a network error or a similar condition, the node would not be able to fall back to cached data about AD users.

• Pros: I/O operations on the cache are much faster.
• Cons: The cache does not persist across reboots. Practically, this means the cache must be rebuilt after machine reboot, but also that cached password are lost after a reboot.

All in all, we’re looking at adding the following changes to the server side’s sssd.conf:

[domain/domname]
subdomain_inherit = ignore_group_members, ldap_purge_cache_timeout
ignore_group_members = True
ldap_purge_cache_timeout = 0

and optionally an fstab change to remount the database directory in a
ramdisk.

Client options

The following options should be added to /etc/sssd/sssd.conf on the IPA clients.

pam_id_timeout

On a Linux system, user group membership is set for processes during login time. Therefore, during PAM conversation, SSSD has to prefer precision over speed and contact the server for accurate information. However, a single login can span over multiple PAM requests as PAM processing is split into several stages – for example, there might be a request for authentication and a separate request for account check (HBAC). It’s not beneficial to contact the server separately for both requests, therefore we set a very short timeout for the PAM responder during which the requests will be answered from in-memory cache. The default value of 5 seconds might not be enough in cases where complex group memberships are populated on server and client side. The recommended value of this option is as long as a single un-cached login takes.

• Recommended value: pam_id_timeout = login_time_in_seconds
• Pros: The remote server would only be contacted once during a login session
• Cons: If the group memberships change rapidly on the server side, SSSD might still only use the cached values

krb5_auth_timeout

In case the Kerberos ticket has a PAC blob attached to it (see above) and password authentication is used, the krb5_child processes the PAC blob to help establish the group memberships. This processing might take longer than the time the krb5_child process is allowed to run. Therefore, for environments where users are members of a large number of groups, the krb5_auth_timeout value should be increased to allow the groups to be processed. In future SSSD versions, we aim at making the processing much faster, so the default 6 seconds timeout would suffice. Contacting the PAC responder can also be avoided completely by disabling the krb5_validate option, however disabling that option has security implications as we can’t any longer verify the TGT has not been spoofed with a MITM attack.

• Recommended value: krb5_auth_timeout = login_time_in_seconds
• Pros: The PAC responder has enough time to process user’s group memberships
• Cons: Detecting legitimate offline situations might take too long

Mount the cache to tmpfs

Please see the server side section. On the client side, there is an additional caveat – the risk the client would reboot and then lose connectivity to the server is higher. Please do not set this option on roaming clients where you rely on offline logins!

To sum up the client side login changes, we’re looking at these additions to the config file:

[pam]
pam_id_timeout = N

[domain/domname]
krb5_auth_timeout = N
# Disabling the validation is dangerous!!
# krb5_validate = false

Don’t forget to restart the SSSD instance for the new settings to take effect!

Get rid of manually calling kinit with SSSD’s help

For quite some time, my laptop doesn’t have a UNIX /etc/passwd user for the account I normally use at all. Instead, I’ve used SSSD exclusively, logging in with our Red Hat corporate account. That allows me to use some cool SSSD features, most notably the krb5_store_password_if_offline option.

When that option is enabled and you log in while SSSD is offline (typically when a laptop was just started and is not connected to corporate VPN yet) using your cached password, SSSD stores the password you use to the kernel keyring. Then, when SSSD figures out that the servers are reachable, it attempts to kinit using that password on your behalf.

Because SSSD can detect when you connected to the VPN by listening to libnl events or watching resolv.conf for changes, the routine of connecting to Kerberos simplifies rapidly to:

• open laptop
• login using Kerberos password
• connect to VPN

That’s it. kinit happens on the background and Kerberos ticket is acquired without any user intervention. Now you can just start Firefox and start accessing HTML pages or start Thunderbird and read your mail. No more starting an extra terminal and typing kinit or starting a separate Gnome dialog.

However, until now, this setup required that you also use an account from a centralized store that contains the Kerberos principal as one of its attributes or at least has the same user name as the first part of the Kerberos principal (called primary, up to the ‘@’ sign). And that’s a deal breaker for many people who are used to a certain local user name for ages, irrespective of their company’s Kerberos principal.

SSSD 1.12.5 that was released recently includes a very nice new feature, that allows you to work around that and map a local UNIX user to a particular Kerberos principal. The setup includes a fairly simple sssd.conf file:

 [sssd]
 domains = example.com
 config_file_version = 2
 services = nss,pam

 [domain/example.com]
 id_provider = proxy
 proxy_lib_name = files

 auth_provider = krb5
 krb5_server = kdc.example.com
 krb5_realm = EXAMPLE.COM
 krb5_map_user = jakub:jhrozek
 krb5_store_password_if_offline = True

 cache_credentials = True

All the interesting configuration is in the [domain] section. The identities are read from good old UNIX files by proxying nss_files through SSSD’s proxy provider. Authentication is performed using the krb5 provider, with a KDC server set to kdc.example.com and Kerberos realm EXAMPLE.COM

The krb5_map_user parameter represents new functionality. The parameter takes a form of username:primary and in this particular case maps a UNIX user jakub to a Kerberos principal which would in this case be jhrozek@EXAMPLE.COM.

On Fedora-22, care must be taken that the pam_sss.so module is called in the PAM stack even if pam_unix.so fails as the default behaviour for pam_unix.so is default=die – end the PAM conversation in case the user logging in exists in /etc/passwd and the password is incorrect. Which is our case, we’re logging in with UNIX user, we just want the authentication to be done by pam_sss.so.

Thanks to Sumit Bose who helped me figure out the PAM config, I used the following auth stack successfully:

auth     required pam_env.so
auth     [default=2 success=ok] pam_localuser.so
auth     sufficient pam_unix.so nullok try_first_pass
auth     [success=done ignore=ignore default=die] pam_sss.so use_first_pass
auth     requisite pam_succeed_if.so uid >= 1000 quiet_success
auth     sufficient pam_sss.so forward_pass
auth     required pam_deny.so

I also recorded a video shows the feature in action. The first login is with the UNIX password, so the ticket is not automatically acquired and the user must run kinit manually. The next login happens with the Kerberos password while the machine is connected to network, so the ticket is acquired on login. Finally, the last login simulates the case where the machine starts disconnected from the network, then connects. In this case, SSSD acquires the ticket on the background when it detects the networking parameters have changed and the KDC is available.

Nice, right? You can also get the video in a nicer resolution.

The Kerberos authentication provider has more features that can beat using kinit from the command line – automatized ticket renewal might be one of them. Please refer to the sssd-krb5 manual page or share your tips in the comments section!

Authenticating a docker container against host’s UNIX accounts

Recently, with the advent of Docker and similar technologies, there’s been an effort to containerize different kinds of setups that previously were running on a single machine or a set of tightly coupled machines. In case the components communicate over a network connection, the containerization might not be that hard, but what about cases where the components would talk over a strictly local interface, such as a UNIX socket?

A colleague of mine asked me to help setup up a container to authenticate against the host’s UNIX accounts using the PAM API the other day. It turned out to be doable, but maybe not obvious to anyone not familiar with some of the more esoteric SSSD options, so I’d like to write down the instructions in a blog post.

In our setup, there was a pam service, sss_test, that previously would run on the host and authenticate against accounts either locally stored on the hosts in /etc/passwd and /etc/shadow. The same setup could in principle be used to authenticate against a remote database using SSSD, just the host SSSD settings would be different.

So how does this work with containers? One possibility, especially with a remote database such as IPA or AD would be to run an SSSD instance in every container and authenticate to the remote store. But that doesn’t help us with cases where we’d like to authenticate against the local database stored on the host, since it’s not exposed outside the host. Moreover, putting SSSD into each container might also mean we’d need to put a keytab in each container, then each container would open its separate connection to the remote server..it just gets tedious. But let’s focus on the local accounts..

The trick we’ll use is bind-mounting. It’s possible to mount part of host’s filesystem into the container’s filesystem and we’ll leverage this to bind-mount the UNIX sockets SSSD communicates over into the container. This will allow the SSSD client side libraries to authenticate against the SSSD running on the host. Then, we’ll set up SSSD on the host to authenticate aganst the hosts’s UNIX files with the proxy back end.

This is the traditional schema:
application -> libpam -> pam_authenticate -> pam_unix.so -> /etc/passwd

If we add SSSD to the mix, it becomes:
application -> libpam -> pam_authenticate -> pam_sss.so -> SSSD -> pam_unix.so -> /etc/passwd


Let’s configure the host and the container, step by step.

1. Host config
1. Install packages

yum -y install sssd-common sssd-proxy

2. create a PAM service for the container.

The name doesn’t matter, it just needs to be referenced from sssd.conf later.
# cat /etc/pam.d/sss_proxy
auth required pam_unix.so
account required pam_unix.so
password required pam_unix.so
session required pam_unix.so


3. create SSSD config file, /etc/sssd/sssd.conf

Please note that the permissions must be 0600 and the file must be owned by root.root.
# cat /etc/sssd/sssd.conf
[sssd]
services = nss, pam
config_file_version = 2
domains = proxy
[nss]
[pam]
[domain/proxy]
id_provider = proxy
# The proxy provider will look into /etc/passwd for user info
proxy_lib_name = files
# The proxy provider will authenticate against /etc/pam.d/sss_proxy
proxy_pam_target = sss_proxy


4. start sssd

# systemctl start sssd

5. verify a user can be retrieved with sssd

$ getent passwd -s sss localuser

2. Container setup

It’s important to bind-mount the /var/lib/sss/pipes directory from the host to the container since the SSSD UNIX sockets are located there.

# docker run -i -t -v=/var/lib/sss/pipes/:/var/lib/sss/pipes/:rw --name sssd-cli fedora:latest /bin/sh


3. Container config

The container runs the PAM-aware application that authenticates against the host. I used a simple program in C that comes from the SSSD source. It pretty much just runs pam_authenticate() against a service called sss_test. Your application might need a different service, but then you just need to adjust the filename in /etc/pam.d/ in the container. All the steps below should be executed on the container itself.

1. install only the sss client libraries

# yum -y install sssd-client

2. make sure sss is configured for passwd and group databases in /etc/nsswitch.conf

# grep sss /etc/nsswitch.conf

3. configure the PAM service that the application uses to call into SSSD


# cat /etc/pam.d/sss_test
auth required pam_sss.so
account required pam_sss.so
password required pam_sss.so
session required pam_sss.so


4. authenticate using your PAM application.

In my case that was
# ./pam_test_client auth localtest
action: auth
user: localtest
testing pam_authenticate
Password:
pam_authenticate: Success


4. Profit!

It would be possible to authenticate against any database, just by changing what the SSSD on the host authenticates against. There’s several gotchas, though, especially should you require that only certain containers are allowed to retrieve users from certain domains. Multitenancy doesn’t really work well, because we don’t have a good mechanism to retrieve the identity of the container.

Anatomy of SSSD user lookup

This blog post describes how a user lookup request is handled in SSSD. It should help you understand how the SSSD architecture looks like, how the data flows in SSSD and as a result help identify which part might not be functioning correctly on your system. It is aimed mostly at users and administrators – for developers, we have a separate document about SSSD internals on the SSSD wiki written by Yassir Elley. This document re-uses some of the info from the internals one.

We’ll look at the most common operation, looking up user info on a remote server. I won’t go into server-specific details, so most of the info should be equally true for LDAP, Active Directory or FreeIPA servers. There’s also more functionality in SSSD than looking up users, such as sudo or autofs integration, but they are out of scope of this post as well.

Before going into SSSD details, let’s do a really quick intro into what happens on the system in general when you request a user from a remote server. Let’s say the admin configured SSSD and tests the configuration by requesting the admin user:

$ getent passwd admin

When user information is requested about a user (with getent, id or similar), typically one of the functions of the Name Service Switch, such as getpwnam() or initgroups() in glibc is called. There’s lots of information about the Name Service Switch in the libc manual, but for our purposes, it’s enough to know that libc opens and reads the config file /etc/nsswitch.conf to find out which modules should be contacted in which order. The module that all of us have on our Linux machines is files which can read user info from /etc/passwd and user info from /etc/groups. There also exists an ldap module that would read the info directly from an LDAP server and of course an sss module that talks to SSSD. So how does that work?

The first thing to keep in mind is that, unlike nss_ldap or pam_ldap, the SSSD is not just a module that is loaded in the context of the application, but rather a deamon that the modules communicate with. Almost no logic is implemented in the modules, all the functionality happens in the deamon. A user-visible effect during debugging is that using strace is not too helpful as it would only show if the request made it to the SSSD. For debugging the rest, the SSSD debug logs should be used.

Earlier I said that SSSD is a deamon. That’s really not too precise, SSSD is actually a set of deamons that communicate with one another. There are three kinds of SSSD processes. One is the sssd process itself. Its purpose is to read the config file after startup and spawn the other processes according to the config file. Then there are responder or front end processes that listen to queries from the applications, like the query that would come from the getent command. If the responder process needs to contact the remote service for data, it talks to the last SSSD process type, which is the data provider or back end process. This architecture allows for a pluggable setup where there are different back end processes talking to different remote servers, while all these remote servers can be accessed from a range of applications or subsystems by the same lookup code in the responders.

Each process is represented by a section in the sssd.conf config file. The main sssd process is represented by the [sssd] section. The front end processes are defined on the services line in the [sssd] section and each can be configured in a section named after the service. And finally, the back end processes are those configured in the [domain] sections. Each process also logs into its own logfile.

Let’s continue with the getent passwd admin example. To illustrate the flow, there is a diagram that the text follows. The full arrows represent local IO operation (like opening a file), the empty arrows represent local IPC over UNIX sockets and the dotted arrow represents a network IO.

The user issued the getent command which calls libc’s getpwnam (diagram step 1), then the libc opens the nss_sss module as per nsswitch.conf and passes in the request. First, the nss_sss memory-mapped cache is consulted, that’s step 2 on the diagram. If the data is present in the cache, it is just returned without even contacting the SSSD, which is extremely fast. Otherwise, the request is passed to the SSSD’s responder process (step 3), in particular sssd_nss. The request first looks into the SSSD on-disk cache (step 4). If the data is present in the cache and valid, the nss responder reads the data from the cache and returns them to the application.

If the data is not present in the cache at all or if it’s expired, the sssd_nss request queries the appropriate back end process (step 5) and waits for reply. The back end process connects to the remote server, runs the search (step 6) and stores the resulting data into the cache (step 7). When the search request is finished, the provider process signals back to the responder process that the cache is updated (step 8). At that point, the front-end responder process checks the cache again. If there’s any data in the cache after the back end has updated it, the data is returned to the application – even in cases when the back end failed to update the cache for some reason, it’s better to return stale data than none. Of course, if no data is found in the cache after the back end has finished, an empty result is returned back. This final cache check is represented by step 9 in the diagram.

When I said the back end “runs a search” against the server, I really simplified the matter a lot. The search can involve many different steps, such as resolving the server to connect to, authenticating to the server, performing the search itself and storing the resulting data into the database. Some of the steps might even require a helper process, for instance authenticating against a remote server using a keytab is done in a heper process called ldap_child that logs into its own logfile called /var/log/sssd/ldap_child.log.

Given most steps happen in the back end itself, then most often, the problem or misconfiguration lies in the back end part. But it is still very important to know the overall architecture and be able to identify if and how the request made it to the back end at all. In the next part, we’ll apply this new information to perform a small case study and we will repair a buggy sssd setup.

Troubleshooting a failing SSSD user lookup.

With the SSSD architecture in mind, we can try a case study. Consider we have an IPA client, but no users, not even the admin show up:

$ getent passwd admin
$ echo $?
2


The admin user was not found! Given our knowledge of the architecture, let’s first see if the system is configured to query sssd for user information at all:

$ grep passwd /etc/nsswitch.conf
passwd: files sss


It is. Then the request was passed on to the nss responder process, since the only other possibility is a successful return from the memory cache. We need to raise the debug_level in the [nss] section like this:

[nss]
debug_level = 7

and restart sssd:

# systemctl restart sssd

Then we’ll request the admin user again and inspect the NSS logs:

[sssd[nss]] [accept_fd_handler] (0x0400): Client connected!
[sssd[nss]] [sss_cmd_get_version] (0x0200): Received client version [1].
[sssd[nss]] [sss_cmd_get_version] (0x0200): Offered version [1].
[sssd[nss]] [nss_cmd_getbynam] (0x0400): Running command [17] with input [admin].
[sssd[nss]] [sss_parse_name_for_domains] (0x0200): name 'admin' matched without domain, user is admin
[sssd[nss]] [nss_cmd_getbynam] (0x0100): Requesting info for [admin] from []
[sssd[nss]] [nss_cmd_getpwnam_search] (0x0100): Requesting info for [admin@ipa.example.com]
[sssd[nss]] [sss_dp_issue_request] (0x0400): Issuing request for [0x4266f9:1:admin@ipa.example.com]
[sssd[nss]] [sss_dp_get_account_msg] (0x0400): Creating request for [ipa.example.com][4097][1][name=admin]
[sssd[nss]] [sss_dp_internal_get_send] (0x0400): Entering request [0x4266f9:1:admin@ipa.example.com]
[sssd[nss]] [sss_dp_get_reply] (0x1000): Got reply from Data Provider - DP error code: 1 errno: 11 error message: Fast reply - offline
[sssd[nss]] [nss_cmd_getby_dp_callback] (0x0040): Unable to get information from Data Provider
Error: 1, 11, Fast reply - offline
Will try to return what we have in cache

Well, apparently the request for the admin user was received and passed on to the back end process, but the back end replied that it switched to offline mode..that means we need to also enable debugging in the domain part and continue investigation there. We need to add debug_level to the [domain] section and restart sssd again. Then run the getent command and inspect the file called /var/log/sssd/sssd_ipa.example.com starting with the time that corresponds to the NSS responder sending the data (as indicated by sss_dp_issue_request in the nss log). In the domain log we see:

[sssd[be[ipa.example.com]]] [fo_resolve_service_done] (0x0020): Failed to resolve server 'master.ipa.example.com:389': Domain name not found
[sssd[be[ipa.example.com]]] [set_server_common_status] (0x0100): Marking server 'master.ipa.example.com:389' as 'not working'
[sssd[be[ipa.example.com]]] [be_resolve_server_process] (0x0080): Couldn't resolve server (master.ipa.example.com:389), resolver returned (11)
[sssd[be[ipa.example.com]]] [be_resolve_server_process] (0x1000): Trying with the next one!
[sssd[be[ipa.example.com]]] [fo_resolve_service_send] (0x0100): Trying to resolve service 'IPA'
[sssd[be[ipa.example.com]]] [get_server_status] (0x1000): Status of server 'master.ipa.example.com:389' is 'not working'
[sssd[be[ipa.example.com]]] [get_port_status] (0x1000): Port status of port 0 for server '(no name)' is 'not working'
[sssd[be[ipa.example.com]]] [get_server_status] (0x1000): Status of server 'master.ipa.example.com:389' is 'not working'
[sssd[be[ipa.example.com]]] [fo_resolve_service_send] (0x0020): No available servers for service 'IPA'
[sssd[be[ipa.example.com]]] [be_resolve_server_done] (0x1000): Server resolution failed: 5
[sssd[be[ipa.example.com]]] [sdap_id_op_connect_done] (0x0020): Failed to connect, going offline (5 [Input/output error])
[sssd[be[ipa.example.com]]] [be_ptask_create] (0x0400): Periodic task [Check if online (periodic)] was created
[sssd[be[ipa.example.com]]] [be_ptask_schedule] (0x0400): Task [Check if online (periodic)]: scheduling task 70 seconds from now [1426087775]
[sssd[be[ipa.example.com]]] [be_run_offline_cb] (0x0080): Going offline. Running callbacks.

OK, that gets us somewhere. Indeed, our /etc/resolv.conf file was ponting to a bad nameserver. And indeed, after fixing the resolver settings and restarting SSSD, everything seems to be working:

$ getent passwd admin
admin:*:1546600000:1546600000:Administrator:/home/admin:/bin/bash

Awesome, we were able to repair a broken SSSD setup!