Overview
Identities and authentication are key principals in Kubernetes security. Although implementations might differ between various cloud providers, in general, we can divide authentication in Kubernetes into three main areas:
- How users (or applications) authenticate with the cluster.
- How applications authenticate within the cluster.
- How applications in the cluster authenticate with outside resources in the cloud.
In this article, we will focus on areas (2) and (3). We will demonstrate how attackers might leverage those identities to escalate their permissions and how defenders can detect those threats. But first, let’s have a quick overview of all three areas.
Users authenticate with the cluster in various methods. Kubernetes supports several authentication mechanisms, including basic authentication, Bearer tokens, client certificates and OpenID Connect, which is implemented by most cloud providers.
In some cases, applications running in Kubernetes need to authenticate with the Kubernetes API server. This is done by Kubernetes service accounts that represent an application identity in the cluster. The main use case of a service account is authentication between the running pods and the API server, but service account’s tokens are valid from outside the cluster as well.
Applications might also need to authenticate with cloud resources that reside outside the cluster. There are several solutions for such authentication, but a common method is to use cloud identities that are attached to the nodes or the workloads in the cluster.
In this article we will elaborate on the security aspects of cluster service accounts and the usage of cloud managed identities in the cluster.
Service account activity
Service accounts are used to authenticate applications with the Kubernetes API. Kubernetes uses RBAC as an authorization mechanism: service accounts are bound to roles (in the namespace level) and cluster-roles (in the cluster level). Service account tokens are mounted to the running pods in the cluster. If attackers gain access to a running pod, they can extract the token and use it for accessing the cluster’s control plane. Service account tokens are stored as Kubernetes secrets, so if attackers have read permissions to the secrets of a certain namespace, they can get tokens of all the service accounts in the same namespace.
In this context, selfsubjectrulesreviews is a very useful Kubernetes API call for attackers. This API, also known as “can-i” due to its kubectl implementation, retrieves the permissions of the specified service account\user. Attackers who gain access to a container can use this API to check the permissions of the mounted service account and act accordingly. For example, attackers might discover that while they don’t have permissions to create new containers in the cluster, they have permissions to update existing deployments, and therefore they can run their own code by modifying an existing deployment and even escape to the host that way (e.g. by changing the configuration to privileged). Such kind of RBAC misconfigurations were observed in production environments and were exploited for cluster takeover.
One can use the selfsubjectrulesreviews API by kubectl client or by using the API call directly. For calling the API directly, a JSON with the request’s body should be prepared.
Then, this JSON is passed as the body in the request to the selfsubjectrulesreviews API:
How to detect such activity?
The Kubernetes audit log records the cluster’s control plane. It gives visibility to the operations in the Kubernetes API, including those that were initiated by service accounts. Therefore, we can use audit log to detect suspicious operations that were performed by service accounts in the cluster.
Tracking calls to selfsubjectrulesreviews API can help finding reconnaissance activities of service accounts in the cluster.
To detect lateral movement in the cluster, we can track the activity of the service accounts. In many cases, the behavior of a service account is predictable: Usually a service account is attached to a specific application that accesses the API server with a specific pattern. In such cases, abnormal behavior of the service account is suspicious. By tracking the normal behavior of the various service accounts in the cluster and looking for deviations from this behavior, we can find malicious usage of service accounts.
Microsoft Defender for Cloud (MDC) can now detect suspicious operations of service accounts. MDC tracks the behavior of service accounts and alerts when a suspicious operation is detected. In the example below, MDC alerted on an attempt to read Kubernetes secrets by a service account that doesn’t legitimately perform this action.
Cloud Identity activity
Workloads in Kubernetes might need access to cloud resources. Common use cases are accessing cloud storage services, cloud secret stores, and more.
There are several approaches to achieve that: one of them is using cloud identities that are attached to the nodes, allowing the nodes and the workloads to securely authenticate with other cloud resources. For example, in Azure, AKS uses managed identities (MI) to manage the cloud resources that are required for the cluster operation.
Tokens for those identities are retrieved by a request to a metadata service that is accessible from the nodes. Depending on the configuration, the metadata service might be accessible also from the running containers. Therefore, in case attackers have access to a running container, they can acquire tokens for the identities that are attached to the nodes by accessing the metadata service.
As Kubernetes is often run in the cloud, this method allows lateral movement from the Kubernetes cluster to other resources outside the cluster. Such behavior has already been observed in large scale campaigns that target Kubernetes clusters. For detecting such behavior, it isn’t enough to monitor the cluster’s control plane, we also need to monitor the cloud’s control plane. While the monitoring is cloud-specific, the principal is similar: We should track the behavior of the cloud identities for suspicious activity. MDC can help you to detect such activity in AKS clusters. MDC monitors Azure Resource Manager (ARM) for identifying suspicious operations of managed identities assigned to AKS clusters and alerts when a suspicious operation occurs.
To conclude, identities are a key aspect of Kubernetes security, and monitoring their activity is crucial for keeping your cluster secured. The Kubernetes audit log and the cloud control plane logs can be used for identifying suspicious activity of the identities in Kubernetes in both levels: the cluster-level and the cloud-level. Microsoft Defender for Cloud can help you detect malicious operations of the service account in your Kubernetes clusters. Also, MDC monitors Azure Resource Manager and detects suspicious operations of the managed identities that are assigned to your cluster.