Securing Containers from Build to Runtime

Introduction

Over the last decade the industry has seen a shift from traditional application deployment to a faster Continuous Integration/Continuous Delivery (CI/CD) approach that is causing significant challenges for security teams. These threats on new services, including Containers, Serverless Functions, APIs (Application Programming Interfaces), Source code, Infrastructure-as-Code, and other PaaS (platform as a service) services, differ from what teams are used to with typical server environments. Understanding these new threats and vulnerabilities in the application development process is becoming critical as the adversaries' understanding of the development lifecycle is increasing and making applications more vulnerable.

 

One of the new services mentioned above, containers, has become a major part of cloud native application development. A container is a standalone package of software files that contains everything you need to run an application, including the application’s code, dependencies, library, runtime, and system tools. Because of this, containers have made application development simpler, faster, and much more powerful. To manage containers, Kubernetes, an open-source platform, is used to make it easy to build an automated set of processes such as application development, deployment, and management.

 

Despite the benefits of using containers, security teams are being tasked with the new challenge of securing these new cloud-native applications. Unlike traditional compute environments, containerized applications are elastic, spawn, die and re-size rapidly. The container images are immutable, and containers are short-lived, meaning that vulnerabilities are required to be fixed early in the delivery pipeline. To fully secure containerized workloads, a team must secure the containers and the code running within them, software dependencies and libraries, CI/CD pipelines, runtime, and everything in between. The end goal of container security should be to have zero vulnerabilities in running deployments as outlined in Figure 1 below.

 

Figure 1. Container Pipeline Security

 

How Does the CI/CD Development Process Work?

Understanding how these development tools work together is vital in understanding how to secure the end-to-end development process, e.g., Pre-Deployment, CI/CD Integrations, Runtime protection. There are many methodologies such as the Microsoft Security Development Lifecycle. Many of these methodologies follow a pattern of using source code management/visual code tools such as GitHub or Azure DevOps to store the source code and is part of the pre-deployment stage. The most common next step is checking the image into things such as a Container Registry. The integration of connecting the pre-deployment stage to the runtime stage is the CI/CD integration. The next step is the runtime piece or the container registry which is a repository that can connect directly to a container orchestration platform including Kubernetes. Kubernetes allows for the automation of deployment, scaling, and management of containerized applications.

 

Understanding threats and vulnerabilities in the Build and Ship Stage

IaC is code that organizations use to deploy services in a repeatable preconfigured manner, e.g., a server configured a specific way, containers that reside in certain locations, databases sized a specific way, etc. One common vector that adversaries are attacking is the Infrastructure as Code. If attackers can compromise the code that deploys the infrastructure, then they have the capability to introduce new vulnerabilities.

 

Source code security has come to forefront of security because it is the start of the CI/CD process where threats and vulnerabilities can be inserted. An example of this type of attack was “Apache Struts”.  This was a free, open-source framework for creating Java web applications. It had a vulnerability that allowed for remote code execution. Apache Struts was a widely used framework and resulted in many organizations being vulnerable to this type of attack. 

 

Additional areas of concern include developer code that has secrets or credentials that are publicly exposed. Leaving secrets or credentials in code can result in malicious activity such as SQL injections to be successful.

 

How Do We Protect Against Build Threats?

Implementing protections in the CI/CD pipeline can be challenging and will require organizational changes. The security team needs to be more involved in the earliest stages of application development, including at the building stage. The industry term for this process is referred to as DevSecOps or Shift-Lift. Securing source code requires looking at potential vulnerabilities and whether secrets are exposed in the code could allow malicious activity and validating that safe dependencies are being used. This is where tools such as GitHub’s Advanced Security piece become a vital part of the overall development process.

 

GitHub’s Advanced security services include a “Dependabot” which looks for vulnerable dependencies or malware in repos. Also included is a tool called “Secret Scanning” that scans repositories looking for known types of secrets, then it alerts developers or block usage of the secrets to prevent fraudulent use. GitHub Advanced Security is included with GitHub Enterprise.

 

Understanding Deployment and Runtime Threats

Threats in Kubernetes environments can come from six primary areas:

1. Compromised accounts: in cases where Kubernetes clusters are deployed in public clouds (e.g., AKS in Azure or GKE in GCP), compromised cloud credentials can lead to cluster takeover as attackers who have access to account credentials can get access to the cluster’s management layer.
2. Vulnerable or mis-configured images: images that are not updated regularly may contain vulnerabilities that can be exploited in malicious attacks
3. Environment misconfigurations: an attacker with access to the Kubernetes API, either through an exposed Kubeflow dashboard or lack of appropriate authentication/authorization controls, can completely take down the server, deploy malicious containers, or hijack your entire cluster.
4. App level attacks: applications can be exploited through several types of “typical methods”, such as SQL injection, cross-site scription, and remote file inclusion.
5. Node level attacks: since containers run on host machines, called nodes, those machines must not run vulnerable code or software that would allow for an attacker to gain initial access.
6. Unauthorized traffic: insecure networking between the different containers within the cluster and between the pods and outside world can be subject to malicious traffic if not secured.

The various threats that Kubernetes environments are subject can be found in Figure 2 below:

 

Figure 2. Kubernetes Threats

 

Securing Container Deployment and Runtime

Container deployment refers to the phase of the lifecycle where container images are pulled from the static container registry to be run on virtual machines hosts. During deployment, you should ensure you are following the best security practices listed here:

• Ensure containers are immutable: prevent patches to running containers whenever possible. As a best practice, if you notice a running container needs updates, you should rebuild the image and deploy the new container. Introducing new code in running containers can introduce new vulnerabilities.
• Leverage Admission Controllers: configure policies to prevent containers from being deployed from untrusted registries, from running with root OS (Operating System) privileges, and from utilizing too many resources in the event of a denial-of-service attack. These can be enforced with Azure Policy Add-On for Kubernetes.
• Gate deployments of vulnerable images: ensure that the containers being deployed are free of vulnerabilities and misconfigurations by running a vulnerability scan in the Build and Ship phases. Any image with high or critical severity vulnerabilities should be blocked from deployment.

Container runtime refers to the phase of the lifecycle where containers are running on the virtual hosts. During runtime, the following best security practices are recommended:

• Gain visibility into the container and worker nodes: monitor each node, pod, and containers during runtime for any sort of anomalous or malicious activity that may be occurring.
• Identify vulnerabilities in running containers: monitor your running containers for any new vulnerabilities that may have been introduced during runtime.
• Look for malicious API calls and unusual logins: have a monitoring system in place to identify any abnormal Kubernetes API server requests for malicious activity.
• Define mitigation steps in the event of an attack: have processes in place to isolate containers on different networks, stop malicious containers, and block suspicious activity.

 

How Defender for Containers Can Help

Understanding the attack surface of containerized environments is the first step of building security solutions for these environments. Microsoft Defender for Cloud can help you protect your containers environment from build to deployment to runtime with the best security practices listed in this article.

 

Microsoft Defender for Containers is a plan within Microsoft Defender for Cloud that offers a cloud-native solution to secure your containers so you can improve, monitor, and maintain the security of your clusters, containers, and their applications whether they are running in Azure Kubernetes Service, Amazon Elastic Kubernetes Service, Google Kubernetes Engine, or any other Cloud Native Computing Foundation (CNCF) certified cluster. This plan helps you achieve the core aspects of security within the build, ship, deployment, and runtime phases of the container lifecycle:

 

Build/Ship

 

• Control-plane hardening: Defender for Containers continuously assesses your clusters and presents you with recommendations as to how you can remediate misconfigurations in your environment. For example, Defender for Containers will display any Kubernetes clusters that are not using Role-Based Access Control and any Container registries that allow for unrestricted network access.
• Vulnerability assessment: Defender for Containers offers vulnerability scanning for images in Azure Container Registries that triggers the scan when images are pushed to a registry, pulled from the registry, and imported into the registry. 

Deployment/Runtime

 

• Data-plane hardening: Defender for Containers includes an admission controller via the Azure Policy for Kubernetes component that extends the open-source Gatekeeper v3 admission controller webhook for Open Policy Agent. This add-on monitors every request to the Kubernetes API server against a predefined set of best practices to help you ensure your workloads are secure. As an example, you can enforce container CPU and memory limits to protect against DDoS attacks and prevent containers from running as root user.
• Run-time monitoring and threat protection: Defender for Containers provides real-time threat protection and generates alerts for suspicious activities. Threat protection at the cluster level is provided by the Defender agent and analysis of the Kubernetes audit logs. Examples of some of the 60+ Kubernetes-aware analytics, AI, and anomaly detections for your runtime workloads include abnormal Kubernetes service account operations, digital currency mining behavior, and anomalous pod deployments. For a list of all detection, visit this document. Another benefit of this monitoring is that the Defender agents allows Defender for Containers to also scan images that are currently running within a cluster by correlating the inventory of your running containers to the images stored within the registry and presenting you with the vulnerability assessment results.

For a single view and deep visibility into the protections that the Defender for Containers provides, you can leverage the new Containers Security Mapping Dashboard.

 

More Information

To learn more about how Defender for Containers can help you achieve container security with zero vulnerabilities from build to runtime, please leverage the following sources:

• Microsoft Docs: Container security with Microsoft Defender for Cloud | Microsoft Docs
• Defender for Cloud in the field: Microsoft Defender for Containers | Microsoft Docs
• Webinar: Demystifying Microsoft Defender for Containers | Microsoft Defender for Cloud Webinar - YouTube
• Blogs: Microsoft launches dedicated Container protection plan, How to demonstrate the new containers features in Microsoft Defender for Cloud - Microsoft Tech Community