A guide to configure and use Sentinel Connectors to collect logs and data from your Kubernetes clusters and GitHub CI/CD pipelines.
Part 2 of 3 part series about security monitoring of your Kubernetes Clusters and CI/CD pipelines by @singhabhi and @Umesh_Nagdev
Link to Part 1
Link to Part 3
Introduction
In part 1 of this series, we discussed the type of log sources you should consider for monitoring the security of your Kubernetes environment. This blog will demonstrate how to connect some of the critical log sources using Sentinel Data Connectors.
Sentinel Data Connectors are a set of tools that enable you to collect and analyze logs and data from various sources, such as cloud services, applications, devices, and networks. Sentinel Data Connectors can help you monitor the health, performance, and security of your Kubernetes clusters and GitHub CI/CD pipelines, as well as detect and respond to threats and incidents.
In this document, we will show you how to set up Sentinel Data Connectors for three types of sources: Kubernetes clusters, GitHub CI/CD pipelines, and Defender for Containers alerts and Defender for Cloud recommendations. We will also explain how to use the connectors to view and query the collected data in Sentinel.
Security monitoring use cases
Let’s first highlight some security risks you would want to monitor with Sentinel:
1. Pod Security Monitoring:
- Log source: Defender of Containers
- Risks monitored:
- Detect unauthorized or suspicious pods running in the cluster.
- Monitor for privilege escalation attempts within pods.
- Track and alert on changes to pod security policies.
2. Network Security Monitoring:
- Log source: Defender of Containers
- Risks monitored:
- Identify and alert on unexpected network traffic patterns.
- Monitor for unauthorized ingress and egress traffic.
- Detect and investigate potential denial-of-service (DoS) attacks.
3. Container Image Security:
- Log source: Defender for Cloud - Defender Cloud Security Posture Management (DCSPM)
- Risks monitored:
- Scan container images for vulnerabilities before deployment.
- Monitor for unauthorized or unsigned images.
- Track changes to container image repositories.
4. Kubelet Activity Monitoring:
- Log source: Defender of Containers
- Risks monitored:
- Monitor kubelet logs for signs of compromise or unauthorized access.
- Detect abnormal activities related to node management.
5. API Server Security:
- Log source: Defender of Containers
- Risks monitored:
- Monitor Kubernetes API server logs for suspicious activities.
- Track and alert on failed authentication attempts.
- Detect unusual API server request patterns.
6. RBAC (Role-Based Access Control) Monitoring:
- Log source: AKS Diagnostics Logs, Azure AD logs, Azure Monitor Container Insights
- Risks monitored:
- Monitor changes to RBAC policies and roles.
- Detect and alert on unauthorized access attempts.
- Track role binding changes and escalations.
7. Secrets and ConfigMap Access Monitoring:
- Log source: Defender of Containers
- Risks monitored:
- Monitor for unauthorized access to Kubernetes secrets and ConfigMaps.
- Detect changes to sensitive configuration data.
- Track usage patterns of sensitive information.
8. Audit Logging:
- Log source: AKS Diagnostic Logs
- Risks monitored:
- Enable and monitor Kubernetes audit logs for cluster-wide activities.
- Correlate audit logs to identify security events and policy violations.
- Regularly review audit logs for anomalies and potential threats.
9. Compliance Monitoring:
- Log source: Defender for Cloud - Defender Cloud Security Posture Management (DCSPM)
- Risks monitored:
- Ensure compliance with security standards and policies.
- Monitor for deviations from security best practices.
- Generate reports on compliance status and potential risks.
10. Container Runtime Security:
- Log source: Defender of Containers
- Risks monitored:
- Monitor runtime activities of containers for abnormal behavior.
- Detect and alert on suspicious system calls within containers.
- Integrate with container runtime security tools for enhanced monitoring.
11. Incident Response and Forensics:
- Log source: Defender of Containers
- Risks monitored:
- Develop and test incident response plans for Kubernetes security incidents.
- Monitor for indicators of compromise (IoCs) and initiate investigations in Sentinel
- Collect and analyze forensics data in the event of a security incident in Sentinel
12. Cluster Health Monitoring:
- Log source: AKS Diagnostic Logs
- Risks monitored:
- Regularly monitor the overall health of the Kubernetes cluster.
- Detect and alert on abnormal resource consumption or performance issues.
- Ensure the availability of critical components and services.
Prerequisites
Before you can set up Sentinel Data Connectors, you need to have the following:
- Sentinel workspace. This is where you store and analyze the data collected by the connectors. Enable Sentinel on the Log Analytics Workspace where you are exporting all of the below mentioned log sources . Instructions on how to setup Sentinel
- Kubernetes cluster. This is the source of the data for the Kubernetes Cluster using Diagnostics logs. You can use any Kubernetes cluster that supports the Kubernetes API, such as Azure Kubernetes Service (AKS), Google Kubernetes Engine (GKE), or Amazon Elastic Kubernetes Service (EKS). We will showcase this with AKS. Instructions on how to deploy AKS
- GitHub account. This is the source of the code and manifests used for creating container images which are then deployed in your Kubernetes Clusters. You will need to configure DCSPM DevOps security for secure scanning of artifacts. Or if you are using a third-party scanning tool you will need to send the scan results to Sentinel.
- Container Registry. The images stored in the registry need to be scanned for vulnerabilities. You will need access to the scan logs this can be done via Defender for Cloud DCSPM
- Defender for Containers subscription. This is a service that provides security and compliance monitoring for your Kubernetes clusters. You need to enable Defender for Containers on your subscription where your Kubernetes cluster is located and configure it to send alerts to the Sentinel workspace. Instructions on how to enable Defender for Containers on a subscription.
- A Defender for Cloud DSPM subscription. This is a service that provides security and compliance recommendations for your cloud resources such as AKS, ACR, and Azure tenant. You need to enable Defender for Cloud DCSPM on your subscription with AKS cluster and configure it to send recommendations to the Sentinel workspace. Instructions on how to enable DCSPM on a subscription.
How to Set Up Kubernetes Cluster Connector
The Kubernetes Cluster connector allows you to collect logs and metrics from your Kubernetes cluster, such as cluster events, pod logs, node metrics, and container metrics. To ingest AKS logs into Sentinel, deploy the Azure Kubernetes Solution for Sentinel then, follow the steps below to enable the AKS data connector.
Configure AKS data connector to ingest logs into Sentinel:
- In Microsoft Sentinel, go to the "Data connectors" page.
- Find and configure the "Azure Kubernetes Service (AKS)" connector.
- Launch the Azure Policy wizard under configuration to enable logging.
Verify Integration:
- After configuration, verify that logs from your AKS cluster are flowing into Sentinel.
Create Sentinel Workbooks and Queries (to be elaborated in part 3):
- Leverage Microsoft Sentinel workbooks and Kusto Query Language (KQL) queries to create visualizations and reports based on AKS logs.
- Customize the workbooks and queries based on your specific security and monitoring requirements.
Set Up Alerts and Incidents (to be elaborated in part 3):
- Configure alerts within Microsoft Sentinel based on specific events or patterns detected in AKS logs.
- Set up incidents and response workflows to investigate and respond to security events.
Monitor and Fine-Tune:
- Regularly monitor the integration, alerts, and logs to ensure that the AKS logs are being properly processed in MicrosoftSentinel.
- Fine-tune your configurations based on feedback, new security requirements, or changes to your AKS environment.
How to Set Up GitHub connector
To ingest logs into Sentinel, deploy the Microsoft Sentinel - Continuous Threat Monitoring for GitHub. Enable the two connector that are installed as part of this solution:
- GitHub Enterprise Audit Log connector: this connector collects GitHub audit logs which tracks changes to repository, user added/removed, pull request activities, etc.
- GitHub (using Webhooks) connector: to ingest you can ingest the scan data using a built in data connector for GitHub events. This connector can pull events related to code scanning alert, repository vulnerability alert (via Dependabot) and Secret Scanning Alert.
How to Set Up Defender for Containers Alerts and Defender for Cloud Connector
Sentinel has a buil-in data connector to ingest Defender for Cloud alerts and recommendation. You can find the details https://learn.microsoft.com/en-us/azure/sentinel/connect-defender-for-cloud#connect-to-microsoft-def...
Setting up AKS data connector and additional logging for Sentinel
Setup the Diagnostic Settings for the Azure Kubernetes Services to send the events to a Sentinel-enabled Log Analytics workspace. https://learn.microsoft.com/en-us/azure/aks/monitor-aks#aks-control-planeresource-logs.
In our scenario we are using the following logs
|
In addition, you will also need to enable Container Insights to get the Pod level data so you can run the search queries for risk related to Pod specifics like pods running in Default namespace.
You can refer to this https://learn.microsoft.com/en-us/azure/azure-monitor/containers/kubernetes-monitoring-enable?tabs=c... resource to enable Container Insights. The logs will go to ContainerLogsV2 in Log Analytics Workspace https://learn.microsoft.com/en-us/azure/azure-monitor/containers/container-insights-logs-schema#enab...
The following pic shows the ContainerLogv2 Schema as an example.
|
You will need to for to Sentinel Content Hub and enable the following. This will give you Workbook, several hunting queries and a data connector to ingest AKS data.
|
Your AKS cluster will populate the data in the following tables, which we will use to write custom search queries in the section below.
|