Enhance the ingestion of AWS CloudWatch logs into Microsoft Sentinel with AWS Lambda

Microsoft Sentinel has recently made its AWS S3 data connector generally available (GA), offering users the capability to ingest logs from various AWS services such as CloudTrail, CloudWatch, VPCFlow Logs, and CloudWatch into Microsoft Sentinel using an S3 bucket and AWS's simple message queuing service.

To learn more about the AWS S3 connector, see Connect Microsoft Sentinel to Amazon Web Services to ingest AWS service log data | Microsoft Learn

AWS CloudWatch is a powerful monitoring and observability service offered by Amazon Web Services (AWS) that plays a crucial role in managing the health, performance, and security of AWS resources and applications. Ingesting AWS CloudWatch logs to Microsoft Sentinel is crucial for enhancing the security posture of an organization's cloud infrastructure.

In this blog post, we'll explore the advantages offered by an AWS Lambda function designed to enhance the ingestion of AWS CloudWatch logs into Microsoft Sentinel.

Challenges

1. AWS CloudWatch logs are events in JSON format that record the actions of your AWS resources. However, they do not have the same format as the Azure Log Analytics format. Customers need to run AWS Lambda in their own AWS environment to forward  and reformat their AWS CloudWatch logs to a central AWS S3 Bucket.
2. If your organization operates across multiple AWS Accounts with distinct LogGroup and StreamNames configurations, it becomes necessary to reformat the logs and transmit them to a centralized AWS S3 Bucket.

Solution

To streamline this process, we are offering an AWS Lambda function capable of accessing all AWS accounts to reformat CloudWatch events and seamlessly export them to an AWS S3 bucket, ready to be ingested by Sentinel's AWS S3 Data connector.

Key benefits:

1. Extract all CloudWatch logs from all the AWS Accounts, reformats them and exports them to an AWS S3 bucket to be ingested by Sentinel’s AWS S3 Data connector
2. Provides detailed error messages, making it easier to identify and troubleshoot any unexpected errors are caught and logged, allowing for a more streamlined debugging process

 Deployment:

• Open the Lambda console.
• Choose Create function.
• Configure the following settings:
  • Function name: Enter a name for the function.
  • Runtime: Choose Python 3.12
• Select Permissions, and under Execution role, select Role name.
• Under Permissions policies, select Add permissions > Attach policies.
• 
• Search for the AmazonS3FullAccess and CloudWatchLogsReadOnlyAccess policies and attach them.
• 
• Copy the Python code from https://github.com/Azure/Azure-Sentinel/blob/master/DataConnectors/AWS-S3/CloudWatchLambdaFunction_V2.py
• Deploy the function
• Click on the lambda function name you provided, click Environment Variables and add the following.
  • BUCKET_NAME
  • BUCKET_PREFIX
  • START_TIME_UTC
  • END_TIME_UTC
• Update Lambda Memory and TimeOut
•          

Conclusion:

By implementing the AWS Lambda function for exporting AWS CloudWatch logs enable users to seamlessly align their log management strategy with Microsoft Sentinel. Leveraging Microsoft Sentinel's robust support for Multi Cloud providers, along with a versatile and well-structured log export solution, empowers organizations to maintain a competitive edge in the rapidly evolving landscape of cloud computing.

Try it out and let us know what you think! If you run into any issues, please create issue/PR in Azure Sentinel GitHub Repo.

We hope you find this article useful. Please leave us your feedback and questions in the comments section.

Special thanks to @Javier Soriano @Yael Bergman for reviewing and providing feedback on article.