Issue: Microsoft Sentinel doesn’t receive data from the Amazon Web Services S3 connector or one of its data types
The logs for the AWS S3 connector (or one of its data types) aren’t visible in the Microsoft Sentinel workspace for more than 30 minutes after the connector was connected.
It can take around 20-30 minutes from the moment the connector is connected until data is ingested into the workspace.
The connector's connection status indicates that a collection rule exists; it doesn't indicate that data was ingested. If the status of the Amazon Web Services S3 connector is green, this means that there is a collection rule for one of the data types, but there is still no data.
In this section, we cover these causes:
The AWS S3 connector permissions policies are not set properly.
The data is not ingested to the S3 bucket in AWS.
The SQS (Amazon Simple Queue Service) in the AWS cloud does not receive notifications from the S3 bucket.
The data cannot be read from the SQS/S3 in the AWS cloud. With GuardDuty logs, the issue is usually caused by wrong KMS permissions.
Review the next sections to identify and solve common issues.
Check that the AWS S3 connector permissions policies are set properly
Check that the relevant data exists in the S3 bucket
In AWS, open the S3 bucket, search for the relevant folder according to the required logs, and check if there are any logs inside the folder:
If the data does not exist, there’s an issue with the AWS configuration. Configure an AWS service to export logs to an S3 bucket.
Verify that the data arrived from S3 to the SQS:
In AWS, open the relevant SQS.
In the Monitoring tab, you should see traffic in the Number Of Messages Sent widget.
If there is no traffic in the SQS, there is an AWS configuration problem:
Make sure that the event notifications definition for the SQS includes the correct data filters (prefix and suffix). To see the event notifications, in the S3 bucket, select the Properties tab, and locate the Event notifications section. If you can’t see this section, create it. Make sure that the SQS has the relevant policies to get the data from the S3 bucket. The SQS must contain this policy in the Access policy tab.
Verify that data is read from the SQS
In AWS, open the relevant SQS.
In the Monitoring tab, you should see traffic in the Number Of Messages Deleted and Number Of Messages Received widgets.
Note that one spike of data is not enough, as shown in the screenshot below. Wait until there is enough data, as shown in the screenshot above, and then check for issues.
2. If at least one of the widgets is empty, check the health logs by running this query:
Issue: Data from the AWS S3 connector (or one of its data types) is seen in Microsoft Sentinel with a delay of more than 30 minutes
This issue usually happens because Microsoft can’t read files in the S3 folder, either because the files are encrypted, or the files are in the wrong format. In these cases, many retries eventually cause ingestion delay.
Check if the logs are fully or partially encrypted by the Key Management Service (KMS). In this case, make sure that Microsoft Sentinel has permission for this KMS to decrypt the files. Review the required KMS permissions for the GuardDuty and CloudTrail logs.
To verify that the events notification from S3 to the SQS is defined properly, check that:
The notification is defined from the specific folder that includes the logs, and not from the main folder that contains the bucket.
The notification is defined with the .gz suffix.
3. Verify that there are no errors in the health logs by running this query:
| where TimeGenerated between (ago(startTime)..ago(endTime))
| where SentinelResourceKind == "AmazonWebServicesS3"
| where Status != "Success"
| distinct TimeGenerated, OperationName, SentinelResourceName, Status, Description
4. Make sure that the health feature is enabled:
| take 20
5. If the health feature isn’t enabled, enable it.
Note: thank you Danielle Ohfeld Zats and Sara Gamzu for co-writing this blog