Ingest CEF logs in CommonSecurityLog with Logstasth

[vincenthoag]

Hello
We are migrating to Sentinel from Splunk. For the log ingestion we are using Native Data Connectors where we can and Logstash with the microsoft-sentinel-log-analytics-logstash-output-plugin for the rest. The reason behind Logstash choice is that AMA only has a 10 GB buffer size which is too small for our need in a case of a connection drop.
I am working on getting logs in a CEF format to logstash and then to the CommonSecurityLog table.

I have been following instructions from this page to ingest log in Syslog format with Logstash : https://learn.microsoft.com/en-us/azure/sentinel/connect-logstash-data-connection-rules
I was able to ingest logs in a Custom Table doing so but I want now to ingest the data in the CommonSecurityLog table.
 I have changed the DCR rule accordingly but I only see the entries without the data or parsing :

The modified DCR rule is :

{
    "properties": {
        "immutableId": "dcr-1efc2494a966f2fc95f730e22",
        "dataCollectionEndpointId": "/subscriptions/xxxxxxxxxxxx/resourceGroups/rg_cybersecurity_sentinel_prod/providers/Microsoft.Insights/dataCollectionEndpoints/xxxxxxxx",
        "streamDeclarations": {
            "Custom-test_table_to_delete_CL": {
                "columns": [
                    {
                        "name": "message",
                        "type": "string"
                    },
                    {
                        "name": "event",
                        "type": "dynamic"
                    },
                    {
                        "name": "ls_timestamp",
                        "type": "datetime"
                    },
                    {
                        "name": "ls_version",
                        "type": "string"
                    }
                ]
            }
        },
        "dataSources": {},
        "destinations": {
            "logAnalytics": [
                {
                    "workspaceResourceId": "/subscriptions/xxxxxxx/resourcegroups/xxxxxxxxx/providers/microsoft.operationalinsights/workspaces/xxxxxxxxxxx",
                    "workspaceId": "558b9a62-adf5-4a0b-957e-e04d82719877",
                    "name": "558b9a62adf54a0b957ee04d82719877"
                }
            ]
        },
        "dataFlows": [
            {
                "streams": [
                    "Custom-test_table_to_delete_CL"
                ],
                "destinations": [
                    "558b9a62adf54a0b957ee04d82719877"
                ],
                "transformKql": "source | project-away event, ls_timestamp, ls_version | project-rename CEF=message | extend TimeGenerated = todatetime(now())",
                "outputStream": "Microsoft-CommonSecurityLog"
            }
        ],
        "provisioningState": "Succeeded"
    },
    "location": "westus2",
    "id": "/subscriptions/xxxxxxxx/resourceGroups/xxxxxx/providers/Microsoft.Insights/dataCollectionRules/DCR_logs_ingestion",
    "name": "DCR_logs_ingestion",
    "type": "Microsoft.Insights/dataCollectionRules",
    "etag": "\"d1075091-0000-0800-0000-655509870000\"",
    "systemData": {
        "createdBy": "xxxxxxx",
        "createdByType": "User",
        "createdAt": "2023-11-08T20:46:49.8781378Z",
        "lastModifiedBy": "xxxxxx",
        "lastModifiedByType": "User",
        "lastModifiedAt": "2023-11-15T18:10:14.0501104Z"
    }
}

Am I missing something ? Is this even possible ?

Thank you for your help

[#2]

Hey @vincenthoag 

Have you tried to rename the streams to "commonsecuritylog" by chance

				"dataFlows": [
					{
					"streams": [
						"Custom-SyslogStream"
					],
					"destinations": [
						"clv2ws1"

[#3]

Thank you for your answer I am working on a different configuration. I am using logstash to collect the logs and play a buffer role then send encrypted logs to a VMSS with AMA in Azure that will parse the logs and send them to sentinel.
I am still working on the terraform file, I'll post it when my config is working.
Thank you for your help.