Forum Discussion

vincenthoag's avatar
vincenthoag
Copper Contributor
Nov 15, 2023

Ingest CEF logs in CommonSecurityLog with Logstasth

Hello
We are migrating to Sentinel from Splunk. For the log ingestion we are using Native Data Connectors where we can and Logstash with the microsoft-sentinel-log-analytics-logstash-output-plugin for the rest. The reason behind Logstash choice is that AMA only has a 10 GB buffer size which is too small for our need in a case of a connection drop.
I am working on getting logs in a CEF format to logstash and then to the CommonSecurityLog table.

I have been following instructions from this page to ingest log in Syslog format with Logstash : https://learn.microsoft.com/en-us/azure/sentinel/connect-logstash-data-connection-rules
I was able to ingest logs in a Custom Table doing so but I want now to ingest the data in the CommonSecurityLog table.
 I have changed the DCR rule accordingly but I only see the entries without the data or parsing :

The modified DCR rule is :

 

{
    "properties": {
        "immutableId": "dcr-1efc2494a966f2fc95f730e22",
        "dataCollectionEndpointId": "/subscriptions/xxxxxxxxxxxx/resourceGroups/rg_cybersecurity_sentinel_prod/providers/Microsoft.Insights/dataCollectionEndpoints/xxxxxxxx",
        "streamDeclarations": {
            "Custom-test_table_to_delete_CL": {
                "columns": [
                    {
                        "name": "message",
                        "type": "string"
                    },
                    {
                        "name": "event",
                        "type": "dynamic"
                    },
                    {
                        "name": "ls_timestamp",
                        "type": "datetime"
                    },
                    {
                        "name": "ls_version",
                        "type": "string"
                    }
                ]
            }
        },
        "dataSources": {},
        "destinations": {
            "logAnalytics": [
                {
                    "workspaceResourceId": "/subscriptions/xxxxxxx/resourcegroups/xxxxxxxxx/providers/microsoft.operationalinsights/workspaces/xxxxxxxxxxx",
                    "workspaceId": "558b9a62-adf5-4a0b-957e-e04d82719877",
                    "name": "558b9a62adf54a0b957ee04d82719877"
                }
            ]
        },
        "dataFlows": [
            {
                "streams": [
                    "Custom-test_table_to_delete_CL"
                ],
                "destinations": [
                    "558b9a62adf54a0b957ee04d82719877"
                ],
                "transformKql": "source | project-away event, ls_timestamp, ls_version | project-rename CEF=message | extend TimeGenerated = todatetime(now())",
                "outputStream": "Microsoft-CommonSecurityLog"
            }
        ],
        "provisioningState": "Succeeded"
    },
    "location": "westus2",
    "id": "/subscriptions/xxxxxxxx/resourceGroups/xxxxxx/providers/Microsoft.Insights/dataCollectionRules/DCR_logs_ingestion",
    "name": "DCR_logs_ingestion",
    "type": "Microsoft.Insights/dataCollectionRules",
    "etag": "\"d1075091-0000-0800-0000-655509870000\"",
    "systemData": {
        "createdBy": "xxxxxxx",
        "createdByType": "User",
        "createdAt": "2023-11-08T20:46:49.8781378Z",
        "lastModifiedBy": "xxxxxx",
        "lastModifiedByType": "User",
        "lastModifiedAt": "2023-11-15T18:10:14.0501104Z"
    }
}

 


Am I missing something ? Is this even possible ?

Thank you for your help

data collection
KQL
monitoring
siem

2 Replies

  • BillClarksonAntill's avatar
    BillClarksonAntill
    Iron Contributor
    Dec 04, 2023

    Hey vincenthoag 

     

    Have you tried to rename the streams to "commonsecuritylog" by chance

     

    				"dataFlows": [
					{
					"streams": [
						"Custom-SyslogStream"
					],
					"destinations": [
						"clv2ws1"

     

    • vincenthoag's avatar
      vincenthoag
      Copper Contributor
      Dec 08, 2023
      Thank you for your answer I am working on a different configuration. I am using logstash to collect the logs and play a buffer role then send encrypted logs to a VMSS with AMA in Azure that will parse the logs and send them to sentinel.
      I am still working on the terraform file, I'll post it when my config is working.
      Thank you for your help.

Resources