Ingesting logs from Event Hub

[deniskutin]

Hey guys,

I wanted to give a try to Sentinel. But there is one thing I'd like to clarify before. 

Our current ingestion pipeline: we are receiving logs into Event Hubs (EH), read them by Logstash and put them into Elastic. According to this article [1] we just need to change (add) the destination as Logstash output and route logs into Log Analytics (LA). And we are good to go.

This is what confuses me: EH and LA, both are located in Azure and I hoped to remove Logstash completely from the design: EH -> LA -> Sentinel. Is it possible? Did I miss something here? Or, maybe it is planned in some future?

[1]: https://techcommunity.microsoft.com/t5/azure-sentinel/azure-sentinel-to-go-part1-a-lab-w-prerecorded...

[#2]

@deniskutin 

It depends on the data sources you want to send to Log Analytics & Azure Sentinel, IaaS (Azure or hybrid) devices will need an agent, either the Microsoft Management Agent(MMA) or Logstash - you decide which you prefer.  You can log forward with Linux / Logstash as well.   
Data that you send to an EventHub today, if that comes from Azure, you are typically sending from the Diagnostics settings of your resource (SQL DB etc...), and each resource diagnostic blade normally has alternative options to send to Storage or Log Analytics - so you just need to re-map those resources.  

Example from the Public IP resource, you can check Log Analytics instead as well as Event Hub, or just the one you need.

[#3]

Hey @CliveWatson 

Thank you for your answer. 

Sorry about that: didn't cover this point in my question.

We have a big deal of agents we are gathering logs from. Ex., clients' endpoints (Win, Linux, Mac), network devices (via syslog), Azure Insights, 3-rd party tools integrations (reading files), info from SPAN (raw network data). All of them come into EH. Than the pipeline I mentioned before happens.

Not all these log source are possible to send directly to LA, unfortunately. And it would be a big task to re-engineer agent's infrastructure we are using now. 

So, yeah, my question more about Azure EH and LA integration (not directly Sentinel related): is it possible to route (copy) data from EH to LA to use in Sentinel afterward? But was hoping someone in the community faced this task.

[#4 · Accepted solution]

@deniskutin 

You might get another response from the Log Analytics members, but I'd start by looking at a Logic App  https://docs.microsoft.com/en-us/azure/connectors/connectors-create-api-azure-event-hubs or maybe an Azure Function https://docs.microsoft.com/en-us/azure/azure-functions/functions-bindings-event-hubs-trigger?tabs=cs... to do this