Determine events per second for a potential Sentinel deployment

%3CLINGO-SUB%20id%3D%22lingo-sub-2629328%22%20slang%3D%22en-US%22%3EDetermine%20events%20per%20second%20for%20a%20potential%20Sentinel%20deployment%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2629328%22%20slang%3D%22en-US%22%3E%3CP%3EI%20have%20been%20tasked%20to%20determine%20the%20estimate%20of%20EPS%20(events%20per%20second)%20for%204%20subscriptions.%26nbsp%3B%20Need%20to%20get%20an%20idea%20of%20the%20cost%20of%20creating%20an%20Event%20Hub%20to%20send%20data%20to%20the%20SIEM.%26nbsp%3B%20Any%20assistance%2Fguidance%20would%20be%20appreciated.%26nbsp%3B%20I%20was%20trying%20to%20use%20Monitor%20%26gt%3B%20Metrics%20but%20you%20have%20to%20drill%20down%20to%20a%20specific%20resource%20and%20I%20was%20hoping%20to%20get%20a%20general%20query%20per%20subscription.%26nbsp%3B%26nbsp%3B%3C%2FP%3E%3CP%3EPlease%20advise%2C%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3ESerge%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2632555%22%20slang%3D%22en-US%22%3ERe%3A%20Determine%20events%20per%20second%20for%20a%20potential%20Sentinel%20deployment%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2632555%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F616520%22%20target%3D%22_blank%22%3E%40SergioT1228%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EIf%20you%20have%20the%20data%20in%20a%20workspace%20already%2C%20you%20can%20query%20that%20for%20EPS%2C%20you%20may%20need%20to%20add%20a%20filter%2C%20something%20like%20this%20(not%20all%20tables%20store%20SubscriptionId%20though!)%3CBR%20%2F%3E%3CBR%20%2F%3E%7C%20where%20SubscriptionId%20%3D%3D%20%22%26lt%3B%20sub%20id%26gt%3B%22%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-cpp%22%3E%3CCODE%3Eunion%20withsource%3D_TableName1%20*%0A%7C%20where%20_TimeReceived%20%20%26gt%3B%20ago(1d)%0A%7C%20summarize%20count()%20%20by%20bin(_TimeReceived%2C%201m)%2C%20Type%0A%7C%20extend%20counttemp%20%3Dcount_%20%2F%2060%0A%7C%20summarize%20%0A%20%20%20%20%20%20%20%20%20%20%20%5B'Average%20Events%20per%20Second%20(eps)'%5D%20%3D%20avg(counttemp)%2C%20%5B'Minimum%20eps'%5D%3Dmin%20(counttemp)%2C%0A%20%20%20%20%20%20%20%20%20%20%20%5B'Maximum%20eps'%5D%3Dmax(counttemp)%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2633578%22%20slang%3D%22en-US%22%3ERe%3A%20Determine%20events%20per%20second%20for%20a%20potential%20Sentinel%20deployment%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2633578%22%20slang%3D%22en-US%22%3E%3CP%3EHello%20Clive%2C%20Thank%20you%20for%20your%20reply.%20I'm%20new%20to%20gathering%20data%20from%20Azure.%20I%20mainly%20deal%20with%20ATP%20deployments%20and%20making%20sure%20to%20get%20all%20endpoints%20covered%20by%20Defender.%20I%20have%20been%20asked%20to%20help%20determine%20the%20EPS%20for%20some%20subscriptions.%20I%20have%20a%20couple%20of%20questions%20regarding%20your%20statement.%20I%20understand%20the%20need%20to%20specify%20which%20subscription.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EUnder%20Monitor%20%26gt%3B%20Logs%2C%20I%20have%20selected%20the%20scope%20to%20be%20a%20specific%20subscription.%20As%20far%20as%20withsource%20%3D%20_TableName1%2C%20which%20table%20are%20you%20referring%3F%20I%20AzureMetrics%3F%20Diagnostics%3F%20Activity%3F%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3ESorry%20if%20this%20should%20be%20obvious%20but%20I'm%20just%20getting%20started%20on%20learning%20how%20to%20obtain%20logs%2Fdata%20from%20Azure.%20I%20did%20do%20a%20count%20for%20the%20Three%20tables%20I%20saw%2C%20are%20those%20counts%20worth%20anything%3F%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26lt%3B%3C%2FP%3E%3CP%3EAzureActivity%20%7C%20where%20SubscriptionId%20%3D%3D%20%22subscriptionId%22%20%7C%20count%3C%2FP%3E%3CP%3E%2F%26gt%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2633824%22%20slang%3D%22en-US%22%3ERe%3A%20Determine%20events%20per%20second%20for%20a%20potential%20Sentinel%20deployment%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2633824%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F239477%22%20target%3D%22_blank%22%3E%40Clive%20Watson%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EHey%20Clive%2C%3C%2FP%3E%3CP%3EOk%2C%20I%20think%20I%20figured%20it%20out.%20the%20_TableName1%20is%20a%20way%20to%20run%20through%20all%20tables%20without%20naming%20a%20specific%20table%20which%20allows%20you%20to%20search%20all%20Tables%20available.%3CBR%20%2F%3Ealso%2C%20after%20reviewing%20the%20TimeReceived%20information%20in%20this%20table%3A%3CBR%20%2F%3E%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fazure-monitor%2Flogs%2Fdata-ingestion-time%23checking-ingestion-time%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3Ehttps%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fazure-monitor%2Flogs%2Fdata-ingestion-time%23checking-ingestion-time%3C%2FA%3E%3CBR%20%2F%3EI%20was%20able%20to%20substitute%20as%20needed.%20I%20think%20I%20got%20the%20needed%20information.%20Thank%20you%20again.%3C%2FP%3E%3C%2FLINGO-BODY%3E
Occasional Contributor

I have been tasked to determine the estimate of EPS (events per second) for 4 subscriptions.  Need to get an idea of the cost of creating an Event Hub to send data to the SIEM.  Any assistance/guidance would be appreciated.  I was trying to use Monitor > Metrics but you have to drill down to a specific resource and I was hoping to get a general query per subscription.  

Please advise,

 

Serge

5 Replies

@SergioT1228 

 

If you have the data in a workspace already, you can query that for EPS, you may need to add a filter, something like this (not all tables store SubscriptionId though!)

| where SubscriptionId == "< sub id>" 

 

union withsource=_TableName1 *
| where _TimeReceived  > ago(1d)
| summarize count()  by bin(_TimeReceived, 1m), Type
| extend counttemp =count_ / 60
| summarize 
           ['Average Events per Second (eps)'] = avg(counttemp), ['Minimum eps']=min (counttemp),
           ['Maximum eps']=max(counttemp)

 

Hello Clive, Thank you for your reply. I'm new to gathering data from Azure. I mainly deal with ATP deployments and making sure to get all endpoints covered by Defender. I have been asked to help determine the EPS for some subscriptions. I have a couple of questions regarding your statement. I understand the need to specify which subscription.

 

Under Monitor > Logs, I have selected the scope to be a specific subscription. As far as withsource = _TableName1, which table are you referring? I AzureMetrics? Diagnostics? Activity?

 

Sorry if this should be obvious but I'm just getting started on learning how to obtain logs/data from Azure. I did do a count for the Three tables I saw, are those counts worth anything?

 

<

AzureActivity | where SubscriptionId == "subscriptionId" | count

/>

@Clive Watson 

 

Hey Clive,

Ok, I think I figured it out. the _TableName1 is a way to run through all tables without naming a specific table which allows you to search all Tables available.
also, after reviewing the TimeReceived information in this table:
https://docs.microsoft.com/en-us/azure/azure-monitor/logs/data-ingestion-time#checking-ingestion-tim...
I was able to substitute as needed. I think I got the needed information. Thank you again.

@snteran 

 

I'm glad you figured it out.  You can also do a similar thing in M365 - in "Advanced Hunting".  Rather than union you can name the single Table or event use union to wildcard ie. 

union withsource =MDTables Device*

 

union withsource=MDTables *
| where Timestamp  > ago(1d)
| summarize count() by bin(Timestamp, 1m), MDTables
| extend EPS = count_ /60
| summarize avg(EPS) by MDTables 
| sort by avg_EPS desc

// Also show as GBytes (estimated, using 500bytes as a default value)


let bytes_ = 500;
union withsource=MDTables *
| where Timestamp  > ago(1d)
| summarize count() by bin(Timestamp, 1m), MDTables
| extend EPS = count_ /60
| summarize avg(EPS), estimatedGBytes = (avg(EPS) * bytes_) / (1024*1024*1024) by MDTables 
| sort by toint(estimatedGBytes) desc

 

@Clive Watson 

 

That worked perfect.  I also added "by" statement to get the logs per table:

 

union withsource=_TableName1 *

| where TimeGenerated > ago(1d)

| summarize count() by bin(TimeGenerated, 1m), Type

| extend counttemp =count_ / 60

| summarize

['Average Events per Second (eps)'] = avg(counttemp), ['Minimum eps']=min (counttemp),

['Maximum eps']=max(counttemp)

by ['Table Name']=Type
 
It gave me the table names, hopefully this is correct.  A lot to learn.  Hopefully sharing this will help others.
 
Cheers,