Azure Sentinel triggers incident when it shouldn't

%3CLINGO-SUB%20id%3D%22lingo-sub-2735675%22%20slang%3D%22en-US%22%3EAzure%20Sentinel%20triggers%20incident%20when%20it%20shouldn't%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2735675%22%20slang%3D%22en-US%22%3E%3CP%3EGreetings%2C%20I%20just%20ran%20into%20something%20interesting.%20I%20have%20created%20a%20analytics%20rule%20that%20looks%20like%20this%3A%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CPRE%20class%3D%22lia-code-sample%20language-cpp%22%3E%3CCODE%3Elet%20exceptionUsers%20%3D%20IdentityInfo%0A%7C%20where%20TimeGenerated%20%26gt%3B%20ago(22d)%20%2F%2FIdentityInfo%20refreshes%20its%20information%20every%2021%20days%0A%7C%20where%20todynamic(GroupMembership)%20contains%20%22SG-U%20Guest%20users%20excluded%20from%20CA%20blocked%20countries%22%0A%7C%20distinct%20MailAddress%3B%0A%2F%2FCreates%20a%20set%20of%20users%20that%20is%20to%20be%20ignored%20when%20looking%20for%20logins%20outside%20of%20europe.%20%0ASigninLogs%0A%7C%20where%20TimeGenerated%20%26gt%3B%20ago(4h)%0A%7C%20where%20Location%20!in%20(%20%22AL%22%2C%22AD%22%2C%22AM%22%2C%22AT%22%2C%22BY%22%2C%22BE%22%2C%22BA%22%2C%22BG%22%2C%22CH%22%2C%22CY%22%2C%22CZ%22%2C%22DE%22%2C%22DK%22%2C%22EE%22%2C%22ES%22%2C%22FO%22%2C%22FI%22%2C%22FR%22%2C%22GB%22%2C%22GE%22%2C%22GI%22%2C%22GR%22%2C%22HU%22%2C%22HR%22%2C%22IE%22%2C%22IS%22%2C%22IT%22%2C%22LI%22%2C%22LT%22%2C%22LU%22%2C%22LV%22%2C%22MC%22%2C%22MK%22%2C%22MT%22%2C%22NO%22%2C%22NL%22%2C%22PL%22%2C%22PT%22%2C%22RO%22%2C%22RU%22%2C%22SE%22%2C%22SI%22%2C%22SK%22%2C%22SM%22%2C%22TR%22%2C%22UA%22%2C%22VA%22%2C%22SJ%22%2C%22%22)%20%2F%2F%20List%20of%20country%20codes%20in%20europe.%20%0A%7C%20where%20UserPrincipalName%20!in%20(%20exceptionUsers%20)%0A%7C%20extend%20AccountCustomEntity%20%3D%20Identity%0A%7C%20extend%20IPCustomEntity%20%3D%20IPAddress%3C%2FCODE%3E%3C%2FPRE%3E%3CP%3EMight%20not%20be%20the%20greatest%20of%20queries%2C%20but%20still%2C%20I%20run%20this%20query%20and%20get%20no%20results.%20As%20i%20expect.%20However%2C%20the%20analytics%20rule%20with%20this%20configuration%20still%20manages%20to%20trigger.%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThis%20is%20the%20view%20from%20the%20analytics%20rule%20wizard%20when%20i%20test%20with%20current%20data.%3C%2FP%3E%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22stianhoydal_0-1631180987240.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F309207iD7C3C490ACCF02E4%2Fimage-size%2Fmedium%3Fv%3Dv2%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22stianhoydal_0-1631180987240.png%22%20alt%3D%22stianhoydal_0-1631180987240.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%3CP%3EThe%20last%20spike%20indicates%20the%20one%20i%20saw%20today.%20How%20can%20the%20analytics%20rule%20wizard%20get%20different%20results%20from%20the%20same%20query%20i%20run%20in%20the%20Logs%20tab%3F%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2736107%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20Sentinel%20triggers%20incident%20when%20it%20shouldn't%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2736107%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F819982%22%20target%3D%22_blank%22%3E%40stianhoydal%3C%2FA%3E%26nbsp%3BThe%20analytic%20rule%20will%20ignore%20any%20time%20details%20set%20in%20the%20query.%26nbsp%3B%20As%20it%20states%20in%20the%20%3CSTRONG%3ESet%20rule%20logic%3C%2FSTRONG%3E%20tab%3A%26nbsp%3B%26nbsp%3B%3CSPAN%3EAny%20time%20details%20set%20here%20will%20be%20within%20the%20scope%20defined%20below%20in%20the%20Query%20scheduling%20fields.%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThat%20may%20have%20something%20to%20do%20with%20it.%26nbsp%3B%20Try%20running%20the%20query%20manually%20but%20use%20the%20value%20that%20is%20set%20In%20the%20%3CSTRONG%3ELookup%20data%20from%20the%20last%3C%2FSTRONG%3E%20field%20and%20see%20if%20that%20returns%20any%20results.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2736169%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20Sentinel%20triggers%20incident%20when%20it%20shouldn't%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2736169%22%20slang%3D%22en-US%22%3EI%20see%2C%20that%20probably%20explains%20why%20the%20users%20that%20should%20be%20excluded%20shows%20up%20anyway.%20Since%20the%20IdentityInfo%20table%20is%20only%20updated%20every%2021%20days%20i%20seem%20to%20be%20unable%20to%20get%20this%20information%20as%20queries%20against%20the%20table%20sometimes%20return%20empty%20if%20the%20users%20in%20question%20haven't%20been%20updates%20within%20the%20timeframe.%20Seeing%20as%20the%20lookback%20time%20you%20can%20set%20in%20the%20query%20wizard%20is%20max%2014%20days%20i%20need%20to%20figure%20out%20a%20workaround%20i%20suppose.%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2739817%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20Sentinel%20triggers%20incident%20when%20it%20shouldn't%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2739817%22%20slang%3D%22en-US%22%3E%3CP%3ESo%20i%20figured%20out%20a%20simple%20workaround%2C%20but%20still%20the%20query%20wizard%20shows%20that%20it%20would%20trigger%20the%20alarm%20several%20times%20although%20it%20shouldn't%20have.%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CPRE%20class%3D%22lia-code-sample%20language-cpp%22%3E%3CCODE%3Elet%20excludedUsers%20%3D%20GuestAccountsExcludedFromCAPolicy_CL%20%0A%7C%20distinct%20UserEmail_s%3B%0ASigninLogs%20%0A%7C%20where%20Location%20!in%20(%20%22AL%22%2C%22AD%22%2C%22AM%22%2C%22AT%22%2C%22BY%22%2C%22BE%22%2C%22BA%22%2C%22BG%22%2C%22CH%22%2C%22CY%22%2C%22CZ%22%2C%22DE%22%2C%22DK%22%2C%22EE%22%2C%22ES%22%2C%22FO%22%2C%22FI%22%2C%22FR%22%2C%22GB%22%2C%22GE%22%2C%22GI%22%2C%22GR%22%2C%22HU%22%2C%22HR%22%2C%22IE%22%2C%22IS%22%2C%22IT%22%2C%22LI%22%2C%22LT%22%2C%22LU%22%2C%22LV%22%2C%22MC%22%2C%22MK%22%2C%22MT%22%2C%22NO%22%2C%22NL%22%2C%22PL%22%2C%22PT%22%2C%22RO%22%2C%22RU%22%2C%22SE%22%2C%22SI%22%2C%22SK%22%2C%22SM%22%2C%22TR%22%2C%22UA%22%2C%22VA%22%2C%22SJ%22%2C%22%22)%20%2F%2F%20List%20of%20country%20codes%20in%20europe.%20%0A%7C%20where%20UserPrincipalName%20!in%20(excludedUsers)%0A%7C%20extend%20AccountCustomEntity%20%3D%20Identity%0A%7C%20extend%20IPCustomEntity%20%3D%20IPAddress%3C%2FCODE%3E%3C%2FPRE%3E%3CP%3EThe%20GuestAccountsExcludedFromCAPolicy_CL%20is%20simply%20a%20table%20filled%20with%20users%20fetched%20from%20AAD%20via%20logic%20apps.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EStill%20the%20query%20wizard%20shows%20that%20it%20would%20trigged%20multiple%20alarms%20within%20the%20last%2048%20hours%20although%20there%20should%20only%20be%20one.%26nbsp%3B%3C%2FP%3E%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22stianhoydal_0-1631279840851.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F309607iCB6F582C9916D562%2Fimage-size%2Fmedium%3Fv%3Dv2%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22stianhoydal_0-1631279840851.png%22%20alt%3D%22stianhoydal_0-1631279840851.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%3CP%3EIt%20seems%20to%20me%20as%20if%20the%20query%20is%20just%20ignoring%20the%20line%3C%2FP%3E%3CP%3E%7C%20where%20UserPrincipalName%20!in%20(excludedUsers)%3C%2FP%3E%3CP%3Ebecause%20it%20would%20be%20correct%20otherwise%2C%20but%20the%20whole%20point%20is%20to%20not%20get%20alerted%20when%20one%20of%20the%20excluded%20members%20tries%20to%20log%20on.%26nbsp%3B%3C%2FP%3E%3CP%3EAnyone%20have%20any%20ideas%20on%20why%20this%20is%20happening%2C%20or%20potential%20solutions%3F%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2741123%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20Sentinel%20triggers%20incident%20when%20it%20shouldn't%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2741123%22%20slang%3D%22en-US%22%3EIt%20looks%20right.%20I%20would%20double%20check%20the%20values%20you%20are%20getting%20in%20your%20custom%20tables%20to%20make%20sure%20they%20are%20matching%20what%20you%20are%20seeing%20in%20the%20SigninLogs.%3CBR%20%2F%3E%3CBR%20%2F%3EYou%20may%20also%20want%20to%20use%20a%20Watchlist%20for%20the%20locations%20to%20make%20it%20easier%20to%20keep%20up%20to%20date.%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2748827%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20Sentinel%20triggers%20incident%20when%20it%20shouldn't%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2748827%22%20slang%3D%22en-US%22%3EFor%20anyone%20else%20that%20might%20have%20been%20wondering%2C%20seemingly%20the%20best%20way%20i%20found%20to%20make%20this%20work%20is%20to%20fetch%20the%20AAD%20group%20members%20into%20a%20custom%20table%20and%20update%20this%20according%20to%20how%20often%20you%20would%20want%20to%20run%20the%20analytics%20rule%20since%20the%20analytics%20rule%20wizard%20overrides%20any%20time%20references%20made%20in%20a%20query.%20If%20i%20want%20the%20query%20to%20run%20every%201%20hour%20with%20the%20latest%201%20hour%20of%20data%20i%20would%20need%20to%20update%20the%20custom%20table%20every%201%20hour%20or%20less.%3C%2FLINGO-BODY%3E
Contributor

Greetings, I just ran into something interesting. I have created a analytics rule that looks like this:

 

let exceptionUsers = IdentityInfo
| where TimeGenerated > ago(22d) //IdentityInfo refreshes its information every 21 days
| where todynamic(GroupMembership) contains "SG-U Guest users excluded from CA blocked countries"
| distinct MailAddress;
//Creates a set of users that is to be ignored when looking for logins outside of europe. 
SigninLogs
| where TimeGenerated > ago(4h)
| where Location !in ( "AL","AD","AM","AT","BY","BE","BA","BG","CH","CY","CZ","DE","DK","EE","ES","FO","FI","FR","GB","GE","GI","GR","HU","HR","IE","IS","IT","LI","LT","LU","LV","MC","MK","MT","NO","NL","PL","PT","RO","RU","SE","SI","SK","SM","TR","UA","VA","SJ","") // List of country codes in europe. 
| where UserPrincipalName !in ( exceptionUsers )
| extend AccountCustomEntity = Identity
| extend IPCustomEntity = IPAddress

Might not be the greatest of queries, but still, I run this query and get no results. As i expect. However, the analytics rule with this configuration still manages to trigger. 

 

This is the view from the analytics rule wizard when i test with current data.

stianhoydal_0-1631180987240.png

The last spike indicates the one i saw today. How can the analytics rule wizard get different results from the same query i run in the Logs tab?

5 Replies

@stianhoydal The analytic rule will ignore any time details set in the query.  As it states in the Set rule logic tab:  Any time details set here will be within the scope defined below in the Query scheduling fields.

 

That may have something to do with it.  Try running the query manually but use the value that is set In the Lookup data from the last field and see if that returns any results.

I see, that probably explains why the users that should be excluded shows up anyway. Since the IdentityInfo table is only updated every 21 days i seem to be unable to get this information as queries against the table sometimes return empty if the users in question haven't been updates within the timeframe. Seeing as the lookback time you can set in the query wizard is max 14 days i need to figure out a workaround i suppose.

So i figured out a simple workaround, but still the query wizard shows that it would trigger the alarm several times although it shouldn't have. 

 

let excludedUsers = GuestAccountsExcludedFromCAPolicy_CL 
| distinct UserEmail_s;
SigninLogs 
| where Location !in ( "AL","AD","AM","AT","BY","BE","BA","BG","CH","CY","CZ","DE","DK","EE","ES","FO","FI","FR","GB","GE","GI","GR","HU","HR","IE","IS","IT","LI","LT","LU","LV","MC","MK","MT","NO","NL","PL","PT","RO","RU","SE","SI","SK","SM","TR","UA","VA","SJ","") // List of country codes in europe. 
| where UserPrincipalName !in (excludedUsers)
| extend AccountCustomEntity = Identity
| extend IPCustomEntity = IPAddress

The GuestAccountsExcludedFromCAPolicy_CL is simply a table filled with users fetched from AAD via logic apps.

 

Still the query wizard shows that it would trigged multiple alarms within the last 48 hours although there should only be one. 

stianhoydal_0-1631279840851.png

It seems to me as if the query is just ignoring the line

| where UserPrincipalName !in (excludedUsers)

because it would be correct otherwise, but the whole point is to not get alerted when one of the excluded members tries to log on. 

Anyone have any ideas on why this is happening, or potential solutions? 

 

It looks right. I would double check the values you are getting in your custom tables to make sure they are matching what you are seeing in the SigninLogs.

You may also want to use a Watchlist for the locations to make it easier to keep up to date.
For anyone else that might have been wondering, seemingly the best way i found to make this work is to fetch the AAD group members into a custom table and update this according to how often you would want to run the analytics rule since the analytics rule wizard overrides any time references made in a query. If i want the query to run every 1 hour with the latest 1 hour of data i would need to update the custom table every 1 hour or less.