Azure AD: sign-in risk calculation is wrong?

%3CLINGO-SUB%20id%3D%22lingo-sub-2438576%22%20slang%3D%22en-US%22%3EAzure%20AD%3A%20sign-in%20risk%20calculation%20is%20wrong%3F%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2438576%22%20slang%3D%22en-US%22%3E%3CP%3EHi%20all%2C%26nbsp%3B%3CBR%20%2F%3E%3CBR%20%2F%3EI%20noticed%20these%20past%20weeks%20some%20weird%20logic%20in%20the%20way%20sign-in%20risk%20is%20calculated%20or%20handled.%26nbsp%3B%3CBR%20%2F%3EIt%20especially%20impactful%20when%20MFA%20is%20enforced%20via%20an%20CA%20policy%20that%20is%20based%20on%20the%20users%20sign-in%20risk.%3CBR%20%2F%3E%3CBR%20%2F%3E%3C%2FP%3E%3CP%3E%3CSTRONG%3ESome%20examples%3A%26nbsp%3B%3C%2FSTRONG%3E%3CBR%20%2F%3E%3CBR%20%2F%3E%3CU%3E%3CEM%3EExample%201%3A%3C%2FEM%3E%3C%2FU%3E%3C%2FP%3E%3CP%3EUser%20is%20suddenly%20active%20from%20Russia.%26nbsp%3B%3CBR%20%2F%3ENo%20registered%20device%3CBR%20%2F%3EFirst%20time%20sign-in%20this%20month%20in%20Russia%3CBR%20%2F%3Eno%20exclusions%20in%20the%20policy.%26nbsp%3B%3CBR%20%2F%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%222021-06-11%2010_32_40.png%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F288014i75A4435AC09D5EDD%2Fimage-size%2Flarge%3Fv%3Dv2%26amp%3Bpx%3D999%22%20role%3D%22button%22%20title%3D%222021-06-11%2010_32_40.png%22%20alt%3D%222021-06-11%2010_32_40.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%3CP%3EOn%20the%20interrupted%20sign-in%20there%20are%20no%20CA%20policies%20applied.%26nbsp%3B%3CBR%20%2F%3EThe%20next%20sing-in%20Russia%20on%20office365%20shell%20has%20has%20a%20sign-in%20risk%20of%26nbsp%3B%3CSTRONG%3Enone%3C%2FSTRONG%3E%3F%26nbsp%3B%3CBR%20%2F%3E%3CBR%20%2F%3E%3CEM%3EWhy%20do%20I%20find%20this%3C%2FEM%3E%20weird%3F%3CBR%20%2F%3EUser%20activity%20moves%20from%20Paris%20to%20Russia.%3CBR%20%2F%3EThe%20sign-in%20with%20the%20interrupt%20does%20not%20lower%20the%20risk%20as%20it%20has%20been%20interrupted.%3CBR%20%2F%3ESo%20the%20next%20sign-in%20in%20Russia%20should%20still%20have%20a%20risk%20factor%20as%20there%20has%20not%20been%20a%20completed%20mfa%20request.%3CBR%20%2F%3E%3CBR%20%2F%3E%3CU%3E%3CEM%3EExample%202%3A%26nbsp%3B%3CBR%20%2F%3E%3C%2FEM%3E%3C%2FU%3Euser%20is%20suddenly%20in%20Italy.%26nbsp%3B%3CBR%20%2F%3EAgain%20no%20previous%20history%20in%20Italy.%26nbsp%3B%3CBR%20%2F%3ENo%20azure%20ad%20joined%20device.%26nbsp%3B%3CBR%20%2F%3ESign-in%20risk%20was%20considered%26nbsp%3B%3CSTRONG%3Enone%3CBR%20%2F%3E%3C%2FSTRONG%3E%3C%2FP%3E%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%222021-06-11%2010_52_%20Microsoft%E2%80%8B%20Edg.png%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F288017i6915DF461948137B%2Fimage-size%2Flarge%3Fv%3Dv2%26amp%3Bpx%3D999%22%20role%3D%22button%22%20title%3D%222021-06-11%2010_52_%20Microsoft%E2%80%8B%20Edg.png%22%20alt%3D%222021-06-11%2010_52_%20Microsoft%20Edg.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%3CBR%20%2F%3E%3CEM%3EWhy%20do%20I%20find%20this%3C%2FEM%3E%20weird%3F%3CBR%20%2F%3EAgain%20a%20user%20moved%20to%20a%20country%20where%20haven't%20seen%20any%20activity%20in%20the%20last%20month.%26nbsp%3B%3CBR%20%2F%3EThere%20is%20no%20registered%20device%20in%20any%20sign-in%20log.%26nbsp%3B%3CBR%20%2F%3E%3CBR%20%2F%3E%3CU%3E%3CEM%3EExample%203%3A%3CBR%20%2F%3E%3C%2FEM%3E%3C%2FU%3EUser%20is%20suddenly%20active%20from%20Tunisia.%26nbsp%3B%3CBR%20%2F%3ENormal%20activity%20is%20France.%26nbsp%3B%3CBR%20%2F%3EThe%20user%20first%20fails%202%20times%20to%20sign-in%20as%20his%20primary%20authentication%20due%20to%20wrong%20password.%26nbsp%3B%3CBR%20%2F%3E3rd%20sign-in%20log%20he%20is%20interrupted%20as%20his%20device%20requires%20authentication.%26nbsp%3B%3CBR%20%2F%3E4rd%20sign-in%20log%20he%26nbsp%3B%3CSTRONG%3EFAILS%26nbsp%3B%3C%2FSTRONG%3Eon%20completing%20the%20MFA%20request%3CBR%20%2F%3E5th%20sign-in%20log%20the%20user%20signs-in%20using%20with%20%3CSTRONG%3Enone%3C%2FSTRONG%3E%20sign-in%20risk.%26nbsp%3B%3C%2FP%3E%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%222021-06-11%2011_01_03png.png%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F288034iA7D6F0054D990A22%2Fimage-size%2Flarge%3Fv%3Dv2%26amp%3Bpx%3D999%22%20role%3D%22button%22%20title%3D%222021-06-11%2011_01_03png.png%22%20alt%3D%222021-06-11%2011_01_03png.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%3CEM%3EScreenshot%20of%20the%20failed%20mfa%20request%3C%2FEM%3E%3C%2FP%3E%3CP%3E%3CEM%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%222021-06-11%2011_16_36-%20Microsoft%E2%80%8B%20Edge.png%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F288037i1CC670C65FD67538%2Fimage-size%2Flarge%3Fv%3Dv2%26amp%3Bpx%3D999%22%20role%3D%22button%22%20title%3D%222021-06-11%2011_16_36-%20Microsoft%E2%80%8B%20Edge.png%22%20alt%3D%222021-06-11%2011_16_36-%20Microsoft%20Edge.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FEM%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%3CEM%3EScreenshot%20of%20the%20successful%26nbsp%3Bsign-in%20his%20risk%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%222021-06-11%2011_08_55-PMicrosoft%E2%80%8B%20Edge.png%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F288036i355BAB00ED515B3F%2Fimage-size%2Flarge%3Fv%3Dv2%26amp%3Bpx%3D999%22%20role%3D%22button%22%20title%3D%222021-06-11%2011_08_55-PMicrosoft%E2%80%8B%20Edge.png%22%20alt%3D%222021-06-11%2011_08_55-PMicrosoft%20Edge.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FEM%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%3CEM%3EWhy%20do%20I%20find%20this%20weird%3F%3C%2FEM%3E%3C%2FP%3E%3CP%3EThis%20is%20the%20one%20that%20blows%20my%20mind%20the%20most.%3CBR%20%2F%3EUsers%20go's%20to%20new%20country.%3CBR%20%2F%3EFails%202%20times%20on%20his%20password.%3CBR%20%2F%3EThen%20gets%20prompted%20with%20mfa%20but%20fails%20to%20complete%20authentication.%26nbsp%3B%3CBR%20%2F%3ELogicaly%20thinking%20you%20would%20thing%20that%20this%20would%20raise%20his%20sign-in%20risk.%26nbsp%3B%3CBR%20%2F%3E2%20failures%20and%201%20mfa%20prompt%20not%20completed.%26nbsp%3B%3CBR%20%2F%3ENext%20sign-in%20%3D%200%20sign-in%20risk.%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EAs%20documented%20Microsoft%20states%20the%20following%20about%20risk%20calculations%3A%26nbsp%3B%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%222021-06-11%2011_23_48-What%20is%20risk_%20Azure%20AD%20Identity%20Protection%20_%20Microsoft%20Docs%20and%2011%20more%20pages%20-%20.png%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F288041iF41A930453B89ACE%2Fimage-size%2Flarge%3Fv%3Dv2%26amp%3Bpx%3D999%22%20role%3D%22button%22%20title%3D%222021-06-11%2011_23_48-What%20is%20risk_%20Azure%20AD%20Identity%20Protection%20_%20Microsoft%20Docs%20and%2011%20more%20pages%20-%20.png%22%20alt%3D%222021-06-11%2011_23_48-What%20is%20risk_%20Azure%20AD%20Identity%20Protection%20_%20Microsoft%20Docs%20and%2011%20more%20pages%20-%20.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%3CP%3EBut%20I%20believe%20my%203%20scenario's%20have%20enough%20evidence%20that%20the%20risk%20state%20of%20these%20sign-ins%20should%20not%20have%20the%20state%20of%20none.%20Especially%20when%20previous%20attempt%20are%20failed.%26nbsp%3B%3CBR%20%2F%3E%3CBR%20%2F%3E%3C%2FP%3E%3CP%3EHaving%20insights%20on%20how%20this%20%22risk%22%20is%20calculated%20would%20bring%20us%20a%20step%20further.%26nbsp%3B%3C%2FP%3E%3CP%3E%3CBR%20%2F%3EAnyone%20who%20can%20share%20an%20opinion%2Finsights%20on%20this%3F%26nbsp%3B%3CBR%20%2F%3E%3CBR%20%2F%3EKind%20regards%3C%2FP%3E%3CP%3ELouis%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-2438576%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EAzure%20Active%20Directory%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EConditional%20Access%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3ESecurity%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2452350%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20AD%3A%20sign-in%20risk%20calculation%20is%20wrong%3F%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2452350%22%20slang%3D%22en-US%22%3E%3CP%3E%3CSTRONG%3Eupdate%3A%3C%2FSTRONG%3E%3C%2FP%3E%3CP%3EToday%20I%20have%20another%20case%20of%20what%20is%20according%20to%20me%20wrong%20risk%20calculation%20or%20risk%20loss.%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%222021-06-16%2009_10_38-Microsoft%E2%80%8B%20Edge.png%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F289060iA1E4FE37525050D9%2Fimage-size%2Flarge%3Fv%3Dv2%26amp%3Bpx%3D999%22%20role%3D%22button%22%20title%3D%222021-06-16%2009_10_38-Microsoft%E2%80%8B%20Edge.png%22%20alt%3D%222021-06-16%2009_10_38-Microsoft%20Edge.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E1)%20the%20user%20signs-in%20on%20apple%20internet%20accounts%2C%20is%20interrupted%20and%20needs%20to%20confirm%20mfa.%3CBR%20%2F%3EBased%20on%20the%20logs%20the%20does%20not%20complete%20the%20mfa%20request.%3C%2FP%3E%3CP%3E2)%20the%20next%20sign-in%20log%20to%20apple%20internet%20accounts%20is%20a%20success.%3CBR%20%2F%3ESign-in%20risk%20is%20suddenly%20%22none%22.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EDetails%20of%20sign-in%20log%201%3CBR%20%2F%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%222021-06-16%2009_12_36-details%201%20Microsoft%E2%80%8B%20Edge.png%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F289061i60D0EBE82A7E78FD%2Fimage-size%2Flarge%3Fv%3Dv2%26amp%3Bpx%3D999%22%20role%3D%22button%22%20title%3D%222021-06-16%2009_12_36-details%201%20Microsoft%E2%80%8B%20Edge.png%22%20alt%3D%222021-06-16%2009_12_36-details%201%20Microsoft%20Edge.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EDetails%20of%20sign-in%20log%202%3CBR%20%2F%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%222021-06-16%2009_14_18-Policy%20details%20%20-%20Microsoft%E2%80%8B%20Edge.png%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F289062i04F2F97AAE4B68E3%2Fimage-size%2Flarge%3Fv%3Dv2%26amp%3Bpx%3D999%22%20role%3D%22button%22%20title%3D%222021-06-16%2009_14_18-Policy%20details%20%20-%20Microsoft%E2%80%8B%20Edge.png%22%20alt%3D%222021-06-16%2009_14_18-Policy%20details%20-%20Microsoft%20Edge.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EIt%20does%20not%20make%20sense%20to%20me%20that%20the%20user%20loses%20his%20risk%20after%20a%20failed%20mfa%20request%20according%20to%20the%20logs.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E
Occasional Contributor

Hi all, 

I noticed these past weeks some weird logic in the way sign-in risk is calculated or handled. 
It especially impactful when MFA is enforced via an CA policy that is based on the users sign-in risk.

Some examples: 

Example 1:

User is suddenly active from Russia. 
No registered device
First time sign-in this month in Russia
no exclusions in the policy. 
2021-06-11 10_32_40.png

On the interrupted sign-in there are no CA policies applied. 
The next sing-in Russia on office365 shell has has a sign-in risk of none

Why do I find this weird?
User activity moves from Paris to Russia.
The sign-in with the interrupt does not lower the risk as it has been interrupted.
So the next sign-in in Russia should still have a risk factor as there has not been a completed mfa request.

Example 2: 
user is suddenly in Italy. 
Again no previous history in Italy. 
No azure ad joined device. 
Sign-in risk was considered none

2021-06-11 10_52_ Microsoft​ Edg.png


Why do I find this weird?
Again a user moved to a country where haven't seen any activity in the last month. 
There is no registered device in any sign-in log. 

Example 3:
User is suddenly active from Tunisia. 
Normal activity is France. 
The user first fails 2 times to sign-in as his primary authentication due to wrong password. 
3rd sign-in log he is interrupted as his device requires authentication. 
4rd sign-in log he FAILS on completing the MFA request
5th sign-in log the user signs-in using with none sign-in risk. 

2021-06-11 11_01_03png.png

Screenshot of the failed mfa request

2021-06-11 11_16_36- Microsoft​ Edge.png

 

Screenshot of the successful sign-in his risk2021-06-11 11_08_55-PMicrosoft​ Edge.png

 

Why do I find this weird?

This is the one that blows my mind the most.
Users go's to new country.
Fails 2 times on his password.
Then gets prompted with mfa but fails to complete authentication. 
Logicaly thinking you would thing that this would raise his sign-in risk. 
2 failures and 1 mfa prompt not completed. 
Next sign-in = 0 sign-in risk. 

 

As documented Microsoft states the following about risk calculations: 2021-06-11 11_23_48-What is risk_ Azure AD Identity Protection _ Microsoft Docs and 11 more pages - .png

But I believe my 3 scenario's have enough evidence that the risk state of these sign-ins should not have the state of none. Especially when previous attempt are failed. 

Having insights on how this "risk" is calculated would bring us a step further. 


Anyone who can share an opinion/insights on this? 

Kind regards

Louis

 

 

1 Reply

update:

Today I have another case of what is according to me wrong risk calculation or risk loss. 

 

2021-06-16 09_10_38-Microsoft​ Edge.png

 

1) the user signs-in on apple internet accounts, is interrupted and needs to confirm mfa.
Based on the logs the does not complete the mfa request.

2) the next sign-in log to apple internet accounts is a success.
Sign-in risk is suddenly "none".

 

Details of sign-in log 1
2021-06-16 09_12_36-details 1 Microsoft​ Edge.png

 

Details of sign-in log 2
2021-06-16 09_14_18-Policy details  - Microsoft​ Edge.png

 

 

It does not make sense to me that the user loses his risk after a failed mfa request according to the logs.