Announcing Improved Identity Protection Signal Quality and Visibility

Howdy folks, 

 

I’m excited to share our recent improvements in risk evaluation and reporting visibility for Identity Protection. These changes are a step forward in our ability to detect emerging attack vectors and help you focus on the most critical alerts. We improved signal quality and reduced alert volume for low-risk sign-ins by more than 60%, introduced unfamiliar sign-in properties for session cookies, and added visibility into non-interactive risky sign-ins. 

 

Sarah Handler, Senior Program Manager, and Feifan Jian, Data Scientist, both from our Identity Security team, will take you through these improvements and the data science behind the scenes. 

 

Best Regards, 

Alex Simons (@Alex_A_Simons) 

Corporate Vice President Program Management 

Microsoft Identity Division 

 

------------------------------------------------------

 

Hi everyone – 

 

We’re excited to share with you how these changes can better protect your organization’s identities and improve your investigative experience! Identity-based attacks have evolved and expanded over the last year. In response, our team has expanded our detection surface area and improved our systems to ensure that we’re surfacing high fidelity alerts— so that you can focus on what matters most.  

 

First, we expanded where we detect unfamiliar sign-in properties to include non-interactive sign-ins. Unfamiliar sign-in properties evaluates in real-time the amount a user’s current sign-in deviates from the user’s past sign-in behavior. This detection was previously available for interactive sign-ins, but now we also evaluate session cookies. For most tenants, this will not lead to a significant increase in unfamiliar sign-in properties detections; but non-interactive sign-ins that do get flagged for unfamiliar sign-in properties deserve increased scrutiny due to the possibility of a token replay attack.  

 

You can see these non-interactive unfamiliar sign-in properties detections in the Risk detections report and in the Risky sign-ins report, which were updated to support non-interactive sign-ins. The Risky sign-ins report now defaults to showing you both interactive and non-interactive risky sign-ins. You can toggle this using the “sign-in type” filter.  

 

 

Additionally, we have significantly improved the signal-to-noise ratio for low-risk risky sign-ins. We heard your feedback that for many organizations there were simply too many low-risk sign-ins to investigate. We want your admins and security professionals to focus on the most important detections and to trust the fidelity of our signal, so we tuned our detections and have reduced the number of low-risk Risky sign-ins by more than 60% while also significantly improving precision!  

 

Let’s hear from Fefian Jian, the data scientist behind these changes, on how we did it. 

 

Under the hood: the data science behind our changes 

Identity Protection’s detection systems run both in real-time (during authentication) and offline (post authentication) to understand whether sign-ins and users are compromised. Our offline machine learning model, which runs post authentication, scores sign-ins with different features and algorithms to determine whether a sign-in was compromised. The output of the model is the aggregate sign-in risk level, which represents our most recent evaluation of whether that sign-in was compromised.  

 

We made a change to our offline machine learning model to improve its accuracy, allowing us to reduce the noise for low-risk risky sign-ins. Since this change, the volume of sign-ins with low aggregate risk dropped by more than 60%, and the precision, which means the quality of alerts, improved by 100%. This means you will get fewer, but higher quality, low-risk risky sign-ins in your environment! 

 

Use these improved features today! 

These improvements have automatically rolled out in Identity Protection, and you can start using these improved features today! To best protect your environment and benefit from our risk evaluation, make sure you also set up conditional access policies to automatically mitigate risky sign-ins and risky users in your organization. To learn more, read how to configure risk policies. 

 

Stay secure! 

 

Sarah Handler (@sarahhandler) 

Senior Program Manager  

Microsoft Identity Security and Protection Team  

 

Feifan Jian 

Data Scientist II 

Microsoft Identity Security and Protection Team  

 

 

Learn more about Microsoft identity:

• Return to the Azure Active Directory Identity blog home
• Join the conversation on Twitter and LinkedIn
• Share product suggestions on the Azure Feedback Forum