Cloud App Security IP block in Conjunction with Azure AD Conditional Access Policy

[Jim Hill]

I have a conditional access policy which rejects Office 365 logins from IP's probably located outside of the US (and Bahamas, Canada).  I still see alerts in Cloud App Security when foreign hackers attempt to log into various Office 365 accounts from those regions. We have MFA on all admin accounts and most others as well.  Question, why does Cloud App Security flag those login attempts when we already have a conditional access policy blocking those regions?  Is there some kind of ordering that happens with these rules?   I notice that when I block the IP (make it Risky, a conditional access policy also blocks all risky IP logins), the attack goes away until they try another IP.  

[#2]

@Jim Hill 

Can you elaborate on the alerts you are seeing in Cloud App Security? Is it one of the anomaly detection alerts such as 'Risky Sign in', 'Activity from anonymous IP address', or 'Multiple failed login attempts'? Or is this an access policy you have in place in MCAS that corresponds to your Azure AD Conditional Access Policy?  

[#3]

@Anisha Gupta we have the Cloud App Security set to alert only on the rule which fires when it sees multiple failed login attempts. This usually come from outside of our region, so I thought that any login attempt would first be blocked in Azure AD by having a conditional access policy blocking any login from outside of our region.  I am guessing that the conditional access policy allows the user outside of the region to attempt to login, but just blocks it at that point, so it then shows in the Cloud App Security alert.  Once we add that IP address as a risky IP it is blocked thereafter.   

[#4]

@Anisha Gupta  I think I see what was happening.  I had only a subset of users to which the conditional access policy "block login from risky IP's."  Once I expanded that rule I see that by using the What If tool that the login attempt was blocked.  Regardless, my users know to reject and report any incident during which they see an MFA authentication request on their smart phone apps since that would mean that the login passed the password authentication portion. We also have branding all over our sign in page so hopefully between that, the various rules, and Bitdefender we hope to minimize breaches.  Thanks for looking at this. 

[#5]

#Microsoft #CloudAppSecurity 

What's happening here is users are paying for a premium fee for a feature 'Cloud Security App' where Microsoft is failing to provide service to its own dashboard. It is like going through your Windows logs on Event viewer. 

• It also gives you alerts from Microsoft server
  i.e. Your file was accessed by Teams on US Microsoft Server (eeehhh!)
• It gives you false alerts from failed sign-in.
  Is it really a CloudAppSecurity alert? No, you cannot fix it (even if you mark it as false +ve)
  i.e. Conditional access is blocking the sign-in attempt and that information is logged in AAD user sign-in alerts. 
• Similary 'Top users to investigate' score is wrong. As all the attempts are false +ve's, hence dropping your compliance score.

Concluding that a user will have a hard time monitoring the logs or will switch to another solution. 
The ideal solution though would be to have option to upload a list of suspecious IP's, where user get brute force attacks to be blocked from attempting to login. 

Edit - Issue is noted since 2019 and yet the same till date. Good luck!

[#6]

@kkalra  Yep, that is my thinking too. The CA policy actually is fired only after a hacker makes their breach and they would then be prevented from access at that point. I am wondering about a few things now:

• What is the impact of flagging an IP or CIDR range as "risky" when investigating events in Cloud App Security?  Does this then just flag future events from that login attempt as coming from a risky IP address or does it do any sort of blocking?  I guess that it would feed this tag info over to Azure Sentinel. 
• What is the impact of adding IP's and CIDR range to the connection filter policy in Microsoft Defender / Threat Policies / Anti-spam policies?  I know from experience that senders with that mail server IP are indeed prevented from sending out domain email because once I accidentally listed an IP for one of our customer's email servers and their emails to us bounced.  I can tell you that if you use a tool like AdminDroid and audit the login attempts you will see attempted logins for invalid usernames (meaning that the UPN does not exist in our AD domain) and when you block these in the connection filter policy you then don't see future attempts from this IP - or so it seems to me!  

[#7]

@Jim Hill thank you for your insight. I explored the two options 

1. Flagging the IP/s as Risky will be useful when linked to a CloudAppSecurity Policy. Reference article 
   
   Now, looking at the Activity logs, I mark IP as 'Set as Risky IP and add to denylist'. I assume it feeds in Threat detection list of bad IP's.
   
   I still see failed sign-in attempts on AAD sign-in logs, and no alerts in CloudAppSecurity policy.
   
   
2. In regards to Connection filter Policy.
   It is used for email filtering (allow/block messages) to control the emails landing in users inbox. Reference link
   I do not use AdminDroid hence not aware of those features.

All these actions discussed gets triggered after a sign-in attempt has been made.

Another failed instance

- Legacy authentication is blocked, however on failed sign-in one of the attempts is using IMAP4.