Microsoft Secure Tech Accelerator
Apr 03 2024, 07:00 AM - 11:00 AM (PDT)
Microsoft Tech Community
SOLVED

Announcement: Office 365 Secure Score Released to Public Preview

Microsoft

Microsoft is pleased to announce the preview availability of a new security analytics service called the Office 365 Secure Score. The Secure Score is a security analytics tool that will help you understand what you have done to reduce the risk to your data in Office 365, and show you what you can do to further reduce that risk. We think of it as a credit score for security. Our approach to this experience was very simple. First, we created a full inventory of all the security configurations and behaviors that our customers can do to mitigate risks to their data in Office 365 (there are about 77 total things that we identified). Then, we evaluated the extent to which each of those controls mitigated a specific set of risks and awarded the control some points. More points means a more effective control for that risk. Lastly, we measure the extent to which your service has adopted the recommended controls, add up your points, and present it as a single score.

 

The core idea is that it is useful to rationalize and contextualize all of your cloud security configuration and behavioral options into one simple, analytical framework, and to make it very easy for you to take incremental action to improve your score over time. Rather than constructing a model with findings slotted into critical, moderate, or low severity, we wanted to give you a non-reactive way to evaluate your risk and make incremental changes over time that add up to a very effective risk mitigation plan.

 

The Office 365 Secure Score is a preview experience, so you may find issues, and you will note that not all of the controls  are being measured. Please share any issues on the Office Network Group for Security. You can access the Secure Score at https://securescore.office.com.

 

The Secure Score does not express an absolute measure of how likely you are to get breached. It expresses the extent to which you have adopted controls which can offset the risk of being breached. No service can guarantee that you will not be breached, and the Secure Score should not be interpreted as a guarantee in any way.

 

Your Secure Score Summary

The first, most important piece of the Secure Score experience is the Score Summary. This panel gives you your current Secure Score, and the total number of points that are available to you, given your subscription level, the date that your score was measured, as well as a simple pie chart of your score. The denominator of your score is not intended to be a goal number to achieve. The full set of controls includes several that are very aggressive and will potentially have an adverse impact on your users’ productivity. Your goal should be to optimize your action to take every possible risk mitigating action while preserving your users’ productivity.

 

As mentioned, the Office 365 Secure Score is in a preview release. Over the coming months you will see us continue to add new controls, new measurements, and improvements to the remediation experiences. If you like what you see, please share with your network. If you see something we can improve, please share it with us on the Office Network Group for Security. We’re looking forward to seeing your scores go up, and making the Secure Score experience as useful, simple, and easy as it can be.

 

Read More Here: https://blogs.technet.microsoft.com/office365security/new-security-analytics-service-finding-and-fix...

72 Replies

Hey Ole,

Thanks for the feedback! You are reading our minds. We are working on publishing the secure score data through API end points that will live in the same place as the Management Activity API endpoints (which provide activity and audit data), so that our third party ISV partners can /super/ easily include this data in their stores. OMS is one of those key partners.

Thanks!

Brandon Koeller

HI Brandon,

when I try to access my demo tenant, I do not get any current scoring.

The box  "Your Secure Score is:" is missing.  Instead I see only a target score.

 

Do I need a valid subscription in Azure first to setup Scure Score?

 

Untitled.png

 

 

Hi Karsten,

Thanks for reaching out. I am able to confirm that your company does have results in the Secure Score database. Can you email securescore@microsoft.com so we can connect offline and troubleshoot?

Thanks!

Brandon Koeller

Here's Brandon's session from Ignite on Secure score:

 

Interested to see Secure Score has been promoted this week via the Message center, also there have been further advancements -  

 

"We have introduced the first fully remediated workflow under the control "Designate less than 5 tenant admins". Click "Learn more" after expanding the widget"

 

The Get Advice section that points to this community is going to the old address (https://network.office.com/t5/Security-Privacy-Compliance/bd-p/security_privacy), by the way, which generates a certificate mismatch.

Great tool.

 

I made some updates to improve my score.

 

When will I see my score updated?

 

 

My client is not using Exchange Online. It would be helpful if there was some way to exclude the actions related to exchange from the  Secure Score recommendations

Hi Mark,

Thanks for reaching out. The Secure Score is calculated once per day (at 1am PST). Please note that not all of the controls are instrumented, so some actions might not reflect an improvement in your score yet (those controls are labeled [Not Scored]). Otherwise, you should see your score reflect your actions within a maximum of 24 hours!

Thanks,

Brandon Koeller

Hi Dean,
Thanks for the feedback. Long term, we will definitely exclude controls for services that you don't have. For now, we've opted to include all of the controls since several are not scored to help users understand the full range of options. We also plan to give you the ability to exclude specific controls that you know you will never be able to enact, even if they pertain to services you own.
Thanks!
Brandon Koeller

@Brandon Koeller wrote:
Hey Paul,
... It is surprisingly hard to find the source data in the ecosystem, ...

Well, at last! Someone from Microsoft acknowledging this. Perhaps you could also raise the visibility of some of the audit issues - like missing data from the audit reports.

 

Also perhaps you could get someone to finally deal with the issue of trying to identify which users have not used the system recently (e.g. have not logged in in the last 90d). This appears to still be virtually impossible, especially when users are not using Exchange Online.

 

These issues are causing no end of problems.

 

I recently tried to identify people not using the system in order to recover licenses. I used the audit reports for the last 180d thinking that at the very least all active users must have changed their password in that time and that should have been audited. Needless to say that resulted in nearly 10% of identified users that were actively using the system.

Hey Julian,
Thanks for the feedback. My comment about the difficulty of finding source data in the system is related to the complexity of the back end ecosystem, not the availability and accessibility of relevant data for customers. In general, customer-facing data stores are meant to be straightforward, at least through the supported interfaces (usually web, api, and powershell). To your point, however, there are some resources that you can use to get your answers:
-The Admin Center Usage Reports page should allow you to discover which users are using which services for any given period of time: https://portal.office.com/adminportal/home#/reportsUsage
-You can also focus just on logons by looking at the list of users and comparing it to the logon activity logs in the service. I've taken the liberty of whipping up a quick powershell script which dumps the UPNs of users who have not logged in for the last 90 days: https://github.com/OfficeDev/O365-InvestigationTooling/blob/master/InactiveUsersLast90Days.ps1
-The Search-UnifiedAuditLog cmdlet, and its web interface (https://protection.office.com/#/unifiedauditlog) is a great resource to tracking any kind of activity in the service.
-If you are targeting illicit activity detection along discrete threat vectors, you can also use our 'Finding Illicit Activity The Old Fashioned Way' article: https://blogs.technet.microsoft.com/office365security/finding-illicit-activity-the-old-fashioned-way...

Thanks!
Brandon Koeller

It is nice tool and this is what we really need, we couldn't expect to teach everyone about security issue and threats and they just want to press one key to see what is going on and we are the one in background working to make sure that key is working fine.

Many thanks Brandon. I've been tracking these issues for some while but I've struggled to pin down actual evidence.

 

Having just revisited the issues that I'm having. I now have hard evidence from the get-msoluser and the combined audit log that something is very badly wrong. At least with our tenancy if not something wider.

 

Two definitive issues: One is that get-msoluser consistently reports some users with PasswordNeverExpires set to TRUE which should never happen.

 

The second is even more serious. I have found a user who is currently logged into the system but according to the Get-MsoUser data hasn't changed her password for 181 days (our tenant is set to require password change after 90d). Here is some relevant information:

BlockCredential : False
IsLicensed : True
LastPasswordChangeTimestamp : 2016-04-22 11:27:22
LicenseReconciliationNeeded : False
OverallProvisioningStatus : Success
PasswordNeverExpires : False
StrongPasswordRequired : True
StsRefreshTokensValidFrom : 2016-04-22 11:27:22
ValidationStatus : Healthy
WhenCreated : 2013-05-07 10:11:03

Checking the combined audit log I can see that it agrees that the user last changed their password on the 22nd April but they are still logging in. They should not have been able to log in after July 21st. However, the audit log has recorded 23 logins since then.

 

Previously, I'd been assuming that some data was missing from the audit logs but it appears that there may be a more serious issue.

Thanks for sharing this. I hope that this is an isolated event, but I'll do some research on my clients tenant to see if the same problem exists

No problem Dean.

 

I don't believe that it is isolated for us. It's just that this is the first time I've actually been able to prove it happening.

Also, thanks for the pointer to the Investigation Tooling Github. I've run the script to check for users not logged in in the last 90d but the first entry that it reports is one that I know is used daily because the person sits behind me in the office! They are a very heavy Office 365 user as they helped my set up our tenant.

Great service and it's really helped secure and understand Office 365!

 

One of the items counts the number of Exchange mailboxes with auditing enabled and the action is to enable auditing. The mailbox count and number of mailboxes with auditing enabled varies has not matched my mailbox count and their audit status. We're new to Office 365 so we may be overlooking a system database or some other setting. We're simply using get-mailbox and also with the -SoftDeletedMailbox and -Migration switches and check the AuditEnabled status.

 

Thank you!

I've raised a couple of issues in the github log. I think the reason it thought my colleague hadn't logged in is that it only returns 5k records. That's nowhere near enough for a 90d review of logins for 8k users. I'll update the issue with a new script when I've finished it or I can do a pull if you prefer, let me know in the issue (I am TotallyInformation on GitHub).

It turns out that the actual script needed to validate unused logins is significantly more complex than the example. Even reworking the example to deliver the maximum 50k entries is nowhere near enough to cope with the logins from an active 8k seat tenancy for 90d.

 

So I had to turn the script around so that the combined audit log is queried by user. Even then a few users exceeded 5k logins in 90d, one admin user had nearly 20k logins in that period.

 

The new example script is on GitHub here:

https://github.com/TotallyInformation/PowerShell/blob/master/Get-O365UserLoginStats.ps1

 

Oh, and it takes around 10 hours to run!!