Investigating malicious OAuth applications using the Unified Audit Log

[matthewzorich]

Abstract

Supply chain attacks continue to evolve in sophistication with new TTPs emerging every few months. In this article we highlight some of the most recently observed TTPs and our recommendations for investigating and mitigating them in your environments. Specifically, Microsoft is seeing new and ongoing incidents where Threat Actors are compromising a trusted partner such as a cloud solution provider or reseller partner to gain access to a victim organization. This access is often in the form of creating or modifying an enterprise application so that the Threat Actor can use that application for the purpose of gaining access to sensitive information including email messages, Teams messages, and documents stored in SharePoint or OneDrive for Business.

In 2021, Microsoft published general guidance on investigating cloud solution providers in this post. This article provides additional guidance for incident responders on investigating such events.

A key to any investigation is what is forensic data is logged, where that data is available, and how long is the retention of that data. For Microsoft Entra ID and Microsoft 365 services, log data can be found in several places, as visualized below.

Figure 1. Data retention in Microsoft 365 and Microsoft Entra ID

NOTE: As noted here https://learn.microsoft.com/en-us/purview/audit-log-search, Microsoft has recently changed the default retention period for Audit (Standard) from 90 days to 180 days. Audit (Standard) logs generated before October 17, 2023, are retained for 90 days. Audit (Standard) logs generated on or after October 17, 2023, follow the new default retention of 180 days.

Many organizations may be sending the log data from Microsoft Entra ID and Microsoft Defender for Cloud Apps to a SIEM solution such as Microsoft Sentinel. For organizations that are using a SIEM solution which retains data for an extended period, the investigation may be done more easily in those environments but is beyond the scope of this article.

The guidance provided here is focuses on using data from the Unified Audit Log and Microsoft Entra ID sign in & audit logs taken from the portal itself, since this data is available to all organizations.

Unified Audit Log

If you are looking for audit information that is older than logs retained in Microsoft 365 XDR Advanced Hunting, Microsoft Sentinel or another SIEM, then you can use the Unified Audit Log to retrieve information. Depending on your licensing level, data can be available up to a year. The Unified Audit Log is a central collection of audit events relating to Microsoft 365, including activities such as file download events from SharePoint or OneDrive.

Using the Audit search functionality, you can create custom searches to retrieve the relevant information from the Unified Audit Log. This data can then be downloaded as a CSV file and analyzed to understand what happened.

Figure 2. Unified Audit Log search portal

You can select specific date periods, users, even specific activities, and workloads (such as file download events from SharePoint). You can additionally complete free text searches on any indictors of compromise, such as an IP address, or the guid associated with an application object.

In addition to the GUI, you can also retrieve information via PowerShell, using the Search-UnifiedAuditLog cmdlet. Via PowerShell, you can still filter on users, time ranges or events.

As the central location of all audit data for Microsoft 365, the various activities all have different data associated with them. For instance, a file download event would have the name of the file, and the user associated with the activity. When a user creates a mailbox rule in Exchange Online, it has the name of the rule and configuration settings specific to it. Because of the dynamic nature of these different events, it is recommended to do broad searches to ensure all activities are retrieved.

Depending on the nature of your investigation, some recommendations for ensuring all activities are retrieved.

• In some cases, the friendly user principal name (UPN) is not logged in the Unified Audit Log but rather only the User Principal Object Id (UPOID), or in some cases, the value is contains the UPOID value with the hyphens removed. For completeness, doing three sets of searches of the Unified Audit Log for activity from a suspect user identity should be undertaken:
  • Search for the UPN, i.e. search for eric.lang@contosohotels.com both using the -UserIds switch and the -FreeText switch. Searching based on the -UserId switch will return the events where the UserId value is the actor. To recover events where the user identity was the target of the action, the -FreeText search is required.
  • Search for the UPOID, i.e. 8a1e7b36-e60c-409a-992f-dc838f27a102 both using the -UserIds switch and the -FreeText switch
  • Search for the UPOID without hashes with the term “user_” prepended i.e.  user_8a1e7b36e60c409a992fdc838f27a102 using both the -UserIds switch and the -FreeText switch.
• For activity associated with application abuse, search for the Application Id (sometimes known as the Client Id) of the application with and without the hyphens, such as cb457366-b74c-42f2-a8c7-a7648e4572f1 and cb457366b74c42f2a8c7a7648e4572f1. This should be a “-FreeText” search using the Search-UnifiedAuditLog cmdlet.
• When searching for IP indicators, there are few important notes
  • Some events will record the IP address of the Microsoft service rather than the IP address of the actual identity, try to correlate that information with known indicators of compromise or sign in event data to ensure it is malicious.
  • Searching in the GUI or using the Search-UnifiedAuditLog cmdlet by specifying the IP address as a search term may not return all results. Those commands are searching over the indexed terms in the Unified Audit Log.
  • In the case of the Client IP address, that address is stored in the ClientIP address field in the AuditData object. In some cases, that value may not be populated, but another IP address field in the AuditData field may be populated. For these cases, also do a “-FreeText” search against the Unified Audit Log.
• Searching the Unified Audit Log using Search-UnifiedAuditLog powershell cmdlet using the filter parameters (-UserIds, -IPAddresses, -SiteIds, -ObjectIds, -Operations, -RecordType) searches only the values in the “common schema” which are indexed by the system (see https://learn.microsoft.com/en-us/office/office-365-management-api/office-365-management-activity-ap... for the common schema definition).
• The different workloads may store additional content in the raw “AuditData” object. Using the -FreeText switch will search through the raw event data except for items which are indexed in the common schema. Therefore, it is important to search both by property (eg -IpAddresses) and using the -FreeText option to capture all events which contain the specific indicator of attack or indicator of compromise.

Example Investigative Process

Microsoft Incident Response (Microsft IR) recently investigated an attack which included suspicious activity associated with an application in the victim’s tenant. Here is how we went about the investigation. A conceptual framework for the investigation is shown in the figure below where there can be bidirectional iteration between each data collection until the common operating picture of the event converges. That is, Microsoft IR continues to pivot on these interconnected IOCs, until no new events, or compromised identities, are uncovered.

Figure 3. Visualization of the investigative process

Given the indicator of attack/compromise consisting of a suspicious ApplicationId, we collected data from the Unified Audit Log using a query like the following:

Search-UnifiedAuditLog -StartDate <StartDate> -EndDate <EndDate> -FreeText <ApplicationId>

Specifically, we used a variation of the script located https://learn.microsoft.com/en-us/purview/audit-log-search-script to do the search.  

If in your environment you have document names, group names, user display names or other forensic information that may contain non-ascii characters, we recommend including the -Encoding UTF8 switch at the end of your commands, as this will ensure the export handles those characters properly.

In our example, we observed audit events of the type:

• Add application.
• Add service principal.
• Update application.
• Consent to application.
• Add app role assignment to service principal.
• Update application – Certificates and secrets management
• And additional audit events indicating that the application was used to access information from the customer environment

We then analyzed the resultant data and were able to determine the following:

• When was the application created?
• Who created or modified the application?
• What consent was granted to the application?
• Were credentials added to the application or related service principal (aka workload identity) associated with the application.
• What identity was used to grant or modify consent to the application?
• What IP addresses were associated with the creation or modification of the malicious application?

In this case, the analysis revealed that the identities associated with creating the application and modifying the application belonged to cloud service providers (CSPs). Following from that finding, the investigation took the following path.

Understand what additional activities were carried out by the identity that created the application

• Pivot on the identity that created the application to understand what else that account did. Searching the Microsoft Entra ID audit log for the user, as well searching the Unified Audit Log for the username, and the users object id with and without the hyphens.
• To accomplish this using the Search-UnifiedAuditLog cmdlet, doing a search based on the UserIds switch based on the UserPrincipalName (eg eric.lang@contosohotels.com) and then repeat the search using the -FreeText switch and the -UserIds switch, the UPOID, and the user_<UPOID no hyphens> version for the same time frame.
• This can help establish time windows where the activity from that identity should be viewed as potentially malicious or Threat Actor related.
• If this uncovered other applications created, or users created, then the same process should be completed for those applications and users. Investigative work like this is often repeated cycles of determining user or application compromise, and then investigating those users or applications to discover additional indicators and understand impact
• On top of persistence mechanisms, or any other configuration changes, this will also surface any data access events

Pivot through any malicious IP addresses located

• To accomplish this using the Search-UnifiedAuditLog cmdlet, Microsoft IR completed a search based on the -IPAddresses switch on the uncovered addresses (eg 50.50.50.50) and then repeat the search using the -FreeText switch and the IP Address.
• These IP addresses may uncover additional compromised identities - either users or applications
• They also may uncover further data access events or other persistence mechanisms

Pivot through any newly found identities

• As you uncover additional compromised identities, the same cycle of investigation begins

A question often asked by customers in these engagements was what email was accessed, or what Teams messages were accessed. Data sources such as Microsoft Defender for Cloud Apps and the Unified Audit Log will audit the metadata related to email and Teams access events, such as email or message identifiers. The content of the messages or emails however is not logged, which makes sense – it is not practical to log the entire contents of an email. For items such as email or Teams messages, the event metadata is retrieved.

From that surfaced metadata, Microsoft IR then exports the full content from Exchange Online or Microsoft Graph APIs and provides that to the customer to complete a data impact assessment. For file download events, the name and path of the files are provided so that customers can understand what data was exfiltrated. A complete analysis of the content accessed by the Threat Actor is required to determine if the Threat Actor may have gained additional methods to access the environment or establish persistence.

Microsoft IR utilizes Azure Data Explorer (ADX) and Kusto Query Language (KQL) to query all these log sources at scale to understand the story of the compromise. However, there is no requirement to use a specific platform for your investigation, if you have Microsoft Sentinel, or another SIEM with the required data, you can perform your investigation there. For smaller scale compromises, using the inbuilt portals can also be sufficient.

Some examples of forensically interesting events, and how Microsoft IR analyses the data associated are:

• FileDownloaded – This will show the filename and file path of file downloaded events, and the identity that triggered the action – whether that is a user or an application. Microsoft IR provides customers with a list of downloaded files that were attributed to the threat actor so the customer can understand what data has been exfiltrated
• MailItemsAccessed – this event shows that an email within a mailbox in Exchange Online was accessed. Events can either be a bind, in the case of a single email being accessed. Bind operations are aggregated into 2-minute intervals, but each individual message accessed is still listed in each single audit event. The second type of event is a sync, in the case of a folder being synced from the full Outlook client. To reduce noise, only a single sync record is logged for each folder that was synced - and we assume that all emails in a folder have been synced. Additional information about this audit event can be found here
• MessagesListed – When a Teams message is retrieved from the Microsoft Graph API, this event is logged. It is not logged when a message is listed from the Microsoft Teams client itself. Threat actors can use tooling they have created to interact directly with the Microsoft Graph API and access messages in bulk, the MessagesListed event is valuable for understanding the impact of this kind of activity.
• Add service principal: This event is logged when a service principal is added to the tenant. If this is malicious activity during the investigation, then Microsoft IR pivots on that application to understand what other activities it completed, and what data and services it accessed. The ApplicationId, which is a guid, of the newly created application is a strong indicator of compromise.
• New-InboxRule: A mailbox owner or other user with access to the mailbox created an inbox rule in the Outlook web app. With these events Microsoft IR is interested in what actions the rule was taking, was it trying to hide emails with particular words or phrases, that may be an indicator of threat actor intent. If they are hiding emails related to invoices or payment, it may be a sign the threat actor is seeking financial gain through business email compromise.

Several thousand unique events can be written to the Unified Audit Log, a listing of the most common is available here. There is also specific guidance for Microsoft Teams. Microsoft IR has previously covered tips and tricks for using the Unified Audit Log in the post Good UAL Hunting.

It is often not feasible to extract the entire Unified Audit Log for a tenant due to sheer volume of data held within it. Therefore, Microsoft investigators take an iterative approach. Starting with known indicators of compromise and then expanding based on those initial findings. When the operational picture converges and no additional indicators of attack or compromise are revealed, the data collection phase can be viewed as complete.

Recommendations

While the CSP to customer partnership is unique in terms of providing privilege into a customer tenant, the investigative process remains very similar to any investigation of Microsoft Entra ID and Microsoft 365.

We recommend that customers take the following steps:

• Audit current partner agreements and understand if they are required, they may be longer required. Over time, customers can change partners as business requirements or commercial agreements change. Often older partners still hold privilege over customer tenants
• If using delegated admin privileges then migrate to granular delegated admin privileges and grant partners to only the access required to complete their work. From a partner point of view, migrating from DAP to GDAP has no effect on the partner earned credit (PEC) program
• If partners need to complete privileged work and you are uncomfortable with any standing access via DAP or GAP, you can, of course, create them a named account in your own tenant and require they adhere to the security policies, such as Conditional Access, which you dictate
• Understand where the forensic information related to Microsoft 365 and Microsoft Entra ID is available, and how to query that data in case of compromise

If organizations do not have the capability to investigate these types of compromises, it is recommended to engage an experienced incident response team, such as Microsoft IR.

Detailed below is a reminder on the different type of CSP privilege models in Microsoft 365 and some additional notes on finding forensic data directly in the various portals, this can be valuable for finding events in the last 30 days.

Understanding partnership models in Microsoft 365 & Microsoft Entra ID

Several types of partner configurations can exist between a customer and a CSP which provide some level of privilege in the customer tenant, these include.

Microsoft Entra ID delegated administration privileges (DAP)

The DAP permissions model was created to allow Cloud Solution Providers (CSP) to provide services and licensing support to customers. A CSP could send an invitation to a customer to request a partner relationship. This was designed to allow partners to manage various aspects of customers Microsoft 365 and Azure services without needing another set of credentials. For CSP’s that deal with thousands of customers, this provides a much simpler way to manage those relationships, instead of maintaining thousands of unique credentials across customer tenants.

Prior to an update to the permissions model, upon accepting one of these invitations, the CSP would gain Global Administrator rights in the customer tenant.

Microsoft Entra ID granular delegated administration privileges (GDAP)

GDAP is an updated version of DAP built with the principles of Zero Trust at the forefront. GDAP was added not long after the publication of the 2021 article linked above. GDAP gives customers much stronger controls over what access partners are granted into their tenant. Instead of allowing blanket Global Administrator access to partners, customers can instead pick and choose more specific roles better aligned to the work the CSP needs to complete. If the relationship is purely to facilitate licensing requirements, then the Licensing Administrator role may be better fit and more aligned to the principle of least privilege. In addition, GDAP grants are time bound. Customers must re-approve the granting of these privileges.

Identifying existing relationships

Customers can find the existing partnerships configured in their tenant, and which permissions model is currently being used. These can be found by browsing to the Microsoft 365 admin center and then navigating to Settings then to Partner relationships. 

In the Partner relationships pane, you can view a list of all service providers that have established a relationship with the tenant and whether the service provider has any roles assigned.

Figure 4. Identifying DAP & GDAP as a downstream customer

Customers should audit the list of partners visible here and determine if they are still required. Often these partnership configurations may be a relic of prior commercial agreements and the two organizations are not actively working with each other anymore.

For customers, it is important to understand that these partner relationships, especially in the case of DAP, may maintain full Global Administrative rights over your tenant. Just like your own Global Administrators, or other users of privilege, should be monitored, so should any activity from partner accounts, and alerts created for suspicious actions.

While end customers cannot see a list of all users in the service provider’s tenant that can make administrative changes to the end customer tenant, they can see any sign in activity, audit events and other activities in the Unified Audit Log and Microsoft Defender for Cloud Apps (if it is enabled at the time of the event).

Finding Partner Sign in Events

When a partner signs into a customer tenant in Microsoft Entra ID, it is categorized as a unique type of sign in type. Customers can find these logins (refer to Figure 2) by viewing the Microsoft Entra ID sign-in logs and filtering for a Cross tenant access type of Service provider. The results can be exported by clicking Download and leveraged to further target your triage across Azure and Microsoft 365.

Figure 5. Sign-ins by service providers

Hunting for Partner Data Access

In many cases, CSP compromise can lead to downstream data access in a customer tenant, such as the exfiltration of data from SharePoint or OneDrive. Microsoft Defender for Cloud Apps tracks these kinds of events across Microsoft 365, and any third-party applications you have connected.

If you believe an adversary has taken control of a partner account and accessed your tenant, then Microsoft Defender for Cloud Apps is a good place to start to determine the impact to your data.

You can create specific searches in the Microsoft Defender for Cloud Apps activity log, looking for specific users, or IP addresses or activities.

Figure 6. Defender for Cloud Apps activity log

If you need more granular queries, you can select the Advanced filters option. For events older than 30 days, you can select the ‘Investigate 6 months back’ option, though the filtering options for that historical data are limited.

Importantly for organizations, it is important to ensure that Microsoft Defender for Cloud Apps is fully connected with Microsoft Entra ID and Microsoft 365. You can confirm this by browsing to the Settings page in the Microsoft 365 XDR portal, from there, select Cloud Apps. Once the settings for Microsoft Defender for Cloud Apps loads fully, select App Connectors. A list of all the applications connected will be presented, for Microsoft 365, select the Edit settings option.

Figure 7. Editing the Microsoft 365 Connector in Microsoft Defender for Cloud Apps

And ensure that all the options are connected. If you are already licensed for Microsoft Defender for Cloud Apps, there is no additional cost to enable this. If you send this data to Microsoft Sentinel however, there will be added events sent which will incur additional ingestion charges.

Figure 8. Enabling all components of the Microsoft 365 connector

Hunting for Partner Audit Events

Once a partner account has accessed your tenant, any changes they make to Microsoft Entra ID will be logged into the audit log, in the same way as your regular users would be. The audit log will track any platform level changes to Microsoft Entra ID, including user creation events, users being assigned to privileged roles, MFA registration events, application creation events and more. You can search the Microsoft Entra ID Audit Logs for any events for specific accounts.

Figure 9. Filtering the audit log in Microsoft Entra ID

When looking for events associated with users, remember that a user account can be either the initiator (the actor) or the target of an event. If eric.lang@contosohotels.com changes the password of bob.smith@fabrikam.com then Eric Lang is the actor and Bob Smith is the target.

Additionally, in the case of malicious application abuse, the initiator may be the application itself, rather than a user.

As mentioned at the outset, for any investigation understanding where data resides and how to access it when required, and the retention of that data, is crucial to success. Once Microsoft IR gets access to the required forensic data, an iterative process is then undertaken to ensure that the entire story of the compromise is uncovered.