Core Infrastructure and Security Blog

14 MIN READ

Tracking the Source of ADFS Account Lockouts

Microsoft

May 18, 2020

Introduction

I am Eunice Chinchilla, and I am a Premier Field Engineer at Microsoft focusing in the Identity space. In this guide you will find a description of the task at hand and the journey I went on to deliver a solution for my customer. Each section includes a high-level summary of what will be discussed and any pre-requisites to be able to try these in your own test environment. Once you are ready to start testing there are steps accompanied by descriptions and screenshots to walk you through the whole process I went through, including any hiccups I ran into. Please read the Problem Statement/Customer Story up next to provide you with background information and help you get started.

Problem Statement/Customer Story

I was working with a customer and they needed to identify why user lockouts were occurring. They had a hybrid environment with ADFS in place meaning the domains are federated. In this scenario, it is necessary to know if the lockout happened on-prem or on a O365 application. The plan of action was to look at the logs in both the Domain Controller and the ADFS server and compare them to the logs in Azure to see what we could find. Unfortunately, we were limited to only the ADFS server logs because there was no access to the Domain controller due to permissions. All the investigation was done solely on the ADFS server and Azure logs. The following solutions are a summary of the journey I went on with the purpose to find the most information possible related to the lockouts. I spent time testing in my own environment which is set up like theirs. In the end, it was narrowed down to the application used when the lockout occurred.

Approaches

First Approach

The first attempt in searching for the information needed was to look through the security logs and see what we could find.

Second Approach

Since looking through the logs resulted in very little information that was not enough for the customer to figure out where the lockouts where coming from, we resorted to Log Analytics and Azure Security Center to help find more specific and detailed information.

Third Approach

The customer did not have access to Azure Log Analytics and Azure Security Center which left us to explore the Azure Sign in Logs.

First Approach – PowerShell Script run on ADFS Server

Summary

Initially, I looked at the event logs in the ADFS server in my test environment. There were a few things I did before which are listed below in a checklist.

☐ Enable the appropriate logs: ‘Source AD FS Auditing Logs’

☐ Create test lockout events

Note: You can create these for testing purposes or just use the lockouts that already exist in your environment.

☐ Make sure you have PowerShell installed in your ADFS Server

☐ If you have multiple servers, make sure to enable remote access

Steps

1. First, make sure the ‘Source AD FS Auditing Logs’ are enabled in the ADFS server. This allows you to see the events with ID 411. Event 411 occurs when there is a failed token validation attempt (authentication attempts). In the event viewer, the IP address of the device used is provided. This can be useful for tracking the lockout.

Enabling the Source AD FS Auditing Logs

Open the Local Security Policy window from the Start menu on your server.

Once opened, you should see a view like the window below. Click on ‘Advanced Audit Policy Configuration.’

Click on ‘System Audit Policies – Local Group Policy Object.’

Navigate to the ‘Logon/Logoff’ category. You should now see all the subcategories available.

Select the ‘Audit Account Lockout’ subcategory.

A separate window will appear, and you can check the boxes for Success and Failure audit events. After having done this, the ‘Source AD FS Auditing Logs’ have successfully been enabled.

Under the ‘Explain’ tab you can find a detailed explanation of what happens when you enable these audit logs.

2. Skip this step if you already have lockouts in your environment. If not, you can create some account lockouts, as I did in my test environment.

Create test account lockout events

Open the ‘Local Security Policy’ window and click on ‘Account Policies.’

Click on ‘Account Lockout Policy.’

On the right-hand side are the security settings you can customize for the account lockouts. I set lower amounts of time so I could create multiple account lockout in shorter amounts of time.

Note: It is necessary to set the account lockout policy in the Domain Controller as well. Make sure the account lockout policy in the ADFS Server does not surpass the Domain Controller account lockout policy amounts of time and attempts. This will cause an error.

After setting the lockout policy on the appropriate server, start one of your Windows 10 joined Virtual Machines. From this VM, on any browser, you will attempt to log on to your account using any of your test user accounts.

Enter the wrong password on purpose as a simulation of an account lockout.

The ADFS splash page will not notify you when you’ve been locked out and will continue to display the view below. If your “invalid attempt logon” number was 2, repeat this process 3 times to ensure the lockout of the account occurred.

View the lockout event(s)

To verify the lockout happened open the Event Viewer.

Navigate to the ‘Security Logs’ under ‘Windows Logs.’

Here you can view the event(s) generated when the lockout(s) occurred.

You can also filter by error code (once you know which error code to look for). In this case, we can filter by error code 4625.

On the right-hand side, in the ‘Security’ window under ‘Actions’ select the ‘Filter Current Log…’ option.

Check every box for the ‘Event Level’ and enter the event ID. Click ‘Ok.’

Below are the results of the filtered events.

3. In your ADFS Server, open PowerShell ISE to run script that will be pulling the events related the lockout events. In this script we are querying for all the 411 events from the Source AD FS Auditing logs. The reason you want to filter for Event ID 411 is because this event gets created when there is a failed authentication attempt. The expected output is the Username, Activity ID, IP address, Date Created.

If you don’t have PowerShell ISE installed visit this site and resume at this step after installing successfully.

https://docs.microsoft.com/en-us/powershell/scripting/install/installing-windows-powershell?view=powershell-7

Running the Script

In your ADFS Server, Open PowerShell ISE and ‘Run as an Administrator.’

Copy the script below (Ctrl + C). It will be pasted onto the PowerShell ISE Script Window

#Get all the events with Event ID 411
$eventLogs = Get-EventLog -LogName Security -Source "AD FS Auditing" | Where-Object {$_.EventID -eq 411}
$eventList = @() #list that holds event objects

foreach ($event in $eventLogs)
{
    $message = $event.Message
    $emailRegEx = "\b[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,4}\b"
    $message -match $emailRegEx | Out-Null

    $eventObject = New-Object -TypeName psobject #start off with an new object

    #add the User Name
    $eventObject | Add-Member -MemberType NoteProperty -Name Username -Value $Matches.Values

    $activityIDRegEx = "[A-Za-z0-9]+-[A-Za-z0-9]+-[A-Za-z0-9]+-[A-Za-z0-9]+-[A-Za-z0-9]*"
    $message -match $activityIDRegEx | Out-Null

    #add the activity ID
    $eventObject | Add-Member -MemberType NoteProperty -Name "Activity ID" -Value $Matches.Values
      
    $IPAddressRegEx = "[0-9]+\.[0-9]+\.[0-9]+\.[0-9]*"
    $message -match $IPAddressRegEx | Out-Null

    #add the IP address
    $eventObject | Add-Member -MemberType NoteProperty -Name "IP Address" -Value $Matches.Values

    #add the date the event happened
    $eventObject | Add-Member -MemberType NoteProperty -Name "Date Created" -Value $event.TimeGenerated

    $eventList += $eventObject #add each event object to a list

    #Now, there is an event object that contains all the relevant information about the event. 
   

}

#Group by user name and send the list of account info to a CSV file
$eventList | Sort-Object -Property "Username" | Out-File -FilePath "C:Enter your path here\lockouts.csv" -Encoding utf8

Paste the code into the Script window (Ctrl + V).

Note: If you have multiple ADFS Servers you can either run the script above on each server individually and have multiple csv output files or run the code below in PowerShell ISE from any ADFS server, but you must first enable remote access to the other ADFS servers. See link below on how to enable remote access to your other machines.

https://docs.microsoft.com/en-us/windows-server/remote/remote-desktop-services/clients/remote-desktop-allow-access

Multiple Server Script

foreach($s in $adfsServers){

    #Get all the events with Event ID 411
       
    $_xmlLockoutAdfs = "*[System[EventID=411]]"
    $eventLogs =  Get-WinEvent -ComputerName $s -LogName Security -FilterXPath       $_xmlLockoutAdfs
    
    $eventList = @() #list that holds event objects

    foreach ($event in $eventLogs)
    {

        $eventObject = New-Object -TypeName psobject #start off empty object

        $message = $event.Message
        $emailRegEx = "\b[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,4}\b"
        $message -match $emailRegEx | Out-Null

        

        #add the User Name
        $eventObject | Add-Member -MemberType NoteProperty -Name Username -Value $Matches.Values

        $activityIDRegEx = "[A-Za-z0-9]+-[A-Za-z0-9]+-[A-Za-z0-9]+-[A-Za-z0-9]+-[A-Za-z0-9]*"
        $message -match $activityIDRegEx | Out-Null

        #add the activity ID
        $eventObject | Add-Member -MemberType NoteProperty -Name "Activity ID" -Value $Matches.Values
      
        $IPAddressRegEx = "[0-9]+\.[0-9]+\.[0-9]+\.[0-9]*"
        $message -match $IPAddressRegEx | Out-Null

        #add the IP address
        $eventObject | Add-Member -MemberType NoteProperty -Name "IP Address" -Value $Matches.Values

        #add the date the event happened
        $eventObject | Add-Member -MemberType NoteProperty -Name "Date Created" -Value $event.TimeCreated

     

        $eventList += $eventObject #add each event object to a list

        #Now, there is an event object that contains all the relevant information about the event. 
   

    }

    $eventList | Sort-Object -Property "Username" | Out-File -FilePath "C:Enter your path here\lockouts.csv" -Encoding utf8 

}

4. Open the Event Viewer and filter for Events 411. This is useful to confirm the results from the script.

Open the security log

Select filter current log

Filter on 411 events

411 event example

Second Approach – Log Analytics and Kusto Query Language on ADFS Server

Summary

Given the limited results of the event logs we decided to take another approach in the search of more detailed information. The event logs provided the Username, Activity ID, IP address, Date Created which was not enough information to conclude where exactly the lockouts where occurring. In hopes of finding more specific information we resorted to Azure Log Analytics. Through Log Analytics we were able to successfully collect the information needed such as the application being used when the lockout occurred. Below is a list of requirements to attempt this approach.

☐ Log Analytics must be set up with current environment and connected to the correct Virtual Machines.

☐ Have Azure Security Center set up in order to retrieve security events from the ADFS Server.

☐ Admin Credentials to Azure Active Directory to be able to access the logs

Steps

If you don’t have Log Analytics, below are some helpful links to get all set up.

Creating a workspace:

https://docs.microsoft.com/en-us/azure/azure-monitor/learn/quick-create-workspace

Collecting data from Azure Virtual Machines:

https://docs.microsoft.com/en-us/azure/azure-monitor/learn/quick-collect-azurevm

If you do not have Azure Security Center set up, below are some helpful links to get all set up.

Onboard Windows computers to Azure Security Center:

https://docs.microsoft.com/en-us/azure/security-center/quick-onboard-windows-computer

1. Navigate to your Log Analytics workspace -> Logs where you will be querying the security logs.

2. Copy (Ctrl + C) and Paste (Ctrl + V) the Kusto Query shown below and then run it by clicking on the ‘Run’ button.

SecurityEvent
| where EventID == 411
| project EventData, Computer, TimeGenerated, EventSourceName 
| extend p = parse_xml(EventData)
| extend CorrelationId = tostring(parse_json(tostring(parse_json(tostring(p.EventData)).Data))[0])
    | join (SigninLogs
        | where Status.errorCode == "50140" 
        | project Status, UserDisplayName, UserPrincipalName, UserId, AppDisplayName, ClientAppUsed, AppId, IPAddress, CorrelationId 
    ) on CorrelationId 
    | project EventSourceName, Status, UserDisplayName, UserPrincipalName, UserId, AppDisplayName, ClientAppUsed, AppId, IPAddress, TimeGenerated, CorrelationId

3. Once you see the results in the result window you can export the result to a CSV file where it will display the columns specified in the query. You can find the button in the top right-hand corner of the window. Make sure you set the time range to your desired time range before running the query.

Third Approach – Run PS Command on ADFS Server to retrieve Sign in Logs from Azure Active Directory

Summary

The customer did not have a Log Analytics workspace or Azure Security Center set up. After having a couple conversations, we concluded we had to move forward without the help of these services. This meant we were now limited to only viewing the Sign-in Logs from the portal. Since the portal does not allow to filter by error code, we can use a command in PowerShell to pull the right records.

The error code being used to filter by is error code 50140 which is occurs when there is an interruption that has occurred when attempting to authenticate. Since we know the authentication is happening in ADFS and not in Azure that is why we used this error code to help us pinpoint the accounts that were being locked out. An interruption in the Azure Sign-in Logs actually meant a lockout for us.

In order to attempt this approach, you must have all the below items checked off.

☐ Make sure you have PowerShell installed in your ADFS Server

☐ Verify the Azure AD Preview is installed

☐ Connect to Azure AD from PowerShell

Steps

If you don’t have PowerShell ISE installed visit this site and resume at this step after installing successfully.

https://docs.microsoft.com/en-us/powershell/scripting/install/installing-windows-powershell?view=powershell-7

1. In the ADFS Server, open PowerShell and run command with error code. This command is pulling from the Azure AD Sign-in logs. You can export your results to CSV file from PowerShell.

Verify the AzureADPreview Module is installed

In your ADFS Server, Open PowerShell ISE and ‘Run as an Administrator.’

In the Script window, type in the command below into the Script window and hit the ‘Enter’ key

Get-Module azureadpreview -ListAvailable

If you need to install the AzureADPreview module type the command below into the Script window and hit the ‘Enter’ key.

Install-module AzureADPreview -Verbose

Running the Command

In your ADFS Server, Open PowerShell ISE and ‘Run as an Administrator.’

Type in the command ‘Connect-AzureAD’ into the Script window and hit the ‘Enter’ key.

After hitting enter, the ‘Sign in to your account’ window appears. Enter your Azure credentials.

Now, your Azure account is connected to PowerShell, you’ve verified that you have the AzureADPreview module installed and the command can be run.

Copy the command below (Ctrl + C) and paste (Ctrl + V) onto the PowerShell ISE Script Window and hit the ‘Enter’ key.

Get-AzureADAuditSignInLogs -Filter "status/errorCode eq 50140"

Output to CSV file

To export the results onto a CSV file, run the command below.

Get-AzureADAuditSignInLogs -Filter "status/errorCode eq 50140" | Out-File -FilePath "\logs.csv"

How to handle the bug

During testing in my environment, the command ran smoothly, and I acquired the desired results with my test data. Unfortunately, in my customer’s environment we were getting the error below when running the command.

After researching, we discovered this was a bug in the AzureADPreview module, to which this command belongs to. Check out the GitHub blog below if you’d like to read more about the bug and check for updates.

https://github.com/Azure/azure-docs-powershell-azuread/issues/337

We reached out to the product group and it is an ongoing process meaning we would not get a resolution in a timely manner. Then we reached out to Microsoft support and worked with a Support Engineer on finding alternatives. Below is a script provided by him to help us filter and retrieve the sign in logs from Azure and send them to a CSV file for record keeping.

Get Sign-in Logs using the script below in PowerShell

# ------------------------------------------------------------
# Copyright (c) Microsoft Corporation.  All rights reserved.
# ------------------------------------------------------------

Import-Module Azure

# Replace with your tenantId or tenantDomain
$tenantId = ""

# Replace with your desired time ranges
$toDate = "{0:s}" -f (get-date).ToUniversalTime() + "Z"
$fromDate = "{0:s}" -f (get-date).AddDays(-7).ToUniversalTime() + "Z"

# You can add more filters here
$url = "https://graph.microsoft.com/beta/auditLogs/signIns?`$filter=createdDateTime ge $fromDate and createdDateTime le $toDate and status/errorcode eq 50140"

# By default, it saves the result to DownloadedReport_currentTime.csv. Change it to different file name as needed.
$now = "{0:yyyyMMdd_hhmmss}" -f (get-date)
$outputFile = "AAD_SignInReport_$now.csv"

###################################
#### DO NOT MODIFY BELOW LINES ####
###################################

Function Expand-Collections {
    [cmdletbinding()]
    Param (
        [parameter(ValueFromPipeline)]
        [psobject]$MSGraphObject
    )
    Begin {
        $IsSchemaObtained = $False
    }
    Process {
        If (!$IsSchemaObtained) {
            $OutputOrder = $MSGraphObject.psobject.properties.name
            $IsSchemaObtained = $True
        }

        $MSGraphObject | ForEach-Object {
            $singleGraphObject = $_
            $ExpandedObject = New-Object -TypeName PSObject

            $OutputOrder | ForEach-Object {
                Add-Member -InputObject $ExpandedObject -MemberType NoteProperty -Name $_ -Value $(($singleGraphObject.$($_) | Out-String).Trim())
            }
            $ExpandedObject
        }
    }
    End {}
}

Function Get-Headers {
    param( $token )

    Return @{
        "Authorization" = ("Bearer {0}" -f $token);
        "Content-Type" = "application/json";
    }
}

$clientId = "1b730954-1685-4b74-9bfd-dac224a7b894" # Azure Active Directory PowerShell clientId
$redirectUri = "urn:ietf:wg:oauth:2.0:oob"
$MSGraphURI = "https://graph.microsoft.com"

$authority = "https://login.microsoftonline.com/$tenantId"
$authContext = New-Object "Microsoft.IdentityModel.Clients.ActiveDirectory.AuthenticationContext" -ArgumentList $authority
$authResult = $authContext.AcquireToken($MSGraphURI, $clientId, $redirectUri, "Always")
$token = $authResult.AccessToken

if ($token -eq $null) {
    Write-Output "ERROR: Failed to get an Access Token"
    exit
}

Write-Output "--------------------------------------------------------------"
Write-Output "Downloading report from $url"
Write-Output "Output file: $outputFile"
Write-Output "--------------------------------------------------------------"

# Call Microsoft Graph
$headers = Get-Headers($token)

$count=0
$retryCount = 0
$oneSuccessfulFetch = $False

Do {
    Write-Output "Fetching data using Url: $url"

    Try {
        $myReport = (Invoke-WebRequest -UseBasicParsing -Headers $headers -Uri $url)
        $convertedReport = ($myReport.Content | ConvertFrom-Json).value
        $convertedReport | Expand-Collections | ConvertTo-Csv -NoTypeInformation | Add-Content $outputFile
        $url = ($myReport.Content | ConvertFrom-Json).'@odata.nextLink'
        $count = $count+$convertedReport.Count
        Write-Output "Total Fetched: $count"
        $oneSuccessfulFetch = $True
        $retryCount = 0
    }
    Catch [System.Net.WebException] {
        $statusCode = [int]$_.Exception.Response.StatusCode
        Write-Output $statusCode
        Write-Output $_.Exception.Message
        if($statusCode -eq 401 -and $oneSuccessfulFetch)
        {
            # Token might have expired! Renew token and try again
            $authResult = $authContext.AcquireToken($MSGraphURI, $clientId, $redirectUri, "Auto")
            $token = $authResult.AccessToken
            $headers = Get-Headers($token)
            $oneSuccessfulFetch = $False
        }
        elseif($statusCode -eq 429)
        {
            # throttled request, wait for a few seconds and retry
            Start-Sleep -5
        }
        elseif($statusCode -eq 403 -or $statusCode -eq 400 -or $statusCode -eq 401)
        {
            Write-Output "Please check the permissions of the user"
            break;
        }
        else {
            if ($retryCount -lt 5) {
                Write-Output "Retrying..."
                $retryCount++
            }
            else {
                Write-Output "Download request failed. Please try again in the future."
                break
            }
        }
     }
    Catch {
        $exType = $_.Exception.GetType().FullName
        $exMsg = $_.Exception.Message

        Write-Output "Exception: $_.Exception"
        Write-Output "Error Message: $exType"
        Write-Output "Error Message: $exMsg"

         if ($retryCount -lt 5) {
            Write-Output "Retrying..."
            $retryCount++
        }
        else {
            Write-Output "Download request failed. Please try again in the future."
            break
        }
    }

    Write-Output "--------------------------------------------------------------"
} while($url -ne $null)

GitHub Reference to Scripts

I have posted all the scripts used in this guide onto GitHub. If any updates are made in the future you will find them here. Please access through the link below.

https://github.com/Eunice-Chinchilla/ADFSLockoutTracking/tree/master

Conclusion

After exploring different methods and going down different paths to achieve our goal of tracking where the ADFS Account Lockouts were coming from I was able to deliver multiple solutions to our customer. It was a learning experience which allowed me to readjust when presented with blockers or obstacles. Through my journey I did not have too many sources to guide me and therefore I documented the steps I went through. Hopefully this guide provided you with some helpful information.

Thanks for reading!

Disclaimer
The sample scripts are not supported under any Microsoft standard support program or service. The sample scripts are provided AS IS without warranty of any kind. Microsoft further disclaims all implied warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The entire risk arising out of the use or performance of the sample scripts and documentation remains with you. In no event shall Microsoft, its authors, or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the sample scripts or documentation, even if Microsoft has been advised of the possibility of such damages.

Updated May 18, 2020

Version 1.0

EuniceChinchilla