How to Automate Cross-OS File Fixes with Azure Automation and PowerShell

Start your free Azure account here to follow along.

 

Introduction 

Hi, I'm Raunak Dev, a Microsoft Learn Student Ambassador from India, majoring in Electrical and Computer Engineering at Amrita School of Engineering – Amritapuri. 

Have you ever wondered why your scripts suddenly break when switching between operating systems? When collaborating across Windows and Linux, subtle OS differences can cause major headaches. For example, mismatched line endings and file permissions can break your scripts, delay projects, and frustrate teams. 

In this tutorial, I'll show you how to build a serverless file fixer in Azure using Azure Automation, PowerShell, and Blob Storage—so you can say goodbye to manual fixes and hello to streamlined, error-free workflows.

Prerequisites 

Before you begin, ensure you have the following: 

1. An active Azure subscription (start for free). 
2. A Resource Group to organize your Azure resources.   
3. Registration for all appropriate service providers, such as Microsoft.EventGrid. Azure Resource Manager. 
4. Basic familiarity with PowerShell scripting. 

The Problem: Why File Compatibility Matters 

When files are shared between Windows and Linux, two common issues arise: 

• CRLF vs. LF Line Endings: Windows uses Carriage Return + Line Feed (CRLF), while Linux uses Line Feed (LF) only. This mismatch can cause script errors. 
• File Permissions: Linux requires execute permissions (e.g., using chmod +x script.sh), and these permissions often don’t translate automatically when moving files. 

Manually fixing these problems every time is tedious and error-prone. That’s why automating the process not only saves time but also ensures consistency. 

The Solution: Build a Serverless File Fixer in Azure 

In this tutorial, you’ll create a serverless solution that automatically detects and fixes file issues. Using Azure Automation and PowerShell, the process involves: 

• Azure Automation: Run your scripts on-demand, on a schedule, or via event triggers. 
• Blob Storage: Store and monitor files to detect compatibility issues. 
• Azure Event Grid: Automatically trigger your automation runbooks when a new file is uploaded. (Optional)
• Azure Cloud Shell: Use a browser-based terminal for quick, zero-setup access. (Optional) 

Step-by-Step Tutorial 

Step 1: Create a Resource Group 

If you haven’t already created a resource group, follow these steps: 

• Open the Azure Portal. 
• Navigate to Resource groups and click Create. 
• Enter a resource group name (e.g., AutomaticFileFixer) and choose your region. 
• Click Review + Create and then Create. 

Step 2: Create a Storage Account within the Resource Group 

• In the Azure Portal, search for Storage Accounts and click Create. 

• Under the Basics tab, ensure you select your resource group (e.g., AutomaticFileFixer).  
• Configure the following: 
  • Subscription: Your active subscription 
  • Name: filefixerstorage 
  • Region: Use the same region as your resource group. 
  • Primary service:- Azure Blob Storage or Data Lake Storage Gen 2 

• Under the Advanced tab, enable Blob anonymous access. (If available) 
• Click Review + Create, then Create. 

Step 3: Create an Automation Account  

• In the Azure Portal, search for Automation Accounts and click Create. 

• Under the Basics tab, ensure you select your resource group (AutomaticFileFixer). 
• Configure the account by providing a name (FileFixerAutomation) and selecting the same region. 

• Click Review + Create, then Create. 

Step 4: Configure IAM Permissions 

Grant your Automation Account access to the Storage Account by assigning the necessary roles. 

Assign Storage Blob Data Contributor Role 

• Navigate to your Storage Account (filefixerstorage). 
• In the left-hand menu, click Access control (IAM). 
• Click + Add and select Add role assignment. 

• In the Role dropdown, choose Storage Blob Data Contributor. 

• Under Assign access to, select Managed identity, and then click on +Select members. 

• In the Select managed identities dialogue box, select Automation Account   

• Click Select, then Review + assign. 

Step 5: Upload Files to Blob Storage 

• Navigate to your storage account. 
• In the Overview page, click + Upload  
• Upload a test file (for example, a script.sh file with CRLF line endings). 

• Under the Select an existing container dropdown, click Create new. 
• In the New container Section, configure as: 
  • Name: uploads 
  • Anonymous access level: Container (anonymous read access for containers and blobs) 
  • Click Upload 

Step 6: Create the PowerShell Runbook 

• In your Automation Account, click on Runbooks and then + Create a runbook. 

• Name the runbook FixCRLFRunbook and choose PowerShell as the type. Select Runtime Version 7.2. 

• Open the newly created Runbook by clicking Manage a runbook on the Get Started page. 
• Locate your Runbook. Once opened, paste the following code in the editor: 

param(
    [Parameter(Mandatory=$false)]
    [object] $WebhookData
)

# Authenticate to Azure using Managed Identity. This allows the script to access Azure resources without explicit credentials.
Connect-AzAccount -Identity

# Define storage account details.
$storageAccountName = "filefixerstorage"          # The name of your Azure Storage Account.
$resourceGroupName = "AutomaticFileFixer"         # The Resource Group where your storage account is located.
$containerName = "uploads"                        # The Blob container name that holds the files.

# Retrieve the storage account context which contains the necessary authentication and connection details.
$storageAccount = Get-AzStorageAccount -ResourceGroupName $resourceGroupName -Name $storageAccountName
$ctx = $storageAccount.Context

# Retrieve all blobs within the specified container using the storage account context.
$blobs = Get-AzStorageBlob -Container $containerName -Context $ctx

# Loop through each blob (file) in the container.
foreach ($blob in $blobs) {
    try {
        # Create a temporary file path to download the blob for processing.
        $tempPath = [System.IO.Path]::GetTempFileName()

        # Download the blob content to the temporary file.
        Get-AzStorageBlobContent -Container $containerName -Blob $blob.Name -Context $ctx -Destination $tempPath -Force

        # Read the entire content of the temporary file as a single string.
        $content = Get-Content $tempPath -Raw

        # Replace Windows-style CRLF line endings with Linux-style LF line endings.
        $content = $content -replace "`r`n", "`n"

        # Write the updated content back to the temporary file.
        Set-Content -Path $tempPath -Value $content -Force

        # Check if the blob is a shell script (ends with .sh).
        if ($blob.Name -match "\.sh$") {
            # For shell scripts, update the blob and set its content type to indicate it's a shell script.
            Set-AzStorageBlobContent -File $tempPath -Container $containerName -Blob $blob.Name -Context $ctx -Force `
                -Properties @{ ContentType = "text/x-shellscript" }
        }
        else {
            # For non-shell script files, update the blob without changing the content type.
            Set-AzStorageBlobContent -File $tempPath -Container $containerName -Blob $blob.Name -Context $ctx -Force
        }

        # Output a message indicating the blob was processed successfully.
        Write-Output "Processed $($blob.Name) successfully."
    }
    catch {
        # Output an error message if processing fails for the current blob.
        Write-Error "Failed to process $($blob.Name): $_"
    }
    finally {
        # Clean up: Delete the temporary file to free up resources.
        if (Test-Path $tempPath) { Remove-Item $tempPath -Force }
    }
}

 

Step 7: Publish and Test the Runbook 

• Save & Publish: Click Save, then Publish to finalize your runbook. 
• Run Manually: Click Start to execute the runbook immediately. 
• Check the Output: Navigate to Jobs, select the latest job, and review the output to confirm that the files have been processed correctly. 

Step 8.1: Create a Webhook for Your Runbook 

• In the Azure Portal, navigate to your Automation Account and select Runbooks. 
• Choose your runbook (e.g., FixCRLFRunbook). 
• Click Add Webhook and select Create New Webhook. 
• Enter the details: 
  • Name: filefixerWebhook 
  • Enabled: Ensure it is enabled. 
  • Leave any additional parameters as default. 

• Copy the Webhook URL immediately! (store it securely as it is only shown once). 
• Click OK to save the webhook. 

Step 8.2: Create an Event Grid Subscription 

Before setting up the event subscription, verify that your Azure subscription is registered with the Microsoft.EventGrid service provider. Learn more here, Azure Resource Providers and Types. 

• Navigate to your Storage Account (filefixerstorage) in the Azure Portal. 
• In the left-hand menu, select Events and then click + Event Subscription. 

• Configure the subscription as follows: 
• • Name: AutoFixCRLF 
  • Event Types: Select Blob Created. 
  • Endpoint Type: Choose Webhook.  
  • Endpoint: Paste the webhook URL you copied earlier. Click Create. 

Step 8.3: Grant Event Grid Permission to Trigger the Runbook 

(Similar to Step 4: Configure IAM Permissions, but for Event Grid) 

• In the Azure Portal, go to your Automation Account. 
• Click Access Control (IAM). 
• Click + Add and select Add role assignment. 
• In the Role dropdown, choose Azure Event Grid Data Sender.
• Under Assign access to, select the appropriate option (if available, search for Event Grid in the assignee list). 
• Click Review + assign to complete the permission assignment. 

Step 8.4: Test the Integration 

• Upload a test file (for example, testfile.txt with CRLF line endings) to the uploads container in your Storage Account. 
• Verify that the runbook is triggered automatically: 
• In your Automation Account, navigate to Jobs and confirm that a new job appears within 2 minutes. 
• Validate the fix by downloading the processed file from Blob Storage and confirming that the line endings have been converted to LF. 

Final Script Adjustment 

To optimize performance, modify your Runbook to process only the newly uploaded blob instead of scanning the entire container. 

# Extract the blob name from the Event Grid data.
# The webhook payload is converted from JSON to a PowerShell object.
$body = $WebhookData.RequestBody | ConvertFrom-Json

# Extract the blob name by splitting the URL and selecting the last segment.
$blobName = $body.data.url.Split('/')[-1]

# Process only the newly uploaded blob.
# Create a storage context using the details defined in earlier steps (replace "..." with actual parameters).
$ctx = New-AzStorageContext ... # Use earlier steps to create the context

# Retrieve the blob from the 'uploads' container using its name.
$blob = Get-AzStorageBlob -Container "uploads" -Blob $blobName -Context $ctx

# Apply CRLF fixes and set the appropriate permissions.
# (Insert the code from previous steps here to modify file content and update blob properties.)
...

Conclusion 

By following this approach, you’ve created a serverless file fixer in Azure that automates the time-consuming task of correcting CRLF and LF line ending issues while also managing file permissions for Linux scripts. This solution not only saves time but it also reduces errors while working across different operating systems. 

Ready to streamline your workflows further? Explore additional Azure Automation capabilities and integrate with services like Azure Logic Apps or Azure Monitor for even more robust solutions. 

Start your free Azure account here to begin automating today! 

For more detailed documentation on Azure Automation and related services, visit the Azure Automation Documentation.