Mount Azure File Share to Azure Batch Pool via Azure PowerShell

[Bin_Yang]

Background

Azure Batch supports mounting Azure file share with an Azure Batch pool. Our official document only has samples for C#. In this blog, we will include the following content:

• Mount Azure file share via Azure PowerShell for Windows and Linux.

• How to access the mounting files for tasks. 

• How Azure Batch agent implements mounting.

• Troubleshoot the failure of mounting.

• Access to the mounting drive manually.

• Manually mount Azure file share via RDP/SSH.

Pre-requirement

• Prepare an Azure Batch account.

• Prepare an Azure Storage account with Azure Fileshare in the same region as the Batch account.

• Prepare Azure PowerShell or use Azure CloudShell from Portal.

Steps:

1. Log in to your subscription in Azure PowerShell:

Connect-AzAccount -Subscription "<subscriptionID>"

2. Get the context for your Azure Batch account:

$context = Get-AzBatchAccount -AccountName <batch-account-name>

3. For Windows, the following script will create an Azure Batch pool with these settings:

• Azure File share mounting disk “S”
• Image: “WindowsServer", "MicrosoftWindowsServer", "2016-Datacenter", "latest"
• VM size: STANDARD_D2_V2
• Dedicated Compute Nodes: 1
• Low priority Compute Nodes: 0

$fileShareConfig = New-Object -TypeName "Microsoft.Azure.Commands.Batch.Models.PSAzureFileShareConfiguration" -ArgumentList @("<Storage-Account-name>", https://<Storage-Account-name>.file.core.windows.net/batchfileshare1, "S", "Storage-Account-key")

$mountConfig = New-Object -TypeName "Microsoft.Azure.Commands.Batch.Models.PSMountConfiguration" -ArgumentList @($fileShareConfig)

$imageReference = New-Object -TypeName "Microsoft.Azure.Commands.Batch.Models.PSImageReference" -ArgumentList @("WindowsServer", "MicrosoftWindowsServer", "2016-Datacenter", "latest")

$configuration = New-Object -TypeName "Microsoft.Azure.Commands.Batch.Models.PSVirtualMachineConfiguration" -ArgumentList @($imageReference, "batch.node.windows amd64")

New-AzBatchPool -Id "<Pool-Name>" -VirtualMachineSize "STANDARD_D2_V2" -VirtualMachineConfiguration $configuration -TargetDedicatedComputeNodes 1 -MountConfiguration @($mountConfig) -BatchContext $Context

4. For Linux, the following script will create an Azure Batch pool with these settings:

• Azure File share mounting directory “S”
• Image: “ubuntuserver", "canonical", "18.04-lts", "latest"
• VM size: STANDARD_D2_V2
• Dedicated Compute Nodes: 1
• Low priority Compute Nodes: 0

$fileShareConfig = New-Object -TypeName "Microsoft.Azure.Commands.Batch.Models.PSAzureFileShareConfiguration" -ArgumentList @("<Storage-Account-name>", https://<Storage-Account-name>.file.core.windows.net/batchfileshare1, "S", "<Storage-Account-key>", "-o vers=3.0,dir_mode=0777,file_mode=0777,sec=ntlmssp")

$mountConfig = New-Object -TypeName "Microsoft.Azure.Commands.Batch.Models.PSMountConfiguration" -ArgumentList @($fileShareConfig)

$imageReference = New-Object -TypeName "Microsoft.Azure.Commands.Batch.Models.PSImageReference" -ArgumentList @("ubuntuserver", "canonical", "18.04-lts", "latest")

$configuration = New-Object -TypeName "Microsoft.Azure.Commands.Batch.Models.PSVirtualMachineConfiguration" -ArgumentList @($imageReference, "batch.node.ubuntu 18.04")

New-AzBatchPool -Id "<Pool-Name>" -VirtualMachineSize "STANDARD_D2_V2" -VirtualMachineConfiguration $configuration -TargetDedicatedComputeNodes 1 -MountConfiguration @($mountConfig) -BatchContext $Context

How to access the mount files

For the Azure Batch task user

For Windows:

1. We use the Drive path directly to access the files. 
2. For example, I have a file out.txt under path S:\folder1 as shown below:

3. The task can access the file by the following command:

cmd /c "more S:\folder1\out.txt & timeout /t 90 > NULL"

4. Then you will see the result in the output file:

For Linux: 

1. We can use the environment variable “AZ_BATCH_NODE_MOUNTS_DIR” or the path directly.
2. For example, here is the command of the task to access the file via environment variable:

/bin/bash -c 'more $AZ_BATCH_NODE_MOUNTS_DIR/S/folder1/out.txt; sleep 20s'

3. Here is the output file:

For RDP/SSH user

When the mounting is successful, you can manually access the drive S directly in Linux in the following path:

/mnt/batch/tasks/fsmounts/S

However, you will get access denied error when accessing drive S in Windows:

1. This is because the Azure Batch agent only grants access for Azure Batch tasks in Windows.
2. When RDP to the node, the user account doesn’t have access to the mounting drive.
3. You need to use cmdkey to add the credential for yourself in cmd:

cmdkey /add:"<storage-account-name>.file.core.windows.net" /user:"Azure\<storage-account-name>" /pass:"<storage-account-key>"

4. After the credential is added, you will be able to access the S drive directly.

Troubleshoot the failure of mounting

How Azure Batch agent implements mounting:

For Windows:

Azure Batch uses cmdkey to add credential for your Azure Batch account, then issues the mount command via “net use”.

net use S: \\<storage-account-name>.file.core.windows.net\<fileshare> /u:AZURE\<storage-account-name> <storage-account-key>

For Linux:

Batch agent installs package cifs-utils, then will issue the mount command.

How to check the mounting logs

1. When mounting failed, you may observe the following error:

2. We are not able to find more detailed information about the error via Azure Batch Portal or Azure Batch Explorer. So we need to RDP/SSH to the node and check related log files.
3. For Windows, we can connect to the node via Remote Desktop as shown below(the screenshot is from Batch Explorer):

4. The log file “fshare-S.log” is under path D:\batch\tasks\fsmounts.
5. In this sample, we can find the following messages:

CMDKEY: Credential added successfully.

System error 86 has occurred.

The specified network password is not correct.

You can refer to this document to troubleshoot Azure Files problems in Windows:

https://docs.microsoft.com/en-us/azure/storage/files/storage-troubleshoot-windows-file-connection-pr...

6. For Linux, we can connect to the node via SSH as shown below:

7. You can find the log file “fshare-S.log” under /mnt/batch/tasks/fsmounts.
8. In this sample, the account key was wrong and the message is “permission denied”.

You can refer to the following document to troubleshoot Azure Files problems in Linux:

https://docs.microsoft.com/en-us/azure/storage/files/storage-troubleshoot-linux-file-connection-prob...

If you are not able to RDP/SSH, you can check the batch logs directly.

1. Navigate to the node and click “upload batch logs” as shown below:

2. You will be able to download the logs from the selected Azure Storage account:

3. You can check the log named “agent-debug” and will see some error messages.
4. You will find the same error messages as shown below.

..20210322T113107.448Z.00000000-0000-0000-0000-000000000000.ERROR.agent.mount.filesystems.basefilesystem.basefilesystem.py.run_cmd_persist_output_async.59.2912.MainThread.3580.Mount command failed with exit code: 2, output:

CMDKEY: Credential added successfully.

System error 86 has occurred.

The specified network password is not correct.

5. It’s the same to get the logs for Linux.

Manually mount Azure File share

If you are not able to identify the cause of failure, you can RDP/SSH to the node and manually mount the Azure File share to narrow down this issue.

Here are the detailed steps:

1. You can create a pool without mounting configuration by the following command(let’s take Windows for example).

$imageReference = New-Object -TypeName "Microsoft.Azure.Commands.Batch.Models.PSImageReference" -ArgumentList @("WindowsServer", "MicrosoftWindowsServer", "2016-Datacenter", "latest")

$configuration = New-Object -TypeName "Microsoft.Azure.Commands.Batch.Models.PSVirtualMachineConfiguration" -ArgumentList @($imageReference, "batch.node.windows amd64")

New-AzBatchPool -Id "<Pool-Name>" -VirtualMachineSize "STANDARD_D2_V2" -VirtualMachineConfiguration $configuration -TargetDedicatedComputeNodes 1  -BatchContext $Context

2. After the node reaches idle status, you can connect to the node via RDP.
3. You can go to the Azure File Share Portal and get the Azure PowerShell command for mount as shown below:

4. The default the drive is "Z" in this example, you can change it to "S" if you want. 
5. Then you can enter the command inside the node and mount the Azure File share.

5. If there is any connection issue, you may observe the error message:

6. You can troubleshoot Networking related issue in this way.
7. When mouting Azure File Share in Linux, you can easily find the command for Linux in File Storage portal and other steps are the same.