Azure Managed Lustre with Automatic Synchronisation to Azure BLOB Storage

Introduction

This blog post walks through how to setup an Azure Managed Lustre Filesystem (AMLFS) that will automatically synchronise to an Azure BLOB Storage container. The synchronisation is achieved using the Lustre HSM (Hierarchical Storage Management) interface combined with the Robinhood policy engine and a tool that reads the Lustre changelog and synchronises metadata with the archived storage. The lfsazsync repository on GitHub contains a Bicep template to deploy and setup a virtual machine for this purpose.

Disclaimer: The lfsazsync deployment is not a supported Microsoft product you are responsible for the deployment and operation of the solution. There are updates that need applying to AMLFS that will require a Support Request to be raised through the Azure Portal. These updates could effect the stabaility of AMLFS and customer requiring the same level of SLA should speak to their Microsoft representative.

Initial Deployment

The following is required before running the lfsazsync Bicep template:

• Virtual Network
• Azure BLOB Storage Account and container (HNS is not supported)
• AMLFS deployed without HSM enabled

The lfsazsync repository contains a test/infra.bicep example to create the required resources:

To deploy, first create a resource group, e.g.

TODO: set the variables below
resource_group=
location=
az group create --name $resource_group --location $location

Then deploy into this resource group:

az deployment group create --resource-group $resource_group --template-file test/infra.bicep

Note: The bicep file has parameters for names, ip ranges etc. that should be set if you do not want the default values.

Updating the AMLFS settings

Once deployment is complete, navigate to the Azure Portal, locate the AMLFS resource and click on "New Support Request". The following shows the suggested request to get AMLFS updated:

The lctl commands needed are listed here.

Deploying Azure BLOB Storage Synchronisation

The lfsazsync deployment sets up a single virtual machine for all tasks. The HSM copytools could be run on multiple virtual machines to increase transfer peformance. The bandwidth for archiving and retrieval is constrained to approximately half the network bandwidth available to the virtual machine. It is important to note that the same network will be utilized for both accessing the Lustre filesystem and accessing Azure Storage. This should be considered when deciding the virtual machine size. The virtual machine sizes and expected network performance is available here.

The Bicep template has the following parameters:

The SAS key can be generated using the following Azure CLI command:

# TODO: set the account name and container name below
account_name=
container_name=

start_date=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
expiry_date=$(date -u +"%Y-%m-%dT%H:%M:%SZ" --date "next month")

az storage container generate-sas \
   --account-name $account_name \
   --name $container_name \
   --permissions rwld \
   --start $start_date \
   --expiry $expiry_date \
   -o tsv

The following Azure CLI command can be used to get the subnet ID:

# TODO: set the variable below
resource_group=
vnet_name=
subnet_name=

az network vnet subnet show --resource-group $resource_group --vnet-name $vnet_name --name $subnet_name --query id --output tsv

The following Azure CLI command can be used to deploy the Bicep template (as an alterative to setting environment variables, the parameters could be set in a parameters.json file):

# TODO: set the variables below
resource_group=
subnet_id=
vmsku="Standard_D32ds_v4"
admin_user=
ssh_key=
lustre_mgs=
storage_account_name=
storage_container_name=
storage_sas_key=
ssh_port=
github_release="v1.0.1"
os="almalinux87"

az deployment group create \
    --resource-group $resource_group \
    --template-file lfsazsync.bicep \
    --parameters \
        subnet_id="$subnet_id" \
        vmsku=$vmsku \
        admin_user="$admin_user" \
        ssh_key="$ssh_key" \
        lustre_mgs=$lustre_mgs \
        storage_account_name=$storage_account_name \
        storage_container_name=$storage_container_name \
        storage_sas_key="$storage_sas_key" \
        ssh_port=$ssh_port \
        github_release=$github_release \
        os=$os

After this call completes the virtual machine will be deployed although it will take more time to install and import the metadata from Azure BLOB storage into the Lustre filesystem. The progress can be monitored by looking at the /var/log/cloud-init-output.log file on the virtual machine.

Monitoring

The install will set up three systemd services for lhsmd, robinhood and lustremetasync. The log files are located here:

• 'lhsmd': /var/log/lhsmd.log
• 'robinhood': /var/log/robinhood*.log
• 'lustremetasync': /var/log/lustremetasync.log

Default archive settings

The synchronisation parameters can be controlled through the Robinhood config file, /opt/robinhood/etc/robinhood.d/lustre.conf. Below are some of the default settings and their locations in the config file:

To update the config file, edit the file and then restart the robinhood service, systemctl restart robinhood.

The lustremetasync service is processing the Lustre ChangeLog continuously. Therefore, actions will happen immediately unless there is a lot of IO all at once where it may take a few minutes to catch up. The following operations will be handled:

• Create/delete directories

  Directories are created in BLOB storage as an empty object with the name of the directory. There is metadata on this file to indicate that it is a directory. The same object is deleted when removed on the filesystem.

• Create/delete symbolic links

  Symbolic links are create in BLOB storage as an empty object with the name of the symbolic link. There is metadata on this file to indicate that it is a symbolic link and this contains the path that it is linking to. The same object is deleted when removed on the filesystem.

• Moving files or directories

  Moving files or directories requires everything being moved to be restored to the Lustre filesystem. The files are then marked as dirty in their new location and the existing files are deleted from BLOB storage. Robinhood will handle archiving the files again in their new location.

• Updating metadata (e.g. ownership and permissions)

  The metadata will only be updated for archived files that isn't modified. Modified files will have the metadata set when Robinhood updated the archived file.

References

• Azure Managed Lustre File System Documentation
• GitHub repositories
  • lfsazsync
  • Robinhood
  • lemur