Blog Post

Healthcare and Life Sciences Blog
4 MIN READ

RNA sequencing analysis on Azure using Nextflow: low-priority vs. dedicated machines comparison.

Olesya_Melnichenko's avatar
Mar 22, 2023

This is part two of the blog series on RNA sequencing analysis using Nextflow. In the first part we introduced configuration files required to run nf-core/rnaseq pipeline on dedicated virtual machines on Azure. We also presented benchmarking results for two machines series: Ddv4 and Edv4; and discussed possible cost savings due to switching from dedicated to low-cost machines.

 

In this post we describe configuration file change needed to run pipeline on low-cost virtual machines. We used updated configuration to benchmark the pipeline on Edv4-series low-priority machines. The dedicated vs. low-priority comparison shows that the cost to run a pipeline can be reduced by up to 82% without significant changes in pipeline run duration when using low-priority virtual machines.

 

Prerequisites

We highly recommend reading the first part of the blog series since it has detailed description of the benchmarking experiment and data used for it.

 

Updating configuration file

Azure Batch can automatically scale pools based on parameters that you define. The default scale formula that Nextflow uses is shown below, and it only defines the target number of dedicated nodes in the pool

 

// Get pool lifetime since creation.
lifespan = time() - time("{{poolCreationTime}}");
interval = TimeInterval_Minute * {{scaleInterval}}; 

// Compute the target nodes based on pending tasks.
// $PendingTasks == The sum of $ActiveTasks and $RunningTasks
$samples = $PendingTasks.GetSamplePercent(interval);
$tasks = $samples < 70 ? max(0, $PendingTasks.GetSample(1)) : max($PendingTasks.GetSample(1), avg($PendingTasks.GetSample(interval)));
$targetVMs = $tasks > 0 ? $tasks : max(0, $TargetDedicatedNodes/2);
targetPoolSize = max(0, min($targetVMs, {{maxVmCount}}));

// For first interval deploy 1 node, for other intervals scale up/down as per tasks.
$TargetDedicatedNodes = lifespan < interval ? {{vmCount}} : targetPoolSize;
$NodeDeallocationOption = taskcompletion;

 

We can change the type of pool nodes from dedicated to low-priority by modifying default formula and using scaleFormula option for pool configuration. The updated configuration file is shown below

 

// Scale formula to use low-priority nodes only.
lowPriorityScaleFormula = '''
    lifespan = time() - time("{{poolCreationTime}}");
    interval = TimeInterval_Minute * {{scaleInterval}};
    $samples = $PendingTasks.GetSamplePercent(interval);
    $tasks = $samples < 70 ? max(0, $PendingTasks.GetSample(1)) : max($PendingTasks.GetSample(1), avg($PendingTasks.GetSample(interval)));
    $targetVMs = $tasks > 0 ? $tasks : max(0, $TargetLowPriorityNodes/2);
    targetPoolSize = max(0, min($targetVMs, {{maxVmCount}}));
    $TargetLowPriorityNodes = lifespan < interval ? {{vmCount}} : targetPoolSize;
    $TargetDedicatedNodes = 0;
    $NodeDeallocationOption = taskcompletion;
'''

process {
    executor = 'azurebatch'
    queue = 'Standard_E2d_v4'
    withLabel:process_low {queue = 'Standard_E2d_v4'}
    withLabel:process_medium {queue = 'Standard_E8d_v4'}
    withLabel:process_high {queue = 'Standard_E16d_v4'}
    withLabel:process_high_memory {queue = 'Standard_E32d_v4'}
}

azure {
    storage {
        accountName = "<Your storage account name>"
        sasToken = "<Your storage account SAS Token>"
    }
    batch {
        location = "<Your location>"
        accountName = "<Your batch account name>"
        accountKey = "<Your batch account key>"
        autoPoolMode = false
        allowPoolCreation = true
        deletePoolsOnCompletion = true

        pools {
            Standard_E2d_v4 {
                autoScale = true
                vmType = 'Standard_E2d_v4'
                vmCount = 2
                maxVmCount = 20
                scaleFormula = lowPriorityScaleFormula
            }
            Standard_E8d_v4 {
                autoScale = true
                vmType = 'Standard_E8d_v4'
                vmCount = 2
                maxVmCount = 20
                scaleFormula = lowPriorityScaleFormula
            }
            Standard_E16d_v4 {
                autoScale = true
                vmType = 'Standard_E16d_v4'
                vmCount = 2
                maxVmCount = 20
                scaleFormula = lowPriorityScaleFormula
            }
            Standard_E32d_v4 {
                autoScale = true
                vmType = 'Standard_E32d_v4'
                vmCount = 2
                maxVmCount = 10
                scaleFormula = lowPriorityScaleFormula
            }
        }
    }
}

 

Note that scaleFormula option allows you to provide any custom formula, so you can also create pools with a mix of dedicated and low-cost machines if needed. To learn more about creating autoscale formulas check Azure Batch documentation.

 

Benchmarking Edv4-series: dedicated vs. low-priority machines

We used the updated configuration file to repeat benchmarking experiment on Edv4-series low-priority machines. The detailed experiment setup is described in the previous blog post. Sample and reference data, tools and pipeline versions, and pricing data were kept the same.

 

As we mentioned previously, the tradeoff for using low-cost machines in that those machines may be preempted at any time. To collect benchmarking data, we performed 30 pipeline runs, 10 runs per alignment option. The number of tasks in the pipeline depends on the alignment option: 36 tasks for STAR-RSEM and HISAT2, and 42 tasks for STAR-Salmon.  All 1140 tasks were completed on the first attempt, no preemption events were observed.

 

Average cost estimates and durations for each alignment option are presented on Fig.1-3; average changes are in Table 1. The results show that the cost to run a pipeline can be reduced by up to 82% without significant changes in pipeline completion time by using low-priority virtual machines.

 

     

Fig. 1: Average cost estimate and pipeline run duration for nf-core/rnaseq pipeline with STAR-Salmon alignment option; Edv4-series virtual machines.

 

 

     

Fig. 2: Average cost estimate and pipeline run duration for nf-core/rnaseq pipeline with STAR-RSEM alignment option; Edv4-series virtual machines.

 

 

     

Fig. 3: Average cost estimate and pipeline run duration for nf-core/rnaseq pipeline with HISAT2 alignment option; Edv4-series virtual machines.

 

Table 1: Average change in cost and duration for each alignment option when switching from dedicated to low-priority VMs, Edv4-series.

Alignment option

Average change, %

Cost

Duration

STAR-Salmon

-81

-2.8

STAR-RSEM

-82

1.6

HISAT2

-81

0.8

 

Currently Azure Batch offers two types of low-cost virtual machines: Spot VMs and Low-priority VMs. The type of node you get depends on your Batch account's pool allocation mode, which is settable during account creation. Accounts that use the Batch managed pool allocation mode always get low-priority VMs; accounts that use the user subscription pool allocation mode always get Spot VMs. Please keep in mind that low-priority nodes will be retired in September 2025.

 

For the benchmarking experiments we used an account with Batch managed pool allocation mode (default option) due to easier setup process. To replicate the experiments on Spot VMs you need to create Batch account with user subscription pool allocation mode; configuration file might be reused with no changes required.

Updated Mar 22, 2023
Version 2.0
No CommentsBe the first to comment