RNA sequencing analysis on Azure using Nextflow: configuration files and benchmarking.

process {
    executor = 'azurebatch'
    container = 'quay.io/nextflow/bash'
    queue = 'Standard_D4_v3'
    withLabel:process_low   {queue = 'Standard_D4_v3'}
    withLabel:process_medium {queue = 'Standard_D8_v3'}
    withLabel:process_high {queue = 'Standard_D8_v3'}
    withLabel:process_gpu {queue = 'Standard_NV12s_v3'}
}

azure {
    storage {
      accountName = <accountName>
      accountKey = <token>
    }
    batch {
      location = 'francecentral'
      accountName = <accountName>
      accountKey = <token>
      allowPoolCreation = true //Nextflow automatically creates a pool of computing nodes to execute the jobs
      autoPoolMode = false
      deletePoolsOnCompletion = true //The pool is removed when the pipeline execution terminates

      pools {
          Standard_D4_v3 {
              autoScale = false
              vmType = 'Standard_D4_v3'
              vmCount = 2
              maxVmCount = 20
          }
          Standard_D8_v3 {
              autoScale = false
              vmType = 'Standard_D8_v3'
              vmCount = 2
              maxVmCount = 20
          }
          Standard_NV12s_v3 {
              autoScale = false
              vmType = 'Standard_NV12s_v3'
              vmCount = 2
              maxVmCount = 10
          }
      }
  }
}

Hi Venkat,

Thanks!

My configuration is quite similar, i've added a label for gpu computation, since i'm working on variant calling using deepvariant.

By setting the autoscale to false, the pipeline runs without problem but I get an error as soon as I set it to true.

By checking everywhere including github issues and nextflow community I didn't find any solution.

Thanks for your interest,

Tanguy