Home
%3CLINGO-SUB%20id%3D%22lingo-sub-1221351%22%20slang%3D%22en-US%22%3EConfiguring%20InfiniBand%20for%20Ubuntu%20HPC%20and%20GPU%20VMs%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1221351%22%20slang%3D%22en-US%22%3E%3CP%3EThere%20has%20been%20greater%20interest%20in%20the%20usage%20of%20the%20%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fazure-compute%2Fazure-centos-7-6-7-7-hpc-images%2Fba-p%2F977094%22%20target%3D%22_blank%22%20rel%3D%22noopener%22%3EHPC%20optimized%20VM%20images%3C%2FA%3E%20that%20we%20publish%20due%20to%20the%3A%3C%2FP%3E%0A%3COL%3E%0A%3CLI%3EGA%20of%20SR-IOV%20enabled%20HPC%20VMs%20(HB%2C%20HC%2C%20%3CA%20href%3D%22https%3A%2F%2Fazure.microsoft.com%2Fen-us%2Fblog%2Fazure-hbv2-virtual-machines-eclipse-80000-cores-for-mpi-hpc%2F%22%20target%3D%22_self%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%22%3EHB_v2%3C%2FA%3E)%2C%3C%2FLI%3E%0A%3CLI%3ERecent%20platform%20update%20to%20make%20%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fazure-compute%2Fplatform-updates-for-hpc-and-ai-vms-on-azure%2Fba-p%2F937129%22%20target%3D%22_self%22%3ENCr_v3%20SR-IOV%20enabled%2C%3C%2FA%3E%3C%2FLI%3E%0A%3CLI%3E%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fvirtual-machines%2Fndv2-series%3Ftoc%3D%2Fazure%2Fvirtual-machines%2Flinux%2Ftoc.json%26amp%3Bbc%3D%2Fazure%2Fvirtual-machines%2Flinux%2Fbreadcrumb%2Ftoc.json%22%20target%3D%22_self%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%22%3EGA%20of%20NDr_v2%3C%2FA%3E%3C%2FLI%3E%0A%3C%2FOL%3E%0A%3CP%3EWhile%20those%20images%20(CentOS-HPC%207.6%2C%207.7)%20are%20originally%20targeted%20for%20use%20on%20the%20SR-IOV%20enabled%20HPC%20VMs%20(HB%2C%20HC%2C%20HB_v2)%2C%20conceptually%2C%20they%20are%20useful%20for%20the%20other%20now%20SR-IOV%20enabled%20GPU%20VMs%20(NCr_v3%2C%20NDr_v2)%20too.%20Note%20that%20the%20GPU%20VMs%20would%20additionally%20require%20the%20Nvidia%20GPU%20drivers%20(%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fvirtual-machines%2Fextensions%2Fhpccompute-gpu-linux%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3EVM%20extension%3C%2FA%3E%2C%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fvirtual-machines%2Flinux%2Fn-series-driver-setup%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3Emanually%3C%2FA%3E).%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ETypically%20we%20find%20that%20users%20of%20the%20HPC%20VMs%20running%20traditional%20HPC%20applications%20tend%20to%20utilize%20CentOS%20as%20their%20preferred%20OS.%20While%20users%20of%20AI%2FML%20applications%20running%20on%20the%20GPU%20VMs%20tend%20to%20prefer%20Ubuntu%20as%20the%20OS.%20The%20CentOS-HPC%20VM%20OS%20images%20(%26gt%3B%3D7.6%20for%20the%20SR-IOV%20enabled%20VMs%2C%20and%20%26lt%3B%3D7.5%20for%20the%20non-SR-IOV%20enabled%20VMs)%20provide%20a%20ready%20to%20use%20VM%20image%20with%20the%20appropriate%20drivers%20and%20MPI%20runtimes.%20Such%20a%20pre-packaged%20and%20ready%20to%20use%20experience%20isn't%20yet%20available%20for%20Ubuntu%20on%20Azure.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThis%20article%20attempts%20to%20consolidate%20guidance%20on%20configuring%20InfiniBand%20(IB)%20for%20Ubuntu%20across%20both%20SR-IOV%20and%20non-SR-IOV%20enabled%20%3CSPAN%20style%3D%22display%3A%20inline%20!important%3B%20float%3A%20none%3B%20background-color%3A%20%23ffffff%3B%20color%3A%20%23333333%3B%20cursor%3A%20text%3B%20font-family%3A%20inherit%3B%20font-size%3A%2016px%3B%20font-style%3A%20normal%3B%20font-variant%3A%20normal%3B%20font-weight%3A%20300%3B%20letter-spacing%3A%20normal%3B%20line-height%3A%201.7142%3B%20orphans%3A%202%3B%20text-align%3A%20left%3B%20text-decoration%3A%20none%3B%20text-indent%3A%200px%3B%20text-transform%3A%20none%3B%20-webkit-text-stroke-width%3A%200px%3B%20white-space%3A%20normal%3B%20word-spacing%3A%200px%3B%22%3EHPC%20and%20GPU%20VMs%3C%2FSPAN%3E.%20Specifically%20it%20will%20focus%20on%20getting%20the%20right%20drivers%20setup%20and%20in%20bringing%20up%20the%20appropriate%20IB%20interface%20on%20the%20VMs.%20At%20the%20time%20of%20writing%2C%20the%20following%20steps%20at%20least%20apply%20to%20Ubuntu%2018.04%20LTS%20image%20by%20Canonical%20on%20the%20Azure%20Marketplace.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CFONT%20size%3D%224%22%3ENon-%20SR-IOV%20enabled%20VMs%3C%2FFONT%3E%3C%2FP%3E%0A%3COL%20style%3D%22box-sizing%3A%20border-box%3B%20color%3A%20%2324292e%3B%20font-family%3A%20-apple-system%2CBlinkMacSystemFont%2CSegoe%20UI%2CHelvetica%2CArial%2Csans-serif%2CApple%20Color%20Emoji%2CSegoe%20UI%20Emoji%3B%20font-size%3A%2016px%3B%20font-style%3A%20normal%3B%20font-variant%3A%20normal%3B%20font-weight%3A%20400%3B%20letter-spacing%3A%20normal%3B%20margin-bottom%3A%200px%3B%20margin-top%3A%200px%3B%20orphans%3A%202%3B%20padding-left%3A%202em%3B%20text-align%3A%20left%3B%20text-decoration%3A%20none%3B%20text-indent%3A%200px%3B%20text-transform%3A%20none%3B%20-webkit-text-stroke-width%3A%200px%3B%20white-space%3A%20normal%3B%20word-spacing%3A%200px%3B%22%3E%0A%3CLI%20style%3D%22box-sizing%3A%20border-box%3B%22%3E%3CP%20style%3D%22box-sizing%3A%20border-box%3B%20margin-bottom%3A%2016px%3B%20margin-top%3A%2016px%3B%22%3EInstall%20dapl%20(and%20its%20dependencies%20rdma_cm%2C%20ibverbs)%2C%20and%20user%20mode%20mlx4%20library.%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3Esudo%20apt-get%20update%0Asudo%20apt-get%20install%20libdapl2%20libmlx4-1%3C%2FCODE%3E%3C%2FPRE%3E%3C%2FLI%3E%0A%3CLI%20style%3D%22box-sizing%3A%20border-box%3B%20margin-top%3A%200.25em%3B%22%3E%0A%3CP%20style%3D%22box-sizing%3A%20border-box%3B%20margin-bottom%3A%2016px%3B%20margin-top%3A%2016px%3B%22%3EIn%20%2Fetc%2Fwaagent.conf%2C%20enable%20RDMA%20by%20uncommenting%20the%20following%20configuration%20lines%20(root%20access)%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3EOS.EnableRDMA%3Dy%0AOS.UpdateRdmaDriver%3Dy%3C%2FCODE%3E%3C%2FPRE%3E%3C%2FLI%3E%0A%3CLI%20style%3D%22box-sizing%3A%20border-box%3B%20margin-top%3A%200.25em%3B%22%3E%3CSPAN%20style%3D%22display%3A%20inline%20!important%3B%20float%3A%20none%3B%20background-color%3A%20%23ffffff%3B%20color%3A%20%23333333%3B%20cursor%3A%20text%3B%20font-family%3A%20inherit%3B%20font-size%3A%2016px%3B%20font-style%3A%20normal%3B%20font-variant%3A%20normal%3B%20font-weight%3A%20300%3B%20letter-spacing%3A%20normal%3B%20line-height%3A%201.7142%3B%20orphans%3A%202%3B%20text-align%3A%20left%3B%20text-decoration%3A%20none%3B%20text-indent%3A%200px%3B%20text-transform%3A%20none%3B%20-webkit-text-stroke-width%3A%200px%3B%20white-space%3A%20normal%3B%20word-spacing%3A%200px%3B%22%3ERestart%20the%20waagent%20service%3C%2FSPAN%3E%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3Esudo%20systemctl%20restart%20walinuxagent.service%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3C%2FLI%3E%0A%3C%2FOL%3E%0A%3CP%3E%3CSPAN%20style%3D%22display%3A%20inline%20!important%3B%20float%3A%20none%3B%20background-color%3A%20%23ffffff%3B%20color%3A%20%23333333%3B%20cursor%3A%20text%3B%20font-family%3A%20inherit%3B%20font-size%3A%2016px%3B%20font-style%3A%20normal%3B%20font-variant%3A%20normal%3B%20font-weight%3A%20300%3B%20letter-spacing%3A%20normal%3B%20line-height%3A%201.7142%3B%20orphans%3A%202%3B%20text-align%3A%20left%3B%20text-decoration%3A%20none%3B%20text-indent%3A%200px%3B%20text-transform%3A%20none%3B%20-webkit-text-stroke-width%3A%200px%3B%20white-space%3A%20normal%3B%20word-spacing%3A%200px%3B%22%3EThe%20IB%20interface%20eth1%20should%20come%20up%20with%20an%20RDMA%20IP%20address.%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20style%3D%22display%3A%20inline%20!important%3B%20float%3A%20none%3B%20background-color%3A%20%23ffffff%3B%20color%3A%20%23333333%3B%20cursor%3A%20text%3B%20font-family%3A%20inherit%3B%20font-size%3A%2016px%3B%20font-style%3A%20normal%3B%20font-variant%3A%20normal%3B%20font-weight%3A%20300%3B%20letter-spacing%3A%20normal%3B%20line-height%3A%201.7142%3B%20orphans%3A%202%3B%20text-align%3A%20left%3B%20text-decoration%3A%20none%3B%20text-indent%3A%200px%3B%20text-transform%3A%20none%3B%20-webkit-text-stroke-width%3A%200px%3B%20white-space%3A%20normal%3B%20word-spacing%3A%200px%3B%22%3EThe%20IB%20related%20kernel%20modules%20are%20not%20auto-loaded%20on%20Ubuntu%20anymore.%20This%20is%20a%20departure%20from%20earlier%20practice%20where%20the%20kernel%20modules%20were%20built%20into%20the%20image.%20Now%20these%20are%20available%20as%20loadable%20modules%20so%20that%20a%20user%20can%20install%20%3C%2FSPAN%3E%3CA%20style%3D%22background-color%3A%20transparent%3B%20box-sizing%3A%20border-box%3B%20color%3A%20%23146cac%3B%20font-family%3A%20%26amp%3Bquot%3B%20segoeui%26amp%3Bquot%3B%2C%26amp%3Bquot%3Blato%26amp%3Bquot%3B%2C%26amp%3Bquot%3Bhelvetica%20neue%26amp%3Bquot%3B%2Chelvetica%2Carial%2Csans-serif%3B%20font-size%3A%2016px%3B%20font-style%3A%20normal%3B%20font-variant%3A%20normal%3B%20font-weight%3A%20300%3B%20letter-spacing%3A%20normal%3B%20orphans%3A%202%3B%20text-align%3A%20left%3B%20text-decoration%3A%20underline%3B%20text-indent%3A%200px%3B%20text-transform%3A%20none%3B%20-webkit-text-stroke-width%3A%200px%3B%20white-space%3A%20normal%3B%20word-spacing%3A%200px%3B%22%20href%3D%22https%3A%2F%2Fwww.mellanox.com%2Fproducts%2Finfiniband-drivers%2Flinux%2Fmlnx_ofed%22%20target%3D%22_self%22%20rel%3D%22nofollow%20noopener%20noreferrer%20noopener%20noreferrer%22%3EMellanox%20OFED%20driver%3C%2FA%3E%3CSPAN%20style%3D%22display%3A%20inline%20!important%3B%20float%3A%20none%3B%20background-color%3A%20%23ffffff%3B%20color%3A%20%23333333%3B%20cursor%3A%20text%3B%20font-family%3A%20inherit%3B%20font-size%3A%2016px%3B%20font-style%3A%20normal%3B%20font-variant%3A%20normal%3B%20font-weight%3A%20300%3B%20letter-spacing%3A%20normal%3B%20line-height%3A%201.7142%3B%20orphans%3A%202%3B%20text-align%3A%20left%3B%20text-decoration%3A%20none%3B%20text-indent%3A%200px%3B%20text-transform%3A%20none%3B%20-webkit-text-stroke-width%3A%200px%3B%20white-space%3A%20normal%3B%20word-spacing%3A%200px%3B%22%3E.%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%20style%3D%22box-sizing%3A%20border-box%3B%20color%3A%20%23333333%3B%20font-family%3A%20inherit%3B%20font-size%3A%2016px%3B%20font-style%3A%20normal%3B%20font-variant%3A%20normal%3B%20font-weight%3A%20300%3B%20letter-spacing%3A%20normal%3B%20line-height%3A%201.7142%3B%20orphans%3A%202%3B%20text-align%3A%20left%3B%20text-decoration%3A%20none%3B%20text-indent%3A%200px%3B%20text-transform%3A%20none%3B%20-webkit-text-stroke-width%3A%200px%3B%20white-space%3A%20normal%3B%20word-spacing%3A%200px%3B%20margin%3A%200px%3B%22%3E%3CFONT%20size%3D%224%22%20style%3D%22box-sizing%3A%20border-box%3B%22%3ESR-IOV%20enabled%20VMs%20with%20inbox%20driver%3C%2FFONT%3E%3C%2FP%3E%0A%3COL%20style%3D%22box-sizing%3A%20border-box%3B%20color%3A%20%2324292e%3B%20font-family%3A%20-apple-system%2CBlinkMacSystemFont%2CSegoe%20UI%2CHelvetica%2CArial%2Csans-serif%2CApple%20Color%20Emoji%2CSegoe%20UI%20Emoji%3B%20font-size%3A%2016px%3B%20font-style%3A%20normal%3B%20font-variant%3A%20normal%3B%20font-weight%3A%20400%3B%20letter-spacing%3A%20normal%3B%20margin-bottom%3A%200px%3B%20margin-top%3A%200px%3B%20orphans%3A%202%3B%20padding-left%3A%202em%3B%20text-align%3A%20left%3B%20text-decoration%3A%20none%3B%20text-indent%3A%200px%3B%20text-transform%3A%20none%3B%20-webkit-text-stroke-width%3A%200px%3B%20white-space%3A%20normal%3B%20word-spacing%3A%200px%3B%22%3E%0A%3CLI%20style%3D%22box-sizing%3A%20border-box%3B%20font-family%3A%20%26amp%3Bquot%3B%22%3E%0A%3CP%20style%3D%22box-sizing%3A%20border-box%3B%20font-family%3A%20inherit%3B%20font-size%3A%2016px%3B%20font-weight%3A%20300%3B%20line-height%3A%201.7142%3B%20margin%3A%2016px%200px%2016px%200px%3B%22%3ELoad%20following%20kernel%20modules%20(either%20mpdprob%20or%20edit%20%2Fetc%2Fmodules)%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3Eib_uverbs%0Ardma_ucm%0Aib_umad%0Aib_ipoib%3C%2FCODE%3E%3C%2FPRE%3E%3C%2FLI%3E%0A%3CLI%20style%3D%22box-sizing%3A%20border-box%3B%20font-family%3A%20%26amp%3Bquot%3B%22%3E%0A%3CP%20style%3D%22box-sizing%3A%20border-box%3B%20font-family%3A%20inherit%3B%20font-size%3A%2016px%3B%20font-weight%3A%20300%3B%20line-height%3A%201.7142%3B%20margin%3A%2016px%200px%2016px%200px%3B%22%3EIn%20%2Fetc%2Fwaagent.conf%2C%20enable%20RDMA%20by%20uncommenting%20the%20following%20configuration%20lines%20(root%20access)%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3EOS.EnableRDMA%3Dy%3C%2FCODE%3E%3C%2FPRE%3E%3C%2FLI%3E%0A%3CLI%20style%3D%22box-sizing%3A%20border-box%3B%20font-family%3A%20%26amp%3Bquot%3B%20segoeui%26amp%3Bquot%3B%2C%26amp%3Bquot%3Blato%26amp%3Bquot%3B%2C%26amp%3Bquot%3Bhelvetica%20neue%26amp%3Bquot%3B%2Chelvetica%2Carial%2Csans-serif%3B%20margin-top%3A%200.25em%3B%22%3E%0A%3CP%20style%3D%22box-sizing%3A%20border-box%3B%20font-family%3A%20inherit%3B%20font-size%3A%2016px%3B%20font-weight%3A%20300%3B%20line-height%3A%201.7142%3B%20margin%3A%2016px%200px%2016px%200px%3B%22%3EReboot%20VM%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3Esudo%20reboot%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3C%2FLI%3E%0A%3C%2FOL%3E%0A%3CP%3E%3CSPAN%20style%3D%22background-color%3A%20%23ffffff%3B%20box-sizing%3A%20border-box%3B%20color%3A%20%23333333%3B%20cursor%3A%20text%3B%20display%3A%20inline%3B%20float%3A%20none%3B%20font-family%3A%20inherit%3B%20font-size%3A%2016px%3B%20font-style%3A%20normal%3B%20font-variant%3A%20normal%3B%20font-weight%3A%20300%3B%20letter-spacing%3A%20normal%3B%20line-height%3A%201.7142%3B%20orphans%3A%202%3B%20text-align%3A%20left%3B%20text-decoration%3A%20none%3B%20text-indent%3A%200px%3B%20text-transform%3A%20none%3B%20-webkit-text-stroke-width%3A%200px%3B%20white-space%3A%20normal%3B%20word-spacing%3A%200px%3B%22%3EThe%20IB%20interface%20ib0%20should%20come%20up%20with%20an%20RDMA%20IP%20address.%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%20style%3D%22box-sizing%3A%20border-box%3B%20color%3A%20%23333333%3B%20font-family%3A%20inherit%3B%20font-size%3A%2016px%3B%20font-style%3A%20normal%3B%20font-variant%3A%20normal%3B%20font-weight%3A%20300%3B%20letter-spacing%3A%20normal%3B%20line-height%3A%201.7142%3B%20orphans%3A%202%3B%20text-align%3A%20left%3B%20text-decoration%3A%20none%3B%20text-indent%3A%200px%3B%20text-transform%3A%20none%3B%20-webkit-text-stroke-width%3A%200px%3B%20white-space%3A%20normal%3B%20word-spacing%3A%200px%3B%20margin%3A%200px%3B%22%3E%3CFONT%20size%3D%224%22%20style%3D%22box-sizing%3A%20border-box%3B%22%3ESR-IOV%20enabled%20VMs%20with%20OFED%20driver%3C%2FFONT%3E%3C%2FP%3E%0A%3COL%20style%3D%22box-sizing%3A%20border-box%3B%20color%3A%20%2324292e%3B%20font-family%3A%20-apple-system%2CBlinkMacSystemFont%2CSegoe%20UI%2CHelvetica%2CArial%2Csans-serif%2CApple%20Color%20Emoji%2CSegoe%20UI%20Emoji%3B%20font-size%3A%2016px%3B%20font-style%3A%20normal%3B%20font-variant%3A%20normal%3B%20font-weight%3A%20400%3B%20letter-spacing%3A%20normal%3B%20margin-bottom%3A%200px%3B%20margin-top%3A%200px%3B%20orphans%3A%202%3B%20padding-left%3A%202em%3B%20text-align%3A%20left%3B%20text-decoration%3A%20none%3B%20text-indent%3A%200px%3B%20text-transform%3A%20none%3B%20-webkit-text-stroke-width%3A%200px%3B%20white-space%3A%20normal%3B%20word-spacing%3A%200px%3B%22%3E%0A%3CLI%20style%3D%22box-sizing%3A%20border-box%3B%20font-family%3A%20%26amp%3Bquot%3B%22%3E%0A%3CP%20style%3D%22box-sizing%3A%20border-box%3B%20font-family%3A%20inherit%3B%20font-size%3A%2016px%3B%20font-weight%3A%20300%3B%20line-height%3A%201.7142%3B%20margin%3A%2016px%200px%2016px%200px%3B%22%3E%3CSPAN%20style%3D%22display%3A%20inline%20!important%3B%20float%3A%20none%3B%20background-color%3A%20%23ffffff%3B%20color%3A%20%23333333%3B%20cursor%3A%20text%3B%20font-family%3A%20inherit%3B%20font-size%3A%2016px%3B%20font-style%3A%20normal%3B%20font-variant%3A%20normal%3B%20font-weight%3A%20300%3B%20letter-spacing%3A%20normal%3B%20line-height%3A%201.7142%3B%20orphans%3A%202%3B%20text-align%3A%20left%3B%20text-decoration%3A%20none%3B%20text-indent%3A%200px%3B%20text-transform%3A%20none%3B%20-webkit-text-stroke-width%3A%200px%3B%20white-space%3A%20normal%3B%20word-spacing%3A%200px%3B%22%3EThe%20appropriate%20%3C%2FSPAN%3E%3CA%20style%3D%22background-color%3A%20transparent%3B%20box-sizing%3A%20border-box%3B%20color%3A%20%23146cac%3B%20font-family%3A%20%26amp%3Bquot%3B%20segoeui%26amp%3Bquot%3B%2C%26amp%3Bquot%3Blato%26amp%3Bquot%3B%2C%26amp%3Bquot%3Bhelvetica%20neue%26amp%3Bquot%3B%2Chelvetica%2Carial%2Csans-serif%3B%20font-size%3A%2016px%3B%20font-style%3A%20normal%3B%20font-variant%3A%20normal%3B%20font-weight%3A%20300%3B%20letter-spacing%3A%20normal%3B%20orphans%3A%202%3B%20text-align%3A%20left%3B%20text-decoration%3A%20underline%3B%20text-indent%3A%200px%3B%20text-transform%3A%20none%3B%20-webkit-text-stroke-width%3A%200px%3B%20white-space%3A%20normal%3B%20word-spacing%3A%200px%3B%22%20href%3D%22https%3A%2F%2Fwww.mellanox.com%2Fproducts%2Finfiniband-drivers%2Flinux%2Fmlnx_ofed%22%20target%3D%22_self%22%20rel%3D%22nofollow%20noopener%20noreferrer%20noopener%20noreferrer%22%3EMellanox%20OFED%20driver%3C%2FA%3E%3CSPAN%20style%3D%22display%3A%20inline%20!important%3B%20float%3A%20none%3B%20background-color%3A%20%23ffffff%3B%20color%3A%20%23333333%3B%20cursor%3A%20text%3B%20font-family%3A%20inherit%3B%20font-size%3A%2016px%3B%20font-style%3A%20normal%3B%20font-variant%3A%20normal%3B%20font-weight%3A%20300%3B%20letter-spacing%3A%20normal%3B%20line-height%3A%201.7142%3B%20orphans%3A%202%3B%20text-align%3A%20left%3B%20text-decoration%3A%20none%3B%20text-indent%3A%200px%3B%20text-transform%3A%20none%3B%20-webkit-text-stroke-width%3A%200px%3B%20white-space%3A%20normal%3B%20word-spacing%3A%200px%3B%22%3E%20an%20be%20downloaded%20and%20installed%20as%20referenced%20below%3C%2FSPAN%3E%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3Ewget%20http%3A%2F%2Fcontent.mellanox.com%2Fofed%2FMLNX_OFED-5.0-1.0.0.0%2FMLNX_OFED_LINUX-5.0-1.0.0.0-ubuntu18.04-x86_64.tgz%0Atar%20zxvf%20MLNX_OFED_LINUX-5.0-1.0.0.0-ubuntu18.04-x86_64.tgz%0Asudo%20.%2FMLNX_OFED_LINUX-5.0-1.0.0.0-ubuntu18.04-x86_64%2Fmlnxofedinstall%20--add-kernel-support%3C%2FCODE%3E%3C%2FPRE%3E%3C%2FLI%3E%0A%3CLI%20style%3D%22box-sizing%3A%20border-box%3B%20font-family%3A%20%26amp%3Bquot%3B%20segoeui%26amp%3Bquot%3B%2C%26amp%3Bquot%3Blato%26amp%3Bquot%3B%2C%26amp%3Bquot%3Bhelvetica%20neue%26amp%3Bquot%3B%2Chelvetica%2Carial%2Csans-serif%3B%20margin-top%3A%200.25em%3B%22%3E%0A%3CP%20style%3D%22box-sizing%3A%20border-box%3B%20font-family%3A%20inherit%3B%20font-size%3A%2016px%3B%20font-weight%3A%20300%3B%20line-height%3A%201.7142%3B%20margin%3A%2016px%200px%2016px%200px%3B%22%3ELoad%20the%20new%20driver%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3Esudo%20%2Fetc%2Finit.d%2Fopenibd%20restart%3C%2FCODE%3E%3C%2FPRE%3E%3C%2FLI%3E%0A%3CLI%20style%3D%22box-sizing%3A%20border-box%3B%20font-family%3A%20%26amp%3Bquot%3B%22%3E%0A%3CP%20style%3D%22box-sizing%3A%20border-box%3B%20font-family%3A%20inherit%3B%20font-size%3A%2016px%3B%20font-weight%3A%20300%3B%20line-height%3A%201.7142%3B%20margin%3A%2016px%200px%2016px%200px%3B%22%3EIn%20%2Fetc%2Fwaagent.conf%2C%20enable%20RDMA%20by%20uncommenting%20the%20following%20configuration%20lines%20(root%20access)%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3EOS.EnableRDMA%3Dy%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3EIf%20needed%2C%20%3CFONT%20color%3D%22%23001000%22%3Ea%3CSPAN%20style%3D%22display%3A%20inline%20!important%3B%20float%3A%20none%3B%20background-color%3A%20%23ffffff%3B%20color%3A%20%23333333%3B%20cursor%3A%20text%3B%20font-family%3A%20inherit%3B%20font-size%3A%2016px%3B%20font-style%3A%20normal%3B%20font-variant%3A%20normal%3B%20font-weight%3A%20300%3B%20letter-spacing%3A%20normal%3B%20line-height%3A%201.7142%3B%20orphans%3A%202%3B%20text-align%3A%20left%3B%20text-decoration%3A%20none%3B%20text-indent%3A%200px%3B%20text-transform%3A%20none%3B%20-webkit-text-stroke-width%3A%200px%3B%20white-space%3A%20normal%3B%20word-spacing%3A%200px%3B%22%3E%3CSPAN%20style%3D%22display%3A%20inline%20!important%3B%20float%3A%20none%3B%20background-color%3A%20%23ffffff%3B%20color%3A%20%23333333%3B%20cursor%3A%20text%3B%20font-family%3A%20inherit%3B%20font-size%3A%2016px%3B%20font-style%3A%20normal%3B%20font-variant%3A%20normal%3B%20font-weight%3A%20300%3B%20letter-spacing%3A%20normal%3B%20line-height%3A%201.7142%3B%20orphans%3A%202%3B%20text-align%3A%20left%3B%20text-decoration%3A%20none%3B%20text-indent%3A%200px%3B%20text-transform%3A%20none%3B%20-webkit-text-stroke-width%3A%200px%3B%20white-space%3A%20normal%3B%20word-spacing%3A%200px%3B%22%3Essign%20the%20RDMA%20IP%20address%20manually%20to%20the%20ib0%20interface%3C%2FSPAN%3E%3C%2FSPAN%3E%3C%2FFONT%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3EIP%3D%24(sudo%20sed%20'%2FrdmaIPv4Address%3D%2F!d%3Bs%2F.*rdmaIPv4Address%3D%22%5C(%5B0-9.%5D*%5C)%22.*%2F%5C1%2F'%20%2Fvar%2Flib%2Fwaagent%2FSharedConfig.xml)%2F16%0Asudo%20ifconfig%20ib0%20%24IP%3C%2FCODE%3E%3C%2FPRE%3E%3C%2FLI%3E%0A%3C%2FOL%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20style%3D%22background-color%3A%20%23ffffff%3B%20box-sizing%3A%20border-box%3B%20color%3A%20%23333333%3B%20cursor%3A%20text%3B%20display%3A%20inline%3B%20float%3A%20none%3B%20font-family%3A%20inherit%3B%20font-size%3A%2016px%3B%20font-style%3A%20normal%3B%20font-variant%3A%20normal%3B%20font-weight%3A%20300%3B%20letter-spacing%3A%20normal%3B%20line-height%3A%201.7142%3B%20orphans%3A%202%3B%20text-align%3A%20left%3B%20text-decoration%3A%20none%3B%20text-indent%3A%200px%3B%20text-transform%3A%20none%3B%20-webkit-text-stroke-width%3A%200px%3B%20white-space%3A%20normal%3B%20word-spacing%3A%200px%3B%22%3EThe%20following%20is%20an%20optional%20step%2C%20applicable%20to%20all%20the%20three%20modes%20above%2C%20but%20not%20necessarily%20related%20to%20the%20above%20discussion%20of%20configuring%20IB.%26nbsp%3B%3C%2FSPAN%3EW%3CSPAN%20style%3D%22background-color%3A%20%23ffffff%3B%20box-sizing%3A%20border-box%3B%20color%3A%20%23333333%3B%20cursor%3A%20text%3B%20display%3A%20inline%3B%20float%3A%20none%3B%20font-family%3A%20inherit%3B%20font-size%3A%2016px%3B%20font-style%3A%20normal%3B%20font-variant%3A%20normal%3B%20font-weight%3A%20300%3B%20letter-spacing%3A%20normal%3B%20line-height%3A%201.7142%3B%20orphans%3A%202%3B%20text-align%3A%20left%3B%20text-decoration%3A%20none%3B%20text-indent%3A%200px%3B%20text-transform%3A%20none%3B%20-webkit-text-stroke-width%3A%200px%3B%20white-space%3A%20normal%3B%20word-spacing%3A%200px%3B%22%3Ehen%20running%20applications%20as%20non-root%20user%2C%20set%20the%20following%20memory%20limits%20in%20%2Fetc%2Fsecurity%2Flimits.conf.%3C%2FSPAN%3E%3C%2FP%3E%0A%3CDIV%20style%3D%22box-sizing%3A%20border-box%3B%20color%3A%20%23333333%3B%20font-family%3A%20%26amp%3Bquot%3B%20segoe%20ui%26amp%3Bquot%3B%2C%26amp%3Bquot%3Bhelvetica%20neue%26amp%3Bquot%3B%2C%26amp%3Bquot%3Bapple%20color%20emoji%26amp%3Bquot%3B%2C%26amp%3Bquot%3Bsegoe%20ui%20emoji%26amp%3Bquot%3B%2Chelvetica%2Carial%2Csans-serif%3B%20font-size%3A%2014px%3B%20font-style%3A%20normal%3B%20font-variant%3A%20normal%3B%20font-weight%3A%20400%3B%20letter-spacing%3A%20normal%3B%20orphans%3A%202%3B%20text-align%3A%20left%3B%20text-decoration%3A%20none%3B%20text-indent%3A%200px%3B%20text-transform%3A%20none%3B%20-webkit-text-stroke-width%3A%200px%3B%20white-space%3A%20normal%3B%20word-spacing%3A%200px%3B%22%3E%0A%3CDIV%20style%3D%22box-sizing%3A%20border-box%3B%20font-family%3A%20%26amp%3Bquot%3B%20segoe%20ui%26amp%3Bquot%3B%2C%26amp%3Bquot%3Bhelvetica%20neue%26amp%3Bquot%3B%2C%26amp%3Bquot%3Bapple%20color%20emoji%26amp%3Bquot%3B%2C%26amp%3Bquot%3Bsegoe%20ui%20emoji%26amp%3Bquot%3B%2Chelvetica%2Carial%2Csans-serif%3B%20font-size%3A%2014px%3B%22%3E%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3E%3CUSER%20or%3D%22%22%20group_name%3D%22%22%20or%3D%22%22%3E%20hard%20memlock%20%3CMEMORY_REQUIRED_BY_APPLICATION_IN_KB%20or%3D%22%22%20unlimited%3D%22%22%3E%0A%3CUSER%20or%3D%22%22%20group_name%3D%22%22%20or%3D%22%22%3E%20soft%20memlock%20%3CMEMORY_REQUIRED_BY_APPLICATION_IN_KB%20or%3D%22%22%20unlimited%3D%22%22%3E%3C%2FMEMORY_REQUIRED_BY_APPLICATION_IN_KB%3E%3C%2FUSER%3E%3C%2FMEMORY_REQUIRED_BY_APPLICATION_IN_KB%3E%3C%2FUSER%3E%3C%2FCODE%3E%3C%2FPRE%3E%3C%2FDIV%3E%0A%3C%2FDIV%3E%3C%2FLINGO-BODY%3E
Microsoft

There has been greater interest in the usage of the HPC optimized VM images that we publish due to the:

  1. GA of SR-IOV enabled HPC VMs (HB, HC, HB_v2),
  2. Recent platform update to make NCr_v3 SR-IOV enabled,
  3. GA of NDr_v2

While those images (CentOS-HPC 7.6, 7.7) are originally targeted for use on the SR-IOV enabled HPC VMs (HB, HC, HB_v2), conceptually, they are useful for the other now SR-IOV enabled GPU VMs (NCr_v3, NDr_v2) too. Note that the GPU VMs would additionally require the Nvidia GPU drivers (VM extension, manually).

 

Typically we find that users of the HPC VMs running traditional HPC applications tend to utilize CentOS as their preferred OS. While users of AI/ML applications running on the GPU VMs tend to prefer Ubuntu as the OS. The CentOS-HPC VM OS images (>=7.6 for the SR-IOV enabled VMs, and <=7.5 for the non-SR-IOV enabled VMs) provide a ready to use VM image with the appropriate drivers and MPI runtimes. Such a pre-packaged and ready to use experience isn't yet available for Ubuntu on Azure.

 

This article attempts to consolidate guidance on configuring InfiniBand (IB) for Ubuntu across both SR-IOV and non-SR-IOV enabled HPC and GPU VMs. Specifically it will focus on getting the right drivers setup and in bringing up the appropriate IB interface on the VMs. At the time of writing, the following steps at least apply to Ubuntu 18.04 LTS image by Canonical on the Azure Marketplace.

 

Non- SR-IOV enabled VMs

  1. Install dapl (and its dependencies rdma_cm, ibverbs), and user mode mlx4 library.

    sudo apt-get update
    sudo apt-get install libdapl2 libmlx4-1
  2. In /etc/waagent.conf, enable RDMA by uncommenting the following configuration lines (root access)

    OS.EnableRDMA=y
    OS.UpdateRdmaDriver=y
  3. Restart the waagent service
    sudo systemctl restart walinuxagent.service

     

The IB interface eth1 should come up with an RDMA IP address.

 

The IB related kernel modules are not auto-loaded on Ubuntu anymore. This is a departure from earlier practice where the kernel modules were built into the image. Now these are available as loadable modules so that a user can install Mellanox OFED driver.

 

SR-IOV enabled VMs with inbox driver

  1. Load following kernel modules (either mpdprob or edit /etc/modules)

    ib_uverbs
    rdma_ucm
    ib_umad
    ib_ipoib
  2. In /etc/waagent.conf, enable RDMA by uncommenting the following configuration lines (root access)

    OS.EnableRDMA=y
  3. Reboot VM

    sudo reboot

     

The IB interface ib0 should come up with an RDMA IP address.

 

SR-IOV enabled VMs with OFED driver

  1. The appropriate Mellanox OFED driver an be downloaded and installed as referenced below

    wget http://content.mellanox.com/ofed/MLNX_OFED-5.0-1.0.0.0/MLNX_OFED_LINUX-5.0-1.0.0.0-ubuntu18.04-x86_64.tgz
    tar zxvf MLNX_OFED_LINUX-5.0-1.0.0.0-ubuntu18.04-x86_64.tgz
    sudo ./MLNX_OFED_LINUX-5.0-1.0.0.0-ubuntu18.04-x86_64/mlnxofedinstall --add-kernel-support
  2. Load the new driver

    sudo /etc/init.d/openibd restart
  3. In /etc/waagent.conf, enable RDMA by uncommenting the following configuration lines (root access)

    OS.EnableRDMA=y

    If needed, assign the RDMA IP address manually to the ib0 interface 

    IP=$(sudo sed '/rdmaIPv4Address=/!d;s/.*rdmaIPv4Address="\([0-9.]*\)".*/\1/' /var/lib/waagent/SharedConfig.xml)/16
    sudo ifconfig ib0 $IP

 

The following is an optional step, applicable to all the three modes above, but not necessarily related to the above discussion of configuring IB. When running applications as non-root user, set the following memory limits in /etc/security/limits.conf.

<user or group_name or *> hard memlock <memory_required_by_application_in_KB or unlimited>
<user or group_name or *> soft memlock <memory_required_by_application_in_KB or unlimited>