Finding Azure Batch Python client in Conda packaging
I've started working with Azure Batch and use Python, with my Python environment managed by Anaconda. I'd like to install the https://learn.microsoft.com/en-us/python/api/overview/azure/batch-readme?view=azure-python and https://learn.microsoft.com/en-us/python/api/overview/azure/mgmt-batch-readme?view=azure-python from the https://learn.microsoft.com/en-us/azure/developer/python/sdk/azure-sdk-overview in my Anaconda environments, preferably using conda instead of pip. These are the azure.batch and azure.mgmt.batch modules (or "module packages"; whatever they're called), found in the azure-batch and azure-mgmt-batch PyPI packages. But I don't know where to find them in the Conda packaging of Azure SDK for Python. The Azure SDK for Python introduced https://devblogs.microsoft.com/azure-sdk/python-conda-sdk-preview/ back in 2021, and its use is described in the https://learn.microsoft.com/en-us/azure/developer/python/sdk/azure-sdk-install?source=recommendations&tabs=conda. The Conda packaging differs from the PyPI packaging. The Python Azure SDK modules are packaged in to a smaller number of packages in the Conda form, with sometimes different naming. Is the Azure Batch client library available in the Microsoft-supplied Conda packages somewhere (the ones in the microsoft conda channel, instead of the conda-forge channel)? If so, which Conda package? And more generally, if I know what Azure SDK for Python module I want, or what PyPI package it's in, how can I find out which microsoft-channel Conda package it's in? I haven't been able to find a list of which module is in which Conda package anywhere. There's an https://anaconda.org/conda-forge/azure-batch (instead of the microsoft channel). But if I understand correctly, those conda-forge Azure packages are the old ones from before the 2021 introduction of the microsoft conda channel's packaging, and have different dependencies and stuff. I'd prefer to install the Azure Batch client from the microsoft-channel Conda packages, instead of the conda-forge channel package or from PyPI/pip, for consistency with my other Azure Python packages, which are all installed from the microsoft-channel Conda packages. I've read that mixing interdependent packages from different channels can sometimes cause problems, and if you're mixing conda-managed and pip-managed packages in an Anaconda environment, you're supposed to install all the conda packages first, then the pip packages, and then don't go back and install or update any conda packages afterwards, or something like that.Azure Batch AI example using tensorflow
Hi currently I am assessing the Azure frameworks. I found a nice example how to run CNTK on parallel CPUs/GPUs using the CLI interface https://docs.microsoft.com/en-gb/azure/batch-ai/quickstart-cli Now I wonder if it is similarly easy to run a convolutional neural network for the MNIST database using tensorflow? Does anyone have experience with this set up? Regards MarkusHPC containers with Azure Batch
With the https://azure.microsoft.com/en-us/updates/azure-batch-updates-october2017/ to https://azure.microsoft.com/en-us/services/batch/, you now have the option to schedule your tasks as Docker container invocations. Containers and Azure Batch are an ideal way to package, execute, and scale your High Performance Computing (HPC) applications and batch workloads in a consistent, reproducible manner utilizing powerful cloud native job scheduling capabilities. Today, we're excited to announce support for http://singularity.lbl.gov/ in the latest https://github.com/Azure/batch-shipyard/releases. Singularity is a container solution amenable to both administrators and users of shared HPC and cluster computing environments, while still providing access to accelerators such as GPUs and specialized interconnects in container contexts. https://github.com/Azure/batch-shipyard is an open system for enabling simple, configuration-based container execution on Azure Batch, and aims to allow users of these shared computing environments to easily execute their existing Singularity workloads on Azure. Azure's https://docs.microsoft.com/en-us/azure/virtual-machines/linux/sizes-gpu, includinghttps://azure.microsoft.com/en-us/blog/new-gpus-coming-to-azure-accelerate-hpc-and-ai-workloads/ND, NCv2, and the upcoming NCv3 series of VMs, and https://docs.microsoft.com/en-us/azure/virtual-machines/linux/sizes-hpc-enabled instances are potentially an ideal fit for such workloads. Also, with Batch Shipyard and Azure Batch, not only can you http://batch-shipyard.readthedocs.io/en/latest/30-batch-shipyard-autoscale/ your compute pools with ease, you can also opt to execute your workloads on https://docs.microsoft.com/en-us/azure/batch/batch-low-pri-vms for savings up to 80%! Read about it in the https://azure.microsoft.com/en-us/blog/hpc-containers-with-azure-batch/.
