AI on Azure: managed ML

AI on Azure series, Part 4 of 5

Training a model and serving it are separate jobs. A platform can manage one, both, or only part of each. The right choice depends on what you need to customize and how much of the model lifecycle your team wants to operate.

This article compares Azure Machine Learning, fine-tuning and managed compute in Microsoft Foundry, Azure Databricks, Microsoft Fabric Data Science, and Anyscale on Azure without treating them as interchangeable. See Part 1: AI on Azure 101 for the distinction between training, fine-tuning, and inference, and Microsoft's architecture guide for a broader product comparison.

Product details were checked against Microsoft Learn on October 29, 2026. Model eligibility, regions, and preview status can change. Confirm the current support matrix before committing to a training or serving path.

Figure 1. The data platform, training workflow, and serving runtime can be different services.

1. Azure Machine Learning for a model lifecycle you own

Azure Machine Learning is the broad choice when your team needs to organize custom training code, compute, data references, environments, metrics, and model versions. It supports common open-source frameworks such as PyTorch, TensorFlow, and scikit-learn, along with distributed training and MLOps workflows.

When a model is ready to serve, Azure ML managed endpoints abstract infrastructure for both online inference and batch scoring. Online endpoints return responses over HTTPS; batch endpoints process data asynchronously on compute. You still own the model's behavior, test data, runtime configuration, deployment policy, and operational acceptance criteria.

Choose Azure ML when you need repeatable training jobs, pipelines, versioned artifacts, or an endpoint for a model developed with your own code. It may be more platform than you need if your only goal is to call a hosted foundation model.

2. Fine-tune a supported model in Foundry

Fine-tuning adapts a pretrained model using task-specific examples. It changes model weights, unlike retrieval, which supplies external information at inference time. Fine-tuning can help with a repeatable style, format, or behavior, but the result is not guaranteed to improve. Compare it with a baseline on held-out examples.

Microsoft Foundry's fine-tuning runs supported training jobs on Foundry-managed infrastructure. The available method, training type, model, and serving option depend on the model. The current overview documents methods including supervised fine-tuning, preference optimization, and reinforcement fine-tuning for supported models. Interactive training is Preview and intended for teams that need control of lower-level training operations.

Fine-tuning is not a replacement for retrieval when the answer depends on current documents. It also does not remove the need for application-level safety, evaluation, or data governance. Start by checking the supported-model table for the exact training and deployment combination you need.

3. Foundry Managed Compute is a separate preview path

Foundry Managed Compute serves selected open-source and community models from the Foundry catalog on dedicated GPU capacity that Foundry manages. The current deployment path uses the Hugging Face collection; check the supported model and deployment template rather than assuming partner or custom models are eligible. It is different from the Serverless API in Part 2: the Serverless API bills by tokens or provisioned throughput, while Managed Compute bills hourly by accelerator capacity.

As of this documentation check, Foundry Managed Compute is Preview, has no service-level agreement, and is not recommended for production workloads. The current deployment overview describes global processing and a separate Foundry quota process; existing Azure VM quota does not apply. Include the preview terms and operational risks in the production decision.

Do not confuse it with Azure Machine Learning compute. Azure ML is the full ML lifecycle platform; Foundry Managed Compute is a managed inference path for supported model deployments.

4. Keep ML close to the data platform

Azure Databricks combines data engineering and machine learning workflows. Its ML documentation covers feature engineering, MLflow experiment tracking, model training, model serving, batch inference, and governance through Unity Catalog. Evaluate it when the data and ML teams already work in the Databricks environment.

Microsoft Fabric Data Science works directly with governed data in OneLake. Teams can use notebooks and Spark for data preparation and training, track experiments with MLflow, score models, and make predictions available to Fabric analytics and Power BI. It fits workflows where data science, data engineering, and BI share a Fabric data foundation.

These platforms overlap in what they can do, but the surrounding data, governance, and production environment usually decides which is a better fit. Compare the endpoint, deployment, monitoring, and security requirements in addition to the notebook experience.

5. Managed Ray: Anyscale on Azure

Ray is an open-source framework for distributed Python and AI workloads. It can support data processing, distributed training, and inference; it is not itself a hosted model API. Anyscale on Azure adds a managed platform for Ray through an Azure Native Integration. Evaluate it when your application already uses Ray, or when you want a provider to manage Ray cluster lifecycles rather than building that platform yourself.

The Anyscale architecture separates an Anyscale-operated control plane, hosted in Azure, from a data plane on AKS in your Azure subscription. Anyscale manages its operator and shares Ray cluster operations with your team. You still own your application and the Azure resources in your subscription, including their configuration and access controls. "Managed Ray" does not mean that Anyscale takes over every AKS, storage, or networking responsibility.

Check the current feature limitations and supported regions rather than assuming that every feature of the broader Anyscale platform is available in this integration. The FAQ separates Anyscale charges from the Azure resources you also pay for; the support model uses Azure support with Anyscale engaged when needed. This is a different operating model from installing open-source KubeRay yourself or using the Preview managed KubeRay extension. Part 5 compares those paths.

6. Match the service to the job

Before training, confirm that you have enough representative data and a measurable reason to change the model. Before serving, confirm the supported model and runtime, region, quota, data-processing boundary, endpoint behavior, and owner for ongoing operations.

AI on Azure series

1. Part 1: AI on Azure 101
2. Part 2: Models as a service
3. Part 3: Platforms for apps and agents
4. Part 4: Managed ML (this article)
5. Part 5: When you own the infrastructure

References

• Azure Machine Learning
• Azure ML managed endpoints
• Microsoft Foundry's fine-tuning
• Foundry Managed Compute overview
• Azure Databricks
• Microsoft Fabric Data Science
• Anyscale on Azure solution
• Anyscale on Azure overview and limitations
• Anyscale on Azure architecture and responsibility matrix
• Anyscale on Azure FAQ
• Anyscale on Azure support model
• Microsoft's architecture guide