Very often, Solution Providers have enterprise scenarios for deploying ML pipelines, where involves multiple tenants and each tenant may have their own Azure subscription.
There are some situations when designing the enterprise solution, for example:
- Each tenant may want to keep their own data non-shared.
- Each tenant has their own computing and hosting environment for ML training, retraining, and inferencing.
- Each tenant may have different needs for retraining, with different scheduling, using different data sets.
- Even for the same ML algorithm, each tenant may use different parameters.
- The Solution Provider wants to maintain a centralized repository for all tenants’ data, models, environments, components, etc.
- In addition, the Solution Provider wants flexibility to manage all tenants, when and how they want the tenants to share data, models, environments, pipeline environments. Each tenant has their flexibility to share those with other tenants as well.
We provide this multi-tenancy solution for Solution Providers to deploy and manage the ML pipelines across multiple workspaces, where each workspace may belong to a different Azure subscription.
The key element here is Azure Machine Learning (AzureML) Registries. It acts as a middleman for tenants to share data/models/environments/components, with version control. When creating an AzureML registry, it is essential to make it available for multiple regions where the tenants reside in. The tenants outside of the Primary and Additional regions covered by the registry are not able to share data/models/environments/components with the registry. Besides that, the registry needs to add the workspaces as user; in workspaces, need to assign certain role (i.e. contributor) to the ML registry owner at subscription level and workspace level.
The solution works in this way: Each tenant has their own workspace using their own subscription; within the workspace, the tenant is self-sufficient for computing and storage resources, it can build ML pipeline using its own data/models/environments/components. If the tenant wants to share data/models/environments/components with another tenant, it shares to the registry first, which we call ‘share’ in the picture below; then through the registry to share with other tenants, which we call ‘push’ in the picture below. Tenants can also get data/models/environments/components from registry, which we call ‘pull’ as shown in the picture below.
- Share – from tenant (workspace) sends to registry.
- Push – from registry sends to tenant (workspace).
- Pull – tenant (workspace) gets from registry.
This kind of solution design can satisfy multiple scenarios of model sharing:
- Each tenant can have their specific models without sharing with others.
- All tenants can share their models if they want.
- All tenants can pull the shared models and retrain or fine-tune, then share back the retrained or fine-tuned models.
Similar scenarios apply to data, environments, and components.
Notes:
- If a tenant has multiple subscriptions, then the sharing is done at subscription level then workspace level.
- This solution doesn’t apply if the sharing has to be done above subscription level. That means, if the ML registry can’t access the tenant’s subscription directly, then the ML registry can’t share/push entities to that tenant.
Below is an example for model sharing in four workspaces, where three of them belong to one subscription and the other one belongs to a different subscription.
In this example, workspace1 shares its model credit-card-default to registry, then the registry pushes to workspace3; workspace3 shares its model bert-case-uncased_fine-tuned to registry, then the registry pushes to workspace1, workspace2 and workspace4. The registry has both models.
For data, models, environments, components, the way of share/push/pull between tenant and registry are a little different. For ‘share/push’, it can be done in AzureML UI and SDK; for ‘pull’, it can be done in SDK.
Below is one example for ‘share’ model from workspace1 to registry using AzureML UI.
Below are some examples of SDK to get (‘pull’) data/model/environment/component from registry:
Then further register into the workspace, below is an example for environment:
The cool thing about this solution is that, once a tenant gets data/model/environment/component from registry, where originally shared by another tenant, it can build its own ML pipeline, run the pipeline using its own resources. Besides the way described above using SDK, the tenant can retrieve those from AzureML Designer. By filtering the registry creator, the workspace can see the data, model and component, and then create pipeline on the canvas.
Below is one example in workspace4, it gets model from registry where originally from workspace3, it uses its own data set, AzureML pre-built component, and creates a fine-tuning pipeline. After it runs the pipeline, fine-tunes the model, it can share a new version of fine-tuned model back to registry. Remember, workspace4 belongs to a different subscription from workspace3, it’s really cool!
The way registry ‘share/push’ model to a workspace is that it deploys a real-time or batch endpoint to the workspace. The workspace then can do inference using the endpoint. Below is an example, workspace4 gets the model from registry where originated from workspace3, then performs a test using input data.
If the workspace deployed an endpoint from the shared model, also ‘get/pull’ the model from the registry, it can add deployment to the endpoint, as shown below.
References:
- Create and manage registries - Azure Machine Learning | Microsoft Learn
- Share data across workspaces with registries (preview) - Azure Machine Learning | Microsoft Learn
- Share models, components, and environments across workspaces with registries - Azure Machine Learning | Microsoft Learn
- azure-docs/articles/machine-learning/how-to-share-models-pipelines-across-workspaces-with-registries.md at main · MicrosoftDocs/azure-docs (github.com)
Acknowledgement:
Thanks Daniel Scott-Raynsford and Facundo Santiago for encouraging me to write this article. We are glad to share this solution implementation broadly to help our customers.
Reviewers:
Daniel Scott-Raynsford, Takuto Higuchi, Alex Zeltov
Microsoft
