Learn how to deploy a TorchServe model as a managed online endpoint in Azure Machine Learning

With our recent announcement of support for custom containers in Azure Machine Learning comes support for a wide variety of machine learning frameworks and servers including TensorFlow Serving, R, and ML.NET. In this blog post, we'll show you how to deploy a PyTorch model using TorchServe.

The steps below reference our existing TorchServe sample here.

Export your model as a .mar file

To use TorchServe, you first need to export your model in the \"Model Archive Repository\" (.mar) format. Follow the PyTorch quickstart to learn how to do this for your PyTorch model.

Save your .mar file in a directory called \"torchserve.\"

Construct a Dockerfile

In the existing sample, we have a two-line Dockerfile:

FROM pytorch/torchserve:latest-cpu\n\nCMD [\"torchserve\",\"--start\",\"--model-store\",\"$MODEL_BASE_PATH/torchserve\",\"--models\",\"densenet161.mar\",\"--ts-config\",\"$MODEL_BASE_PATH/torchserve/config.properties\"]

Modify this Dockerfile to pass the name of your exported model from the previous step for the \"--models\" argument.

Build an image

Now, build a Docker image from the Dockerfile in the previous step, and store this image in the Azure Container Registry associated with your workspace:

WORKSPACE=$(az config get --query \"defaults[?name == 'workspace'].value\" -o tsv)\nACR_NAME=$(az ml workspace show -w $WORKSPACE --query container_registry -o tsv | cut -d'/' -f9-)\n\nif [[ $ACR_NAME == \"\" ]]\nthen\n    echo \"ACR login failed, exiting\"\n    exit 1\nfi\n\naz acr login -n $ACR_NAME\nIMAGE_TAG=${ACR_NAME}.azurecr.io/torchserve:8080\naz acr build $BASE_PATH/ -f $BASE_PATH/torchserve.dockerfile -t $IMAGE_TAG -r $ACR_NAME

Test locally

Ensure that you can serve your model by doing a local test. You will need to have Docker installed for this to work. Below, we show you how to run the image, download some sample data, and send a test liveness and scoring request.

# Run image locally for testing\ndocker run --rm -d -p 8080:8080 --name torchserve-test \\\n    -e MODEL_BASE_PATH=$MODEL_BASE_PATH \\\n    -v $PWD/$BASE_PATH/torchserve:$MODEL_BASE_PATH/torchserve $IMAGE_TAG\n\n# Check Torchserve health\necho \"Checking Torchserve health...\"\ncurl http://localhost:8080/ping\n\n# Download test image\necho \"Downloading test image...\"\nwget https://aka.ms/torchserve-test-image -O kitten_small.jpg\n\n# Check scoring locally\necho \"Uploading testing image, the scoring is...\"\ncurl http://localhost:8080/predictions/densenet161 -T kitten_small.jpg\n\ndocker stop torchserve-test\n

Create endpoint YAML

Create a YAML file that specifies the properties of the managed online endpoint you would like to create. In the example below, we specify the location of the model we will use as well as the Azure Virtual Machine size to use when deploying.

$schema: https://azuremlsdk2.blob.core.windows.net/latest/managedOnlineEndpoint.schema.json\nname: torchserve-endpoint\ntype: online\nauth_mode: aml_token\ntraffic:\n  torchserve: 100\n\ndeployments:\n  - name: torchserve\n    model:\n      name: torchserve-densenet161\n      version: 1\n      local_path: ./torchserve\n    environment_variables:\n      MODEL_BASE_PATH: /var/azureml-app/azureml-models/torchserve-densenet161/1\n    environment:\n      name: torchserve\n      version: 1\n      docker:\n        image: {{acr_name}}.azurecr.io/torchserve:8080\n      inference_config:\n        liveness_route:\n          port: 8080\n          path: /ping\n        readiness_route:\n          port: 8080\n          path: /ping\n        scoring_route:\n          port: 8080\n          path: /predictions/densenet161\n    instance_type: Standard_F2s_v2\n    scale_settings:\n      scale_type: manual\n      instance_count: 1\n      min_instances: 1\n      max_instances: 2

Create endpoint

Now that you have tested locally and you have a YAML file, you can create your endpoint:

az ml endpoint create -f $BASE_PATH/$ENDPOINT_NAME.yml -n $ENDPOINT_NAME

Send a scoring request

Once your endpoint finishes deploying, you can send it unlabeled data for scoring:

# Get accessToken\necho \"Getting access token...\"\nTOKEN=$(az ml endpoint get-credentials -n $ENDPOINT_NAME --query accessToken -o tsv)\n\n# Get scoring url\necho \"Getting scoring url...\"\nSCORING_URL=$(az ml endpoint show -n $ENDPOINT_NAME --query scoring_uri -o tsv)\necho \"Scoring url is $SCORING_URL\"\n\n# Check scoring\necho \"Uploading testing image, the scoring is...\"\ncurl -H \"Authorization: {Bearer $TOKEN}\" -T kitten_small.jpg $SCORING_URL

Delete resources

Now that you have successfully created and tested your TorchServe endpoint, you can delete it.

# Delete endpoint\necho \"Deleting endpoint...\"\naz ml endpoint delete -n $ENDPOINT_NAME --yes\n\n# Delete model\necho \"Deleting model...\"\naz ml model delete -n $AML_MODEL_NAME --version 1

Next steps

Read our documentation to learn more and see our other samples.

With our recent announcement of support for custom containers in Azure Machine Learning comes support for a wide variety of machine learning frameworks and servers including TensorFlow Serving, R, and ML.NET. In this blog post, we'll show you how to deploy a PyTorch model using TorchServe.

The steps below reference our existing TorchServe sample here.

Export your model as a .mar file

To use TorchServe, you first need to export your model in the \"Model Archive Repository\" (.mar) format. Follow the PyTorch quickstart to learn how to do this for your PyTorch model.

Save your .mar file in a directory called \"torchserve.\"

Construct a Dockerfile

In the existing sample, we have a two-line Dockerfile:

Modify this Dockerfile to pass the name of your exported model from the previous step for the \"--models\" argument.

Build an image

Now, build a Docker image from the Dockerfile in the previous step, and store this image in the Azure Container Registry associated with your workspace:

Test locally

Ensure that you can serve your model by doing a local test. You will need to have Docker installed for this to work. Below, we show you how to run the image, download some sample data, and send a test liveness and scoring request.

Create endpoint YAML

Create a YAML file that specifies the properties of the managed online endpoint you would like to create. In the example below, we specify the location of the model we will use as well as the Azure Virtual Machine size to use when deploying.

Create endpoint

Now that you have tested locally and you have a YAML file, you can create your endpoint:

Send a scoring request

Once your endpoint finishes deploying, you can send it unlabeled data for scoring:

Delete resources

Now that you have successfully created and tested your TorchServe endpoint, you can delete it.

Next steps

Read our documentation to learn more and see our other samples.

Learn how to deploy a TorchServe model as a managed online endpoint in Azure Machine Learning