Q: How do I use connected containerized Azure Cognitive Services?

[PaulHarrison]

Q: How do I use connected containerized Azure Cognitive Services?

A: Deployment is a little different, but you use them the same way as the Azure resource.

Last week we looked at how to deploy Azure Cognitive Services, specifically Translator. If you haven’t read the first part of this series, consider reviewing it first here: https://techcommunity.microsoft.com/t5/ai-cognitive-services-blog/q-how-do-i-use-azure-cognitive-ser...

Part two of this series will cover how to leverage the same Translator service but by deploying and querying a container you have running in your own environment instead of the Azure resource running in Azure. Deploying your own container to host cognitive services requires time, effort, and your own resources to run it on. You will also need to be sure to update the software, update dependencies, and if you’re hosting your own VM to run it you also need to deal with keeping it up to date, securing it, and other hassles that come along with that. If you host it in your own datacenter then you must deal with power, space, cooling, and physical security. Whenever possible use the Translator resource in Azure but if you can’t leverage Azure for the reasons documented in the first blog then this will show you how to leverage containerized Azure Cognitive Services. Not only is this a lot of extra work to do than using the Azure resources online, the containerized solution is a gated feature meaning that it is not open to everyone. Gated features are very similar to preview features with an approval process so be sure you actually need the containerized solution before proceeding. To get access to containerized Translator start by submitting a request here: https://aka.ms/csgate-translator

I’ll work though configuring Azure Cognitive Services using containers for a cruise ship that has limited bandwidth when at sea but plenty of computing power. It works just like the Azure Translator we know and love but can be deployed to containers. The great thing is that any code I have that leverages existing Azure Translator can be used without any modification other than just a couple of variable changes.

Disclaimer

The sample scripts are not supported under any Microsoft standard support program or service. The sample scripts are provided AS IS without warranty of any kind. Microsoft further disclaims all implied warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The entire risk arising out of the use or performance of the sample scripts and documentation remains with you. In no event shall Microsoft, its authors, or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the sample scripts or documentation, even if Microsoft has been advised of the possibility of such damages.

Here is some sample code I ran the other day when I was messaging with a mariachi band over the internet.

$baseURI = 'https://api.cognitive.microsofttranslator.com/'
$Key = '<put your key here>'  
$Region = 'eastus'
$TextToTranslate = 'I like bananas and accordions'
$to = 'es' 
$URI = $baseURI + 'translate?api-version=3.0&to=' + $to
$resp = Invoke-WebRequest -Uri $URI -Method Post -Headers $([hashtable]@{"Ocp-Apim-Subscription-Key"=$Key;"Ocp-Apim-Subscription-Region"=$Region}) -Body "[ {'Text':'$TextToTranslate'} ]"  -ContentType "application/json; charset=utf-8" -ErrorAction Inquire 
$trans = ($resp.Content | ConvertFrom-Json).translations.text 
$trans   

Unfortunately, I’m terrible at Spanish however I can throw it on an Azure Stack Edge in my backpack and run translation wherever I go even if there isn’t internet. In my case I’ll just be deploying it to my Ubuntu VM. Yes, Microsoft loves Linux including WSL, containers, and even has their own Linux distro (CBL-mariner). If you already have Kubernetes running and want to host these containers there then that is fine, I’m just showing one way to do it.

You will still need an Azure Translator resource in Azure which makes it easier to configure and manage. The billing for your Translation that is performed in your containers will be charged to your Azure subscription.

Since I have a Windows background and I figure other Windows folks reading this might not have a lot of Linux background I’ll include some basics. The Linux folks will find me a bit verbose, but that is okay, we don’t want to leave anyone behind.

1. Create a translation resource in Azure as documented in the first blog here:
   https://techcommunity.microsoft.com/t5/ai-cognitive-services-blog/q-how-do-i-use-azure-cognitive-ser...
2. Get your code working against your Azure resource
3. Get everything up to date.
   1. Sudo apt update
   2. Sudo apt upgrade
4. Install Docker as documented here: https://docs.docker.com/engine/install/ubuntu/#set-up-the-repository
   1. sudo apt install gnome-terminal
   2. sudo apt remove docker-desktop
   3. rm -r $HOME/.docker/desktop
   4. sudo rm /usr/local/bin/com.docker.cli
   5. sudo apt purge docker-desktop
   6. sudo apt-get update
   7. sudo apt-get install ca-certificates curl gnupg lsb-release
   8. sudo mkdir -p /etc/apt/keyrings
   9. curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
   10. echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
   11. sudo apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin
   12. sudo docker run hello-world
5. At this point the hello-world container should have run to indicate that you have docker working as expected. The output should look like this:

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

6. Download the translator container
   1. sudo docker pull mcr.microsoft.com/azure-cognitive-services/translator/text-translation
7. Allow use of /mnt/d/TranslatorContainer (put it wherever you want)
   1. Chmod 755 /mnt/d/TranslatorContainer
8. Try to run the container
   1. sudo docker run --rm -it -p 5000:5000 --memory 8g --cpus 4 -v /mnt/d/TranslatorContainer:/usr/local/models -e apikey=<I_don’t_share_my_key> -e eula=accept -e billing=https://paultestforblog.cognitiveservices.azure.com/ -e Languages=en,fr,es  mcr.microsoft.com/azure-cognitive-services/translator/text-translation:latest
   2. Note: If you run into the same situation as I did by trying to leverage your free tier Translator resource, then simply switch it to a different pricing tier and it should work. I tried to change mine from F0 to S1, the GUI confirmed that my tier had been changed, then I continued to get this same fatal error for a couple of minutes. Go get a drink or something, it only takes about 5 minutes, but a little longer than the Azure GUI indicates.

Updated LD_LIBRARY_PATH to /app

Starting model download

2022/12/09 18:21:08 [Info] Fetching Manifest from Model distribution service: https:// paultestforblog.cognitiveservices.azure.com/mods/onprem/v1.0/texttranslation/models/latest?languages=en,fr,es

2022/12/09 18:21:08 [Fatal] Model distribution service returns error, response status: 401, 401 Unauthorized, message: {"error":{"code":"401","message": "The Latest Model Operation under Model Distribution Service - OnPrem is not supported with the current subscription key and pricing tier TextTranslator.F0."}}

9. Validate it translates.
   1. This code looks very similar to what I used in part one, however there are some slight differences. I made sure to specify ‘from’ and changed the $baseURI to the public IP of my VM hosting the container.
   2. Of course, the container returns the same translation result:

      Me gustan los plátanos y los acordeones. 

10. When I try to run translation without specifying ‘from’ field it doesn’t work!
    1. This query worked against my Azure resource but didn’t work against my container.

       #Variable section
       $baseURI = 'http://MyVMIPAddress:5000/'
       $Key = '<I_don’t_share_my_key>'
       $Region = 'eastus2'
       $TextToTranslate = 'I like bananas and accordions.'
       
       #Test Translation: 
       $to = 'es'
       $from = 'en'
       $URI = $baseURI + 'translate?api-version=3.0&to=' + $to + '&from=' + $from
       $resp = Invoke-WebRequest -Uri $URI -Method Post -Headers $([hashtable]@{"Ocp-Apim-Subscription-Key"=$Key;"Ocp-Apim-Subscription-Region"=$Region}) -Body "[ {'Text':'$TextToTranslate'} ]"  -ContentType "application/json; charset=utf-8" -ErrorAction Inquire 
       $trans = ($resp.Content | ConvertFrom-Json).translations.text 
       $trans  

    2. It gives me the following error instead:

       Invoke-WebRequest : {"error":{"code":400035,"message":"The source language is not valid."}}
       At line:11 char:9
       + $resp = Invoke-WebRequest -Uri $URI -Method Post -Headers $([hashtabl ...
       +         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
           + CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest], WebException
           + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
       ConvertFrom-Json : Cannot bind argument to parameter 'InputObject' because it is null.
       At line:12 char:27
       + $trans = ($resp.Content | ConvertFrom-Json).translations.text
       +                           ~~~~~~~~~~~~~~~~
           + CategoryInfo          : InvalidData: (:) [ConvertFrom-Json], ParameterBindingValidationException
           + FullyQualifiedErrorId : ParameterArgumentValidationErrorNullNotAllowed,Microsoft.PowerShell.Commands.ConvertFromJsonCommand 

11. But Paul, you said that the code I wrote to work against my Azure resources would work against containers and I wouldn’t have to re-write it all! Now you’re saying that I must specify the ‘from’ field for it to work, what gives?

    Don’t worry, with the next little bit we’ll make it work without specifying ‘from’, but we want to make sure we have gotten this far correctly before moving on. The Translator container only translates so we need a different container to tell us which language input text is, the text-analytics container.

    Tune back in next week for how to configure the Translator and text-analytics containers to work together so that you don’t need to specify ‘from’.

Have fun scripting!

Resources:

Part 1:  How do I use Azure Cognitive Services like Translator? -

https://techcommunity.microsoft.com/t5/ai-cognitive-services-blog/q-how-do-i-use-azure-cognitive-ser...

Part 2: How do I use connected containerized Azure Cognitive Services? - https://techcommunity.microsoft.com/t5/ai-cognitive-services-blog/q-how-do-i-use-connected-container...

Part 3: How do I get my Translator containers to work the same way as Azure Translator? - <insert later when we have the publication URL>

Part 4: Offline Containers?