Bring your own models on AI Toolkit - using Ollama and API keys

As we have seen in past blog posts, AI toolkit supports a range of models using Github Marketplace of models. However, you might require support for external models hosted by Google, Anthropic and Open AI which are either not available in the Github catalog of models or might want to use the models served by Ollama. We will cover both of these scenarios in this blog post.

OpenAI, Anthropic and Google Hosted models

Once you click on the Model catalog window and selected the models hosted by Google, Anthropic and OpenAI, you should see the following models selected.

OpenAI, Google and Anthropic Hosted Models

Also, you can add your APIs keys to the model in the following way

On the above models, you should click on the "Try in playground", just below the model name model card and you should be able to see the following dialog box on the top search bar of the VS Code window. Here I have clicked on the "Try it in playground" link for Anthropic Claude 3.5 Sonnet model

Add API key

Enter your API Key and you are good to go. Also, as you can see in the dropdown text, you can also edit or change the value later. Similarly, you can perform the same action for the Google and OpenAI hosted models. Once you have done this you are free to use these models in the playground and for using the other features of AI toolkit extension.

Using models served by Ollama

Several developers are also using Ollama to experiment and play with models using the command line. Ollama is an open-source AI tool that allows users to run large language models (LLMs) on their local systems. It's a valuable tool for industries that require data privacy, such as healthcare, finance, and government which might need locally hosted models. So, AI toolkit already supports some locally downloadable models such as those in the Phi-series by Microsoft or those by Mistral. Ollama supports a wider variety of models especially those from Meta's Llama series of LLMs and SLMs. The complete list of models currently supported by Ollama can be found at Ollama library.

We will run ollama on windows and when you run ollama and see help command you get the following output.

ollama help command

Once you have selected the model from the library, you can use the ollama pull or ollama run to download the model. The run command will download the model and then run it if it's not already downloaded. The pull command will just download the model from the repository. Since I want you show you a multimodal model that can be run locally, I will be go to the command line and download and run llama3.2-vision model. See the commands for the same below

ollama pull and run commands

We can also list the models already downloaded.

ollama list command

As you can see, I have downloaded and tried a bunch of models. It might take a bit of time to download the models based on the speed of your internet connection. As you can see some of the models are quite large downloads. Also, you might need to make sure you have enough RAM on your laptop or desktop to run the models.

So now we have downloaded the models, and we have run the models. Now, let's see how we can access them in AI Toolkit for VSCode. Go to the My models window as seen below. Click on the '+' symbol as seen below in the screenshot.

Once you click on this you will see a dropdown in search. Click on add an Ollama model

You will now see the choice to either select a model from ollama library or a custom ollama endpoint. For the purposes of this tutorial, we will select models. Let's select Multimodal modal we downloaded earlier. We should see the models we had seen in ollama list command earlier. See below.

Now we can select the checkbox alongside llama3.2-vision:latest and select okay. You should now see the model appear in the My Models window like below.

You can now right click on it and start using the model by loading in the playground. Since this is a multimodal model, you can use it to generate text as below. The screenshot below shows the model loaded via ollama and at the same time you can see that it is has the clip symbol activated in the window. Since this is a multi-modal model, we can give it an image and ask it questions. Which is what we will do next.

What is a pulsar?

Next, now let's attach an image and ask it questions about this image. This might take a bit more time as it will need to analyze the image and answer any questions. Since it is a generative AI model it can give slightly different inputs when given the same image.

I can also ask the models questions about the image, and it will use the information from the image - the objects shown, the relationship between the objects and its world knowledge from its training to answer the questions. For example, see the session below.

So, as you can the AI toolkit can be a fantastic place to try out different models from various sources. Ollama is also a great tool to try pre-built models locally securely without sending your data to the cloud, which can make it suitable for air-gapped environments and data privacy sensitive industries like fintech, healthcare and government. You have greater control over the models and the environment and the data they run on. It is also possible to customize the models and then serve them via ollama. This helps you choose the best model for your AI application.

Resources

• AI toolkit for VSCode - https://marketplace.visualstudio.com/items?itemName=ms-windows-ai-studio.windows-ai-studio
• AI toolkit for VSCode on Github - https://github.com/microsoft/vscode-ai-toolkit
• Ollama - https://github.com/ollama/ollama
• Ollama library - https://ollama.com/library
• Azure AI Discord - https://aka.ms/AzureAI/Discord