Educator Developer Blog

8 MIN READ

GitHub Model Catalog - Getting Started

Microsoft

Aug 08, 2024

GitHub Models - Limited Public Beta SIGNUP TODAY

Welcome to GitHub Models! We've got everything fired up and ready for you to explore AI Models hosted on Azure AI. So as Student developer you already have access to amazing GitHub Resources like Codespaces and Copilot from http://education.github.com now you get started on developing with Generative AI and Language Models with the Model Catalog.

Access and onboarding

GitHub Models provide free access to a set of AI models for anyone with a GitHub account.
This makes it significantly easier to get familiar with AI models without having to create Azure Resources or download models from Hugging Face.

The GitHub Model; is your opportunity to test out these models for free.

Key features of GitHub Models
Seamless integration with Codespaces allows for quick learning and engagement.

The ease of local use, for free, in code they may have already written.
The ability to switch between model providers using the same API call via the Azure AI inference API, eliminating the need to change code between providers.

For more information about the Models available on GitHub Models, check out the GitHub Model Marketplace

Models Available

Each model has a dedicated playground and sample code available in a dedicated codespaces environment and utilizes the Azure Inference API so swapping models is simply changing the model name.

Getting Started

There are a few basic examples that are ready for you to run. You can find them in the samples directory within the codespaces environment.

If you want to jump straight to your favorite language, you can find the examples in the following Languages:

Python
JavaScript
cURL

The dedicated Codespaces Environment is an excellent way to get started running the samples and models.

Sample Code

Below are example code snippets for a few use cases. For additional information about Azure AI Inference SDK, see full documentation and samples.

Setup

Create a personal access token You do not need to give any permissions to the token. Note that the token will be sent to a Microsoft service.

To use the code snippets below, create an environment variable to set your token as the key for the client code.

If you're using bash:

export GITHUB_TOKEN="<your-github-token-goes-here>"

If you're in powershell:

$Env:GITHUB_TOKEN="<your-github-token-goes-here>"

If you're using Windows command prompt:

set GITHUB_TOKEN=<your-github-token-goes-here>

Python Sample

Install dependencies

Install the Azure AI Inference SDK using pip (Requires: Python >=3.8):

pip install azure-ai-inference

Run a basic code sample

This sample demonstrates a basic call to the chat completion API. It is leveraging the GitHub AI model inference endpoint and your GitHub token. The call is synchronous.

import os
from azure.ai.inference import ChatCompletionsClient
from azure.ai.inference.models import SystemMessage, UserMessage
from azure.core.credentials import AzureKeyCredential

endpoint = "https://models.inference.ai.azure.com"
# Replace Model_Name 
model_name = "Phi-3-small-8k-instruct"
token = os.environ["GITHUB_TOKEN"]

client = ChatCompletionsClient(
    endpoint=endpoint,
    credential=AzureKeyCredential(token),
)

response = client.complete(
    messages=[
        SystemMessage(content="You are a helpful assistant."),
        UserMessage(content="What is the capital of France?"),
    ],
    model=model_name,
    temperature=1.,
    max_tokens=1000,
    top_p=1.
)

print(response.choices[0].message.content)

Run a multi-turn conversation

This sample demonstrates a multi-turn conversation with the chat completion API. When using the model for a chat application, you'll need to manage the history of that conversation and send the latest messages to the model.

import os
from azure.ai.inference import ChatCompletionsClient
from azure.ai.inference.models import AssistantMessage, SystemMessage, UserMessage
from azure.core.credentials import AzureKeyCredential

token = os.environ["GITHUB_TOKEN"]
endpoint = "https://models.inference.ai.azure.com"
# Replace Model_Name
model_name = "Phi-3-small-8k-instruct"

client = ChatCompletionsClient(
    endpoint=endpoint,
    credential=AzureKeyCredential(token),
)

messages = [
    SystemMessage(content="You are a helpful assistant."),
    UserMessage(content="What is the capital of France?"),
    AssistantMessage(content="The capital of France is Paris."),
    UserMessage(content="What about Spain?"),
]

response = client.complete(messages=messages, model=model_name)

print(response.choices[0].message.content)

Stream the output

For a better user experience, you will want to stream the response of the model so that the first token shows up early and you avoid waiting for long responses.

import os
from azure.ai.inference import ChatCompletionsClient
from azure.ai.inference.models import SystemMessage, UserMessage
from azure.core.credentials import AzureKeyCredential

token = os.environ["GITHUB_TOKEN"]
endpoint = "https://models.inference.ai.azure.com"
# Replace Model_Name
model_name = "Phi-3-small-8k-instruct"

client = ChatCompletionsClient(
    endpoint=endpoint,
    credential=AzureKeyCredential(token),
)

response = client.complete(
    stream=True,
    messages=[
        SystemMessage(content="You are a helpful assistant."),
        UserMessage(content="Give me 5 good reasons why I should exercise every day."),
    ],
    model=model_name,
)

for update in response:
    if update.choices:
        print(update.choices[0].delta.content or "", end="")

client.close()

JavaScript

Install dependencies

Install Node.js.

Copy the following lines of text and save them as a file package.json inside your folder.

{
  "type": "module",
  "dependencies": {
    "@azure-rest/ai-inference": "latest",
    "@azure/core-auth": "latest",
    "@azure/core-sse": "latest"
  }
}

Note: azure/core-sse is only needed when you stream the chat completions response.

Open a terminal window in this folder and run npm install.

For each of the code snippets below, copy the content into a file sample.js and run with node sample.js.

Run a basic code sample

This sample demonstrates a basic call to the chat completion API. It is leveraging the GitHub AI model inference endpoint and your GitHub token. The call is synchronous.

import ModelClient from "@azure-rest/ai-inference";
import { AzureKeyCredential } from "@azure/core-auth";

const token = process.env["GITHUB_TOKEN"];
const endpoint = "https://models.inference.ai.azure.com";
// Update your modelname
const modelName = "Phi-3-small-8k-instruct";

export async function main() {

  const client = new ModelClient(endpoint, new AzureKeyCredential(token));

  const response = await client.path("/chat/completions").post({
    body: {
      messages: [
        { role:"system", content: "You are a helpful assistant." },
        { role:"user", content: "What is the capital of France?" }
      ],
      model: modelName,
      temperature: 1.,
      max_tokens: 1000,
      top_p: 1.
    }
  });

  if (response.status !== "200") {
    throw response.body.error;
  }
  console.log(response.body.choices[0].message.content);
}

main().catch((err) => {
  console.error("The sample encountered an error:", err);
});

Run a multi-turn conversation

import ModelClient from "@azure-rest/ai-inference";
import { AzureKeyCredential } from "@azure/core-auth";

const token = process.env["GITHUB_TOKEN"];
const endpoint = "https://models.inference.ai.azure.com";
// Update your modelname
const modelName = "Phi-3-small-8k-instruct";

export async function main() {

  const client = new ModelClient(endpoint, new AzureKeyCredential(token));

  const response = await client.path("/chat/completions").post({
    body: {
      messages: [
        { role: "system", content: "You are a helpful assistant." },
        { role: "user", content: "What is the capital of France?" },
        { role: "assistant", content: "The capital of France is Paris." },
        { role: "user", content: "What about Spain?" },
      ],
      model: modelName,
    }
  });

  if (response.status !== "200") {
    throw response.body.error;
  }

  for (const choice of response.body.choices) {
    console.log(choice.message.content);
  }
}

main().catch((err) => {
  console.error("The sample encountered an error:", err);
});

Stream the output

For a better user experience, you will want to stream the response of the model so that the first token shows up early and you avoid waiting for long responses.

import ModelClient from "@azure-rest/ai-inference";
import { AzureKeyCredential } from "@azure/core-auth";
import { createSseStream } from "@azure/core-sse";

const token = process.env["GITHUB_TOKEN"];
const endpoint = "https://models.inference.ai.azure.com";
// Update your modelname
const modelName = "Phi-3-small-8k-instruct";

export async function main() {

  const client = new ModelClient(endpoint, new AzureKeyCredential(token));

  const response = await client.path("/chat/completions").post({
    body: {
      messages: [
        { role: "system", content: "You are a helpful assistant." },
        { role: "user", content: "Give me 5 good reasons why I should exercise every day." },
      ],
      model: modelName,
      stream: true
    }
  }).asNodeStream();

  const stream = response.body;
  if (!stream) {
    throw new Error("The response stream is undefined");
  }

  if (response.status !== "200") {
    stream.destroy();
    throw new Error(`Failed to get chat completions, http operation failed with ${response.status} code`);
  }

  const sseStream = createSseStream(stream);

  for await (const event of sseStream) {
    if (event.data === "[DONE]") {
      return;
    }
    for (const choice of (JSON.parse(event.data)).choices) {
        process.stdout.write(choice.delta?.content ?? ``);
    }
  }
}

main().catch((err) => {
  console.error("The sample encountered an error:", err);
});

REST

Run a basic code sample

Paste the following into a shell:

curl -X POST "https://models.inference.ai.azure.com/chat/completions" \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $GITHUB_TOKEN" \
    -d '{
        "messages": [
            {
                "role": "system",
                "content": "You are a helpful assistant."
            },
            {
                "role": "user",
                "content": "What is the capital of France?"
            }
        ],
        "model": "Phi-3-small-8k-instruct"
    }'

Run a multi-turn conversation

Call the chat completion API and pass the chat history:

curl -X POST "https://models.inference.ai.azure.com/chat/completions" \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $GITHUB_TOKEN" \
    -d '{
        "messages": [
            {
                "role": "system",
                "content": "You are a helpful assistant."
            },
            {
                "role": "user",
                "content": "What is the capital of France?"
            },
            {
                "role": "assistant",
                "content": "The capital of France is Paris."
            },
            {
                "role": "user",
                "content": "What about Spain?"
            }
        ],
        "model": "Phi-3-small-8k-instruct"
    }'

Stream the output

This is an example of calling the endpoint and streaming the response.

curl -X POST "https://models.inference.ai.azure.com/chat/completions" \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $GITHUB_TOKEN" \
    -d '{
        "messages": [
            {
                "role": "system",
                "content": "You are a helpful assistant."
            },
            {
                "role": "user",
                "content": "Give me 5 good reasons why I should exercise every day."
            }
        ],
        "stream": true,
        "model": "Phi-3-small-8k-instruct"
    }'

FREE Usage and Rate limits for GitHub Models

The rate limits for the playground and free API usage are intended to help you experiment with models and prototype your AI application. For use beyond those limits, and to bring your application to scale, you must provision resources from an Azure account, and authenticate from there instead of your GitHub personal access token. You don't need to change anything else in your code. Use this link to discover how to go beyond the free tier limits in Azure AI.

Updated Aug 16, 2024

Version 6.0

Microsoft

Joined September 25, 2018

View Profile

Educator Developer Blog

Follow this blog board to get notified when there's new activity

import os\nfrom azure.ai.inference import ChatCompletionsClient\nfrom azure.ai.inference.models import SystemMessage, UserMessage\nfrom azure.core.credentials import AzureKeyCredential\n\nendpoint = \"https://models.inference.ai.azure.com\"\n# Replace Model_Name \nmodel_name = \"Phi-3-small-8k-instruct\"\ntoken = os.environ[\"GITHUB_TOKEN\"]\n\nclient = ChatCompletionsClient(\n endpoint=endpoint,\n credential=AzureKeyCredential(token),\n)\n\nresponse = client.complete(\n messages=[\n SystemMessage(content=\"You are a helpful assistant.\"),\n UserMessage(content=\"What is the capital of France?\"),\n ],\n model=model_name,\n temperature=1.,\n max_tokens=1000,\n top_p=1.\n)\n\nprint(response.choices[0].message.content)\n

import os\nfrom azure.ai.inference import ChatCompletionsClient\nfrom azure.ai.inference.models import AssistantMessage, SystemMessage, UserMessage\nfrom azure.core.credentials import AzureKeyCredential\n\ntoken = os.environ[\"GITHUB_TOKEN\"]\nendpoint = \"https://models.inference.ai.azure.com\"\n# Replace Model_Name\nmodel_name = \"Phi-3-small-8k-instruct\"\n\nclient = ChatCompletionsClient(\n endpoint=endpoint,\n credential=AzureKeyCredential(token),\n)\n\nmessages = [\n SystemMessage(content=\"You are a helpful assistant.\"),\n UserMessage(content=\"What is the capital of France?\"),\n AssistantMessage(content=\"The capital of France is Paris.\"),\n UserMessage(content=\"What about Spain?\"),\n]\n\nresponse = client.complete(messages=messages, model=model_name)\n\nprint(response.choices[0].message.content)\n

import os\nfrom azure.ai.inference import ChatCompletionsClient\nfrom azure.ai.inference.models import SystemMessage, UserMessage\nfrom azure.core.credentials import AzureKeyCredential\n\ntoken = os.environ[\"GITHUB_TOKEN\"]\nendpoint = \"https://models.inference.ai.azure.com\"\n# Replace Model_Name\nmodel_name = \"Phi-3-small-8k-instruct\"\n\nclient = ChatCompletionsClient(\n endpoint=endpoint,\n credential=AzureKeyCredential(token),\n)\n\nresponse = client.complete(\n stream=True,\n messages=[\n SystemMessage(content=\"You are a helpful assistant.\"),\n UserMessage(content=\"Give me 5 good reasons why I should exercise every day.\"),\n ],\n model=model_name,\n)\n\nfor update in response:\n if update.choices:\n print(update.choices[0].delta.content or \"\", end=\"\")\n\nclient.close()\n

import ModelClient from \"@azure-rest/ai-inference\";\nimport { AzureKeyCredential } from \"@azure/core-auth\";\n\nconst token = process.env[\"GITHUB_TOKEN\"];\nconst endpoint = \"https://models.inference.ai.azure.com\";\n// Update your modelname\nconst modelName = \"Phi-3-small-8k-instruct\";\n\nexport async function main() {\n\n const client = new ModelClient(endpoint, new AzureKeyCredential(token));\n\n const response = await client.path(\"/chat/completions\").post({\n body: {\n messages: [\n { role:\"system\", content: \"You are a helpful assistant.\" },\n { role:\"user\", content: \"What is the capital of France?\" }\n ],\n model: modelName,\n temperature: 1.,\n max_tokens: 1000,\n top_p: 1.\n }\n });\n\n if (response.status !== \"200\") {\n throw response.body.error;\n }\n console.log(response.body.choices[0].message.content);\n}\n\nmain().catch((err) => {\n console.error(\"The sample encountered an error:\", err);\n});\n

import ModelClient from \"@azure-rest/ai-inference\";\nimport { AzureKeyCredential } from \"@azure/core-auth\";\n\nconst token = process.env[\"GITHUB_TOKEN\"];\nconst endpoint = \"https://models.inference.ai.azure.com\";\n// Update your modelname\nconst modelName = \"Phi-3-small-8k-instruct\";\n\nexport async function main() {\n\n const client = new ModelClient(endpoint, new AzureKeyCredential(token));\n\n const response = await client.path(\"/chat/completions\").post({\n body: {\n messages: [\n { role: \"system\", content: \"You are a helpful assistant.\" },\n { role: \"user\", content: \"What is the capital of France?\" },\n { role: \"assistant\", content: \"The capital of France is Paris.\" },\n { role: \"user\", content: \"What about Spain?\" },\n ],\n model: modelName,\n }\n });\n\n if (response.status !== \"200\") {\n throw response.body.error;\n }\n\n for (const choice of response.body.choices) {\n console.log(choice.message.content);\n }\n}\n\nmain().catch((err) => {\n console.error(\"The sample encountered an error:\", err);\n});\n

import ModelClient from \"@azure-rest/ai-inference\";\nimport { AzureKeyCredential } from \"@azure/core-auth\";\nimport { createSseStream } from \"@azure/core-sse\";\n\nconst token = process.env[\"GITHUB_TOKEN\"];\nconst endpoint = \"https://models.inference.ai.azure.com\";\n// Update your modelname\nconst modelName = \"Phi-3-small-8k-instruct\";\n\nexport async function main() {\n\n const client = new ModelClient(endpoint, new AzureKeyCredential(token));\n\n const response = await client.path(\"/chat/completions\").post({\n body: {\n messages: [\n { role: \"system\", content: \"You are a helpful assistant.\" },\n { role: \"user\", content: \"Give me 5 good reasons why I should exercise every day.\" },\n ],\n model: modelName,\n stream: true\n }\n }).asNodeStream();\n\n const stream = response.body;\n if (!stream) {\n throw new Error(\"The response stream is undefined\");\n }\n\n if (response.status !== \"200\") {\n stream.destroy();\n throw new Error(`Failed to get chat completions, http operation failed with ${response.status} code`);\n }\n\n const sseStream = createSseStream(stream);\n\n for await (const event of sseStream) {\n if (event.data === \"[DONE]\") {\n return;\n }\n for (const choice of (JSON.parse(event.data)).choices) {\n process.stdout.write(choice.delta?.content ?? ``);\n }\n }\n}\n\nmain().catch((err) => {\n console.error(\"The sample encountered an error:\", err);\n});\n

curl -X POST \"https://models.inference.ai.azure.com/chat/completions\" \\\n -H \"Content-Type: application/json\" \\\n -H \"Authorization: Bearer $GITHUB_TOKEN\" \\\n -d '{\n \"messages\": [\n {\n \"role\": \"system\",\n \"content\": \"You are a helpful assistant.\"\n },\n {\n \"role\": \"user\",\n \"content\": \"What is the capital of France?\"\n }\n ],\n \"model\": \"Phi-3-small-8k-instruct\"\n }'\n

curl -X POST \"https://models.inference.ai.azure.com/chat/completions\" \\\n -H \"Content-Type: application/json\" \\\n -H \"Authorization: Bearer $GITHUB_TOKEN\" \\\n -d '{\n \"messages\": [\n {\n \"role\": \"system\",\n \"content\": \"You are a helpful assistant.\"\n },\n {\n \"role\": \"user\",\n \"content\": \"What is the capital of France?\"\n },\n {\n \"role\": \"assistant\",\n \"content\": \"The capital of France is Paris.\"\n },\n {\n \"role\": \"user\",\n \"content\": \"What about Spain?\"\n }\n ],\n \"model\": \"Phi-3-small-8k-instruct\"\n }'\n

curl -X POST \"https://models.inference.ai.azure.com/chat/completions\" \\\n -H \"Content-Type: application/json\" \\\n -H \"Authorization: Bearer $GITHUB_TOKEN\" \\\n -d '{\n \"messages\": [\n {\n \"role\": \"system\",\n \"content\": \"You are a helpful assistant.\"\n },\n {\n \"role\": \"user\",\n \"content\": \"Give me 5 good reasons why I should exercise every day.\"\n }\n ],\n \"stream\": true,\n \"model\": \"Phi-3-small-8k-instruct\"\n }'\n

import os\nfrom azure.ai.inference import ChatCompletionsClient\nfrom azure.ai.inference.models import SystemMessage, UserMessage\nfrom azure.core.credentials import AzureKeyCredential\n\ntoken = os.environ[\"GITHUB_TOKEN\"]\nendpoint = \"https://models.inference.ai.azure.com\"\n# Replace Model_Name\nmodel_name = \"Phi-3-small-8k-instruct\"\n\nclient = ChatCompletionsClient(\n endpoint=endpoint,\n credential=AzureKeyCredential(token),\n)\n\nresponse = client.complete(\n stream=True,\n messages=[\n SystemMessage(content=\"You are a helpful assistant.\"),\n UserMessage(content=\"Give me 5 good reasons why I should exercise every day.\"),\n ],\n model=model_name,\n)\n\nfor update in response:\n if update.choices:\n print(update.choices[0].delta.content or \"\", end=\"\")\n\nclient.close()\n

import ModelClient from \"@azure-rest/ai-inference\";\nimport { AzureKeyCredential } from \"@azure/core-auth\";\n\nconst token = process.env[\"GITHUB_TOKEN\"];\nconst endpoint = \"https://models.inference.ai.azure.com\";\n// Update your modelname\nconst modelName = \"Phi-3-small-8k-instruct\";\n\nexport async function main() {\n\n const client = new ModelClient(endpoint, new AzureKeyCredential(token));\n\n const response = await client.path(\"/chat/completions\").post({\n body: {\n messages: [\n { role: \"system\", content: \"You are a helpful assistant.\" },\n { role: \"user\", content: \"What is the capital of France?\" },\n { role: \"assistant\", content: \"The capital of France is Paris.\" },\n { role: \"user\", content: \"What about Spain?\" },\n ],\n model: modelName,\n }\n });\n\n if (response.status !== \"200\") {\n throw response.body.error;\n }\n\n for (const choice of response.body.choices) {\n console.log(choice.message.content);\n }\n}\n\nmain().catch((err) => {\n console.error(\"The sample encountered an error:\", err);\n});\n

curl -X POST \"https://models.inference.ai.azure.com/chat/completions\" \\\n -H \"Content-Type: application/json\" \\\n -H \"Authorization: Bearer $GITHUB_TOKEN\" \\\n -d '{\n \"messages\": [\n {\n \"role\": \"system\",\n \"content\": \"You are a helpful assistant.\"\n },\n {\n \"role\": \"user\",\n \"content\": \"What is the capital of France?\"\n },\n {\n \"role\": \"assistant\",\n \"content\": \"The capital of France is Paris.\"\n },\n {\n \"role\": \"user\",\n \"content\": \"What about Spain?\"\n }\n ],\n \"model\": \"Phi-3-small-8k-instruct\"\n }'\n

curl -X POST \"https://models.inference.ai.azure.com/chat/completions\" \\\n -H \"Content-Type: application/json\" \\\n -H \"Authorization: Bearer $GITHUB_TOKEN\" \\\n -d '{\n \"messages\": [\n {\n \"role\": \"system\",\n \"content\": \"You are a helpful assistant.\"\n },\n {\n \"role\": \"user\",\n \"content\": \"Give me 5 good reasons why I should exercise every day.\"\n }\n ],\n \"stream\": true,\n \"model\": \"Phi-3-small-8k-instruct\"\n }'\n

Blog Post

GitHub Model Catalog - Getting Started

GitHub Models - Limited Public Beta SIGNUP TODAY

Models Available

Getting Started

Sample Code

Setup

Python Sample

Install dependencies

Run a basic code sample

Run a multi-turn conversation

Stream the output

JavaScript

Install dependencies

Run a basic code sample

Run a multi-turn conversation

Stream the output

REST

Run a basic code sample

Run a multi-turn conversation

Stream the output

FREE Usage and Rate limits for GitHub Models

GitHub Models - Limited Public Beta SIGNUP TODAY

Models Available

Getting Started

Sample Code

Setup

Python Sample

Install dependencies

Run a basic code sample

Run a multi-turn conversation

Stream the output

JavaScript

Install dependencies

Run a basic code sample

Run a multi-turn conversation

Stream the output

REST

Run a basic code sample

Run a multi-turn conversation

Stream the output

FREE Usage and Rate limits for GitHub Models

GitHub Models - Limited Public Beta SIGNUP TODAY

Models Available

Getting Started

Sample Code

Setup

Python Sample

Install dependencies

Run a basic code sample

Run a multi-turn conversation

Stream the output

JavaScript

Install dependencies

Run a basic code sample

Run a multi-turn conversation

Stream the output

REST

Run a basic code sample

Run a multi-turn conversation

Stream the output

FREE Usage and Rate limits for GitHub Models