Integrate GPT4o (Azure Open AI) in Teams channel via Logic App with image supportability

As per the previous blog (Integrate Azure Open AI in Teams Channel via Logic App - Microsoft Community Hub) which only supports for plain text conversation.

For now, Azure Open AI released GPT-4o model which integrates text and images in a single model, enabling it to handle multiple data types simultaneously.

This post is to introduce how to upgrade to GPT-4o with capability for image processing.

Prerequisite

Most of the prerequisites are the same, just instead of GPT-3.5, we need to have a GPT-4o deployment in Azure Open AI environment.

Since GPT-4o is not available for all the regions, so you need to refer to the document (Azure OpenAI Service models - Azure OpenAI | Microsoft Learn) to see whether your current region supports for GPT-4o or not. If not, then you will have to deploy a new Azure Open AI.

Mechanism

In Open AI API, we can see that when send image to API, we need to provide image as base64 encoded content.

[
  {
    "role": "user",
    "content": [
      {
        "type": "image_url",
        "image_url": {
          "url": "data:image/png;base64,iVBORw0AAABJR.....U5ErkJggg=="
        }
      },
      {
        "type": "text",
        "text": "what is this logo"
      }
    ]
  }
]

So the only thing for us is to get image binary data from Teams channel which we can achieve that via using Teams Graph API: Get chatMessageHostedContent - Microsoft Graph v1.0 | Microsoft Learn

Template and parameters

You can find the template in Drac-Zhang/LogicApp_Teams_OpenAI_Integration_WithGPT4o (github.com)

Parameters of the template:

Known Issues

Except the same known issues from previous blog, we have a new one:

In "Send a Microsoft Graph Http Request" action in "Microsoft Teams" connector, we have an issue that cannot correctly response hosted content as binary data, so we have to use "Invoke an HTTP request" which results to extra API connection (webcontents) and it requests to manual authentication after the deployment.

Sample Chat