AMA: GPT-4o Audio model revolutionizes your Copilot and other AI applications

Thanks for joining us - We love opportunities to connect with our wonderful community of developers!

Glad to hear this is helpful, Srini. We will do it more often in the future. Stay tuned for our future events.

It's been great talking with you! This is for sure just the beginning; we'll have more coming soon.

Hello, everyone! Travis here -- I'm the engineering manager, located in Seattle, whose team works on OpenAI and Azure OpenAI SDK libraries. Excited to be part of this AMA and thank you for your questions! From the code perspective, I'll plug the repository we set up to help get started with gpt-4o-realtime-preview audio features: https://github.com/azure-samples/aoai-realtime-audio-sdk . This is all a very new feature area and the resources we have are a rapidly evolving work in progress, but we're excited to see how quickly things are growing and the cool things people are already starting to do with the preview of the new /realtime features.

Hi everybody!! We're excited to be here with the tech community to answer any questions about the new GPT-4o-realtime speech capabilities, including how you can get started with building your applications!

Whether you plan to write code or not, the process to get started with the new gpt-4o-realtime-preview model is fairly straightforward: (0) if you don't have one yet, create an Azure account via Azure Portal; (1) create an Azure OpenAI Service resource in one of the two preview regions (eastus2 or swedencentral); (2) using Azure AI Studio, create a gpt-4o-realtime-preview model deployment in your eastus2 or swedencentral; (3) use the "Real-time audio" playground (left navigation bar) to check out the new model with a live, browser-based voice-in/voice-out experience. From there, there are code samples -- including ones that just require setting environment variables and running -- at https://github.com/azure-samples/aoai-realtime-audio-sdk . We don't have much in the way of "build a new experience with no code whatsoever" yet given how new this all is, but we're continually looking for ways to make it easier to integrate this new /realtime feature set and other Azure OpenAI capabilities.

Hey Srini, thanks for the questions. Since we are the platform team serving these models,

(1) I'm sure the Copilot and Copilot Studio teams are cooking up something for no-code and low-code developers to take advantage of this new modality.

(2) For a non-developer, we have some helpful documentation to get started: https://aka.ms/oai/docs. Without code you can create a resource by using the Azure Portal and launching the OpenAI Studio once a resource has been created. For now, we'd recommend creating the resource in either East US 2 or Sweden Central Azure regions. Check out this quickstart guide for how to interact with GPT-4o using the Realtime API within the Studio. For more code-first development, you can check out these pre-made samples: https://github.com/azure-samples/aoai-realtime-audio-sdk

(3) Much like the text-based capabilities of LLMs, these models can natively interpret many languages other than English. As always, it is best to test GPT-4o's audio capabilities to ensure the model has the fidelity you need to meet your business requirements.

(4) This voice capability unlocks a new modality in interacting with applications where AI becomes the universal interface. I've found inspiration in the scenarios our customers we gave early access like Bosch and Lyrebird Health as well as the examples OpenAI demonstrated in their spring update.

I see you've posted this question twice - it looks like a complete duplicate. Please let us know which of these threads you want us to answer these questions in - I'll delete the other one.

We don't yet have higher-level abstractions for Teams specifically, but between OpenAI and Microsoft we've started some client library coverage to expose the new capabilities of the gpt-4o-realtime-preview model and the /realtime API:

• The OpenAI .NET SDK (https://github.com/openai/openai-dotnet) (as well as the AOAI companion library, Azure.AI.OpenAI) already has early support for a client integrated into 2.1.0-beta.1
• Python has an early standalone client we're iterating on at aoai-realtime-audio-sdk/python at main · Azure-Samples/aoai-realtime-audio-sdk (github.com)
• JavaScript has an early standalone library: openai/openai-realtime-api-beta: Node.js + JavaScript reference client for the Realtime API (beta) (github.com) and we also one at aoai-realtime-audio-sdk/javascript at main · Azure-Samples/aoai-realtime-audio-sdk (github.com)

We've already seen developers prototype applications with multiple agents talking to people (and each other!) using the /realtime capabilities and the results are very cool. Very possible with the tools we have today!

For getting started with no code, Azure AI Studio (https://ai.azure.com/) includes the new /realtime capabilities in its Playground experience (much like OpenAI's) and allows you to interact with the new gpt-4o-realtime-preview model in the browser without needing to write any lines of code -- it's also the best way to create and manage the model deployments on your Azure OpenAI Service resource. Low- and no-code development of /realtime-based experiences is an area we're actively looking into. The new API endpoint and its WebSocket-based capabilities are considerably more complex than the prior REST-based operations like /chat/completions, and making it as approachable and easy to integrate as possible (including with no code at all, where it makes sense) is a major priority for us when it comes to Developer Experience.

(5) Yes! Azure OpenAI Service is an enterprise-grade platform hosting the latest models from OpenAI. Check out our data, privacy, and security documentation for more information.

 

(6) Cost implications are use case dependent. Items that are likely to impact your costs include: Input/output audio ratio, average length of audio session, number of concurrent connections, deployment type, and throughput requirements. We recommend that developers start with small scale tests and development and evaluate costs before scaling to production.

 

(7) The "On your data" feature of Azure OpenAI is always a great way to get started quickly, then you can create a web app or a chatbot to share with your team or organization. Check it out!

 

 

I see you've posted this question twice - it looks like a complete duplicate. Please let us know which of these threads you want us to answer these questions in - I'll delete the other one.

• .NET sample is here
• Javascript sample is here
• Python sample is here

As for pricing, we will be sharing more details very soon! Similar to other Azure OpenAI models, we will start with Pay-as-you-go pricing and introduce others like provisioned, batch, etc. over time. If there are other pricing models that would better help you scale your applications, we always welcome feedback!

Microsoft put security above all else. Security comes first when designing any product or service including LLM. Security protections are enabled and enforces by default, require no extra effort and are not optional. When you use Azure OpenAI Service, your prompts (inputs) and completions (outputs), your embeddings, and your training data, are all your own data, meaning your data are NOT available to others for any uses. As for pricing, please get the latest from your sales representatives or from our pricing page. As for your last question, we will relay it to our CoPilot team to follow up or stay tuned with the Copilot product updates.