Announcing Multimodal Innovations in Generative AI with Azure OpenAI Service: Microsoft Build 2024

Microsoft

May 21, 2024

Welcome to Microsoft Build 2024. Interested in learning more about the latest and greatest developments by Azure OpenAI Service? You’ve come to the right place. With over 53,000 customers building with Azure AI, we are excited to bring more innovation to our developer community heading into the summer. We want you to enjoy the warmer weather, but when you need a break from the beaches and parks, jump into the Azure AI Studio and catch up on all things generative AI. We can't wait to learn what you've built with our new models, features and platform updates.

Product Launches

Today’s Build announcements reflect an evolution in Azure OpenAI Service and the potential multimodality. Discover how Microsoft is serving customers with major generative AI advancements.

GPT-4o: Microsoft has announced the general availability of GPT-4o, OpenAI’s new flagship model on Azure AI. This multimodal model integrates text, vision, and in the future, audio capabilities, setting a new standard for generative and conversational AI experiences. GPT-4o is available now in Azure OpenAI Service API and Azure AI Studio with support for text and image. GPT-4o is our first model offered both with a Global and Regional deployment. Pricing is $5/1M tokens for input and $15.00/1M tokens for output. Pricing is subject to change starting 5/24.

Fine-tuning for GPT-4: GPT-4 is our most advanced fine tunable model yet: it outperforms GPT-35-Turbo on a variety of tasks, and thanks to its alignment training it shows improved factuality, steerability, and instruction following. Today, we're making GPT-4 fine tuning available in public preview, so you can now customize it with your own training data. This allows for unparalleled customization of AI models, ensuring outputs are closely aligned with an organization’s brand voice and specific needs.

Preview refresh of the Assistants API: This paves the way for creating advanced copilots virtual assistants and chatbots their ability to take actions, handle complex tasks, and achieve specific goals autonomously. The latest advancement in OpenAI’s multimodal technology, this model can understand and generate responses in text, vision and audio. GPT-4o is currently available in preview with full API access.

Launch of Multimodal Capabilities with GPT-4 Turbo (featuring vision): Introduces a new dimension to AI applications, enabling the creation of content that spans across text, images, and more for a richer user experience.

Azure AI On Your Data (OYD) Integrated with Retrieval-Augmented Generation (RAG): Facilitates building custom copilots, improving the creation of more intuitive and interactive solutions.

Messaging Insights for WhatsApp with Azure Communication Services: Now in preview through Azure OpenAI Service via Azure Communication Services, this feature enables businesses to extract meaningful insights from WhatsApp messages. It leverages language detection, translation, sentiment analysis, key phrase extraction, and intent recognition to enhance the “user to business” communication flow.

Model Customization with Azure AI: We are thrilled to announce the launch of our Model Customization for Azure AI, an engineering service designed to accelerate our co-innovation with customers to deliver tailored AI solutions. Our commitment to empowering our customers extends beyond the provision of tools and platforms; we are offering an opportunity for selected customers to collaborate closely with our engineering and research teams to develop custom models tailored to their unique domain-specific needs. For more information, please reach out to your Microsoft representatives or account managers.

Deployment Innovation: Azure OpenAI Service provides customers with choice on the hosting structure that fits their business and usage patterns, offering three types of deployments: Batch, a new offering coming soon, in addition to Standard (On-Demand), and Provisioned. With all of them, you can perform the same inference operations, but the billing, scale, and performance differ significantly. Learn more about pricing.

Phi-3 Open Models: Now available in the Azure AI Studio alongside OpenAI models, Phi-3 is a family of small open models developed by Microsoft that supports developers in building cost-efficient and responsible multimodal generative AI applications. Phi-3-mini, Phi-3-small, Phi-3-medium, and Phi-3-vision are all super-sets of previous versions of Phi and can be highlighted for production use cases.

For all announcements and related documentation, bookmark the following What's New with Azure OpenAI Service page.

New Case Studies for featured customers at Build

We are inspired by our customers and partners who are leveraging Azure OpenAI Service to drive innovation and achieve remarkable outcomes across various industries. Learn more about the achievements made by the companies below by attending scheduled breakout sessions and demos during Build.

H&R Block is utilizing AI Tax Assist to help tax professionals and filers reduce both the work and time needed to file their own taxes. Their use of Azure OpenAI Service and Azure AI Studio, alongside the potential integration of Azure AI Search for RAG, exemplifies how AI can streamline complex processes to enhance productivity.

Mercedes-Benz is integrating GPT-4 Turbo with Vision via Azure OpenAI Service to transform their MBUX Voice Assistant and dashcams, enabling the car to understand the surrounding environment and provide context for speech assistance via a "Hey Mercedes“ cue.

TomTom offers an immersive in-car infotainment system powered by Azure OpenAI Service, Azure Kubernetes Service, and CosmosDB. The system Digital Cockpit offers an immersive user interface that demonstrates the impact of advanced technologies in creating a seamless and enriched driving environment.

Unity Technologies created Muse Chat, a copilot for coding and documentation, allowing developers to create and perfect their own video games with the help of Azure OpenAI Service and content filters through Azure AI Content Safety. This demonstrates how AI can foster creativity and streamline development in the gaming industry.

Vodafone's TOBI Chatbot is revolutionizing customer interactions by efficiently handling calls and empowering agents through context-aware conversations and call transcriptions, while the creation of their SuperAgent, a conversational AI search copilot, help agents respond to more complex customer inquiries. Adoption of Azure OpenAI Service and Azure AI Studio Az highlights the potential for AI to enhance customer journeys and agent efficiency.

Additional customers featured at Build

Coca-Cola is leveraging the Assistants API with GPT-4 Turbo to improve the productivity of its 30,000 associates with its standardized assistants for all departments, helping with enhanced business intelligence, data synthesis, strategic planning, and risk management.

Freshworks uses Freddy Copilot to provide conversational assistance and informed insights to employees and customers. Not only has it increased productivity, but it's also enabled more intuitive app development with the support of Assistants API in

T-Mobile's Network Copilot allows engineers to use LLMs over unstructured content and structured data to quickly determine key facts and troubleshoot customer issues. This use case highlights the integration of Azure OpenAI Service and Azure AI Content Safety to mitigate customer issues efficiently.

WPP is exploring the use of video, images, and speech to accelerate content creation with Azure OpenAI Service (GPT-4 with Vision). Generative AI and multimodality are allowing them to push the boundaries of creative expression and efficiency.

Microsoft's Generative AI Hackathon Winners

Multimodal innovation is drawing the world's attention at Microsoft Build. Many use cases came to life with Microsoft's Generative AI Hackathon. Check out the amazing winners that were able to cook up projects that utilized Azure AI and VS Code extensions.

First Place = ChatEDU (aims to shift students’ use of generative AI from task and assignment automation to a dynamic copilot that works and learns with them, not for them)

Second Place = GARVIS (eliminates the need for users to verbally describe visual problems by directly analyzing and understanding the scene, and it replaces text-based instructions with intuitive, visual demonstrations directly in the user's environment)

Third Place = ModeMixer (simplifies fashion design by helping small business owners generate trend-driven, customized collections and manufacturer-ready tech packs using AI)

Honorable Mention:

AI Briefing Room (news source podcast which addresses the issue of tech news being lengthy and difficult to access for those outside the tech community, particularly international listeners)

Best Use of VS Code Extension Bonus Prize):

FarmFundAI (creates a funding platform for farmers with focus on sustainability using Cyberphysical systems and AI)