Model Context Protocol (MCP): Integrating Azure OpenAI for Enhanced Tool Integration and Prompting
Model Context Protocol serves as a critical communication bridge between AI models and external systems, enabling AI assistants to interact directly with various services through a standardized interface. This protocol was designed to address the inherent limitations of standalone AI models by providing them with pathways to access real-time data, perform actions in external systems, and leverage specialized tools beyond their built-in capabilities. The fundamental architecture of MCP consists of client-server communication where the AI model (client) can send requests to specialized servers that handle specific service integrations, process these requests, and return formatted results that the AI can incorporate into its responses. This design pattern enables AI systems to maintain their core reasoning capabilities while extending their functional reach into practical applications that require interaction with external systems and databases MCP has the potential to function as a universal interface, think of it as the virtual / software version of USB-C for AI. Enabling seamless, secure and scalable data exchange between LLMs/AI Agents and external resources. MCP uses a client-server architecture where MCP hosts (AI applications) communicate with MCP servers (data/tool providers). Developers can use MCP to build reusable, modular connectors, with pre-built servers available for popular platforms, creating a community-driven ecosystem. MCP’s open-source nature encourages innovation, allowing developers to extend its capabilities while maintaining security through features like granular permissions. Ultimately, MCP aims to transform AI Agents from isolated chatbots into context-aware, interoperable systems deeply integrated into digital environments. Key elements from the Model Context Protocol: Standardization: MCP provides a standardized way for language models to interact with tools, promoting interoperability. Communication Methods: Supports multiple communication methods, including STDIO and SSE, for flexibility in tool integration. Tool Integration: Enables language models to use external tools, enhancing their functionality and applicability. How Does It Work? MCP operates on a client-server architecture: MCP Hosts: These are the AI applications or interfaces, such as IDEs, or AI tools, that seek to access data through MCP. They initiate requests for data or actions. MCP Clients: These are protocol clients that maintain a one-to-one connection with MCP servers, acting as intermediaries to forward requests and responses. MCP Servers: Lightweight programs that expose specific capabilities through the MCP, connecting to local or remote data sources. Examples include servers for file systems, databases, or APIs, each advertising their capabilities for hosts to utilize. Local Data Sources: These include the computer’s files, databases, and services that MCP servers can securely access, such as reading local documents or querying SQLite databases. Remote Services: External systems available over the internet, such as APIs, that MCP servers can connect to, enabling AI to interact with cloud-based tools or services. Implementation lets try to implement a MCP client using Azure OpenAI with Chainlit and openai python library. By end of this blog you can use attach any MCP server to your client and start using with a simple user interface. So lets get started. First thing we need to ensure is our MCP tools are listed and loaded to our chainlit session. As you install any MCP server , you need to ensure that all the tools of those associated MCP servers are added to your session. .on_chat_start async def start_chat(): client = ChatClient() cl.user_session.set("messages", []) cl.user_session.set("system_prompt", SYSTEM_PROMPT) @cl.on_mcp_connect async def on_mcp(connection, session: ClientSession): result = await session.list_tools() tools = [{ "name": t.name, "description": t.description, "parameters": t.inputSchema, } for t in result.tools] mcp_tools = cl.user_session.get("mcp_tools", {}) mcp_tools[connection.name] = tools cl.user_session.set("mcp_tools", mcp_tools) Next thing we need to do is that we have to flatten the tools as the same will be passed to Azure OpenAI. In this case for each message we pass the loaded MCP server session tools into chat session after flattening it. def flatten(xss): return [x for xs in xss for x in xs] @cl.on_message async def on_message(message: cl.Message): mcp_tools = cl.user_session.get("mcp_tools", {}) tools = flatten([tools for _, tools in mcp_tools.items()]) tools = [{"type": "function", "function": tool} for tool in tools] # Create a fresh client instance for each message client = ChatClient() # Restore conversation history client.messages = cl.user_session.get("messages", []) msg = cl.Message(content="") async for text in client.generate_response(human_input=message.content, tools=tools): await msg.stream_token(text) # Update the stored messages after processing cl.user_session.set("messages", client.messages) Next I define a tool calling step which basically call the MCP session to execute the tool. .step(type="tool") async def call_tool(mcp_name, function_name, function_args): try: print(f"Function Name: {function_name} Function Args: {function_args}") mcp_session, _ = cl.context.session.mcp_sessions.get(mcp_name) func_response = await mcp_session.call_tool(function_name, function_args) except Exception as e: traceback.print_exc() func_response = json.dumps({"error": str(e)}) return str(func_response.content) Next i define a chat client which basically can run as many tools in an iterative manner through for loop (No third party library), simple openai python client. import json from mcp import ClientSession import os import re from aiohttp import ClientSession import chainlit as cl from openai import AzureOpenAI, AsyncAzureOpenAI import traceback from dotenv import load_dotenv load_dotenv("azure.env") SYSTEM_PROMPT = "you are a helpful assistant." class ChatClient: def __init__(self) -> None: self.deployment_name = os.environ["AZURE_OPENAI_MODEL"] self.client = AsyncAzureOpenAI( azure_endpoint=os.getenv("AZURE_OPENAI_ENDPOINT"), api_key=os.getenv("AZURE_OPENAI_API_KEY"), api_version="2024-12-01-preview", ) self.messages = [] self.system_prompt = SYSTEM_PROMPT async def process_response_stream(self, response_stream, tools, temperature=0): """ Recursively process response streams to handle multiple sequential function calls. This function can call itself when a function call is completed to handle subsequent function calls. """ function_arguments = "" function_name = "" tool_call_id = "" is_collecting_function_args = False collected_messages = [] try: async for part in response_stream: if part.choices == []: continue delta = part.choices[0].delta finish_reason = part.choices[0].finish_reason # Process assistant content if delta.content: collected_messages.append(delta.content) yield delta.content # Handle tool calls if delta.tool_calls: if len(delta.tool_calls) > 0: tool_call = delta.tool_calls[0] # Get function name if tool_call.function.name: function_name = tool_call.function.name tool_call_id = tool_call.id # Process function arguments delta if tool_call.function.arguments: function_arguments += tool_call.function.arguments is_collecting_function_args = True # Check if we've reached the end of a tool call if finish_reason == "tool_calls" and is_collecting_function_args: # Process the current tool call print(f"function_name: {function_name} function_arguments: {function_arguments}") function_args = json.loads(function_arguments) mcp_tools = cl.user_session.get("mcp_tools", {}) mcp_name = None for connection_name, session_tools in mcp_tools.items(): if any(tool.get("name") == function_name for tool in session_tools): mcp_name = connection_name break reply_to_customer = function_args.get('reply_to_customer') print(f"reply_to_customer: {reply_to_customer}") # Output any replies to the customer if reply_to_customer: tokens = re.findall(r'\s+|\w+|[^\w\s]', reply_to_customer) for token in tokens: yield token # Add the assistant message with tool call self.messages.append({ "role": "assistant", "content": reply_to_customer, "tool_calls": [ { "id": tool_call_id, "function": { "name": function_name, "arguments": function_arguments }, "type": "function" } ] }) func_response = await call_tool(mcp_name, function_name, function_args) # Add the tool response self.messages.append({ "tool_call_id": tool_call_id, "role": "tool", "name": function_name, "content": func_response, }) # Create a new stream to continue processing new_response = await self.client.chat.completions.create( model=self.deployment_name, messages=self.messages, tools=tools, parallel_tool_calls=False, stream=True, temperature=temperature ) # Use a separate try block for recursive processing try: async for token in self.process_response_stream(new_response, tools, temperature): yield token except GeneratorExit: return return # Check if we've reached the end of assistant's response if finish_reason == "stop": # Add final assistant message if there's content if collected_messages: final_content = ''.join([msg for msg in collected_messages if msg is not None]) if final_content.strip(): self.messages.append({"role": "assistant", "content": final_content}) return except GeneratorExit: return except Exception as e: print(f"Error in process_response_stream: {e}") traceback.print_exc() # Main entry point that uses the recursive function async def generate_response(self, human_input, tools, temperature=0): print(f"human_input: {human_input}") self.messages.append({"role": "user", "content": human_input}) response_stream = await self.client.chat.completions.create( model=self.deployment_name, messages=self.messages, tools=tools, parallel_tool_calls=False, stream=True, temperature=temperature ) try: # Process the initial stream with our recursive function async for token in self.process_response_stream(response_stream, tools, temperature): yield token except GeneratorExit: return Conclusion The Model Context Protocol (MCP) is a pivotal development in AI integration, offering a standardized, open protocol that simplifies how AI models interact with external data and tools. Its client-server architecture, supported by JSON-RPC 2.0 and flexible transports, ensures efficient and secure communication, while its benefits of standardization, flexibility, security, efficiency, and scalability make it a valuable tool for developers. With diverse use cases like knowledge graph management, database queries, and API integrations, MCP is poised to unlock the full potential of AI applications, breaking down data silos and enhancing responsiveness. For those interested in exploring further, the rich documentation, SDKs, and community resources provide ample opportunities to engage with and contribute to this evolving standard. Here is the Githublink for end to end demo: Thanks Manoranjan Rajguru AI Global Black Belt, Asia https://www.linkedin.com/in/manoranjan-rajguru/Exploring Azure OpenAI Assistants and Azure AI Agent Services: Benefits and Opportunities
In the rapidly evolving landscape of artificial intelligence, businesses are increasingly turning to cloud-based solutions to harness the power of AI. Microsoft Azure offers two prominent services in this domain: Azure OpenAI Assistants and Azure AI Agent Services. While both services aim to enhance user experiences and streamline operations, they cater to different needs and use cases. This blog post will delve into the details of each service, their benefits, and the opportunities they present for businesses. Understanding Azure OpenAI Assistants What Are Azure OpenAI Assistants? Azure OpenAI Assistants are designed to leverage the capabilities of OpenAI's models, such as GPT-3 and its successors. These assistants are tailored for applications that require advanced natural language processing (NLP) and understanding, making them ideal for conversational agents, chatbots, and other interactive applications. Key Features Pre-trained Models: Azure OpenAI Assistants utilize pre-trained models from OpenAI, which means they come with a wealth of knowledge and language understanding out of the box. This reduces the time and effort required for training models from scratch. Customizability: While the models are pre-trained, developers can fine-tune them to meet specific business needs. This allows for the creation of personalized experiences that resonate with users. Integration with Azure Ecosystem: Azure OpenAI Assistants seamlessly integrate with other Azure services, such as Azure Functions, Azure Logic Apps, and Azure Cognitive Services. This enables businesses to build comprehensive solutions that leverage multiple Azure capabilities. Benefits of Azure OpenAI Assistants Enhanced User Experience: By utilizing advanced NLP capabilities, Azure OpenAI Assistants can provide more natural and engaging interactions. This leads to improved customer satisfaction and loyalty. Rapid Deployment: The availability of pre-trained models allows businesses to deploy AI solutions quickly. This is particularly beneficial for organizations looking to implement AI without extensive development time. Scalability: Azure's cloud infrastructure ensures that applications built with OpenAI Assistants can scale to meet growing user demands without compromising performance. Understanding Azure AI Agent Services What Are Azure AI Agent Services? Azure AI Agent Services provide a more flexible framework for building AI-driven applications. Unlike Azure OpenAI Assistants, which are limited to OpenAI models, Azure AI Agent Services allow developers to utilize a variety of AI models, including those from other providers or custom-built models. Key Features Model Agnosticism: Developers can choose from a wide range of AI models, enabling them to select the best fit for their specific use case. This flexibility encourages innovation and experimentation. Custom Agent Development: Azure AI Agent Services support the creation of custom agents that can perform a variety of tasks, from simple queries to complex decision-making processes. Integration with Other AI Services: Like OpenAI Assistants, Azure AI Agent Services can integrate with other Azure services, allowing for the creation of sophisticated AI solutions that leverage multiple technologies. Benefits of Azure AI Agent Services Diverse Use Cases: The ability to use any AI model opens a world of possibilities for businesses. Whether it's a specialized model for sentiment analysis or a custom-built model for a niche application, organizations can tailor their solutions to meet specific needs. Enhanced Automation: AI agents can automate repetitive tasks, freeing up human resources for more strategic activities. This leads to increased efficiency and productivity. Cost-Effectiveness: By allowing the use of various models, businesses can choose cost-effective solutions that align with their budget and performance requirements. Opportunities for Businesses Improved Customer Engagement Both Azure OpenAI Assistants and Azure AI Agent Services can significantly enhance customer engagement. By providing personalized and context-aware interactions, businesses can create a more satisfying user experience. For example, a retail company can use an AI assistant to provide tailored product recommendations based on customer preferences and past purchases. Data-Driven Decision Making AI agents can analyze vast amounts of data and provide actionable insights. This capability enables organizations to make informed decisions based on real-time data analysis. For instance, a financial institution can deploy an AI agent to monitor market trends and provide investment recommendations to clients. Streamlined Operations By automating routine tasks, businesses can streamline their operations and reduce operational costs. For example, a customer support team can use AI agents to handle common inquiries, allowing human agents to focus on more complex issues. Innovation and Experimentation The flexibility of Azure AI Agent Services encourages innovation. Developers can experiment with different models and approaches to find the most effective solutions for their specific challenges. This culture of experimentation can lead to breakthroughs in product development and service delivery. Enhanced Analytics and Insights Integrating AI agents with analytics tools can provide businesses with deeper insights into customer behavior and preferences. This data can inform marketing strategies, product development, and customer service improvements. For example, a company can analyze interactions with an AI assistant to identify common customer pain points, allowing them to address these issues proactively. Conclusion In summary, both Azure OpenAI Assistants and Azure AI Agent Services offer unique advantages that can significantly benefit businesses looking to leverage AI technology. Azure OpenAI Assistants provide a robust framework for building conversational agents using advanced OpenAI models, making them ideal for applications that require sophisticated natural language understanding and generation. Their ease of integration, rapid deployment, and enhanced user experience make them a compelling choice for businesses focused on customer engagement. Azure AI Agent Services, on the other hand, offer unparalleled flexibility by allowing developers to utilize a variety of AI models. This model-agnostic approach encourages innovation and experimentation, enabling businesses to tailor solutions to their specific needs. The ability to automate tasks and streamline operations can lead to significant cost savings and increased efficiency. Additional Resources To further explore Azure OpenAI Assistants and Azure AI Agent Services, consider the following resources: Agent Service on Microsoft Learn Docs Watch On-Demand Sessions Streamlining Customer Service with AI-Powered Agents: Building Intelligent Multi-Agent Systems with Azure AI Microsoft learn Develop AI agents on Azure - Training | Microsoft Learn Community and Announcements Tech Community Announcement: Introducing Azure AI Agent Service Bonus Blog Post: Announcing the Public Preview of Azure AI Agent Service AI Agents for Beginners 10 Lesson Course https://aka.ms/ai-agents-beginnersStep-by-step: Integrate Ollama Web UI to use Azure Open AI API with LiteLLM Proxy
Introductions Ollama WebUI is a streamlined interface for deploying and interacting with open-source large language models (LLMs) like Llama 3 and Mistral, enabling users to manage models, test them via a ChatGPT-like chat environment, and integrate them into applications through Ollama’s local API. While it excels for self-hosted models on platforms like Azure VMs, it does not natively support Azure OpenAI API endpoints—OpenAI’s proprietary models (e.g., GPT-4) remain accessible only through OpenAI’s managed API. However, tools like LiteLLM bridge this gap, allowing developers to combine Ollama-hosted models with OpenAI’s API in hybrid workflows, while maintaining compliance and cost-efficiency. This setup empowers users to leverage both self-managed open-source models and cloud-based AI services. Problem Statement As of February 2025, Ollama WebUI, still do not support Azure Open AI API. The Ollama Web UI only support self-hosted Ollama API and managed OpenAI API service (PaaS). This will be an issue if users want to use Open AI models they already deployed on Azure AI Foundry. Objective To integrate Azure OpenAI API via LiteLLM proxy into with Ollama Web UI. LiteLLM translates Azure AI API requests into OpenAI-style requests on Ollama Web UI allowing users to use OpenAI models deployed on Azure AI Foundry. If you haven’t hosted Ollama WebUI already, follow my other step-by-step guide to host Ollama WebUI on Azure. Proceed to the next step if you have Ollama WebUI deployed already. Step 1: Deploy OpenAI models on Azure Foundry. If you haven’t created an Azure AI Hub already, search for Azure AI Foundry on Azure, and click on the “+ Create” button > Hub. Fill out all the empty fields with the appropriate configuration and click on “Create”. After the Azure AI Hub is successfully deployed, click on the deployed resources and launch the Azure AI Foundry service. To deploy new models on Azure AI Foundry, find the “Models + Endpoints” section on the left hand side and click on “+ Deploy Model” button > “Deploy base model” A popup will appear, and you can choose which models to deploy on Azure AI Foundry. Please note that the o-series models are only available to select customers at the moment. You can request access to the o-series models by completing this request access form, and wait until Microsoft approves the access request. Click on “Confirm” and another popup will emerge. Now name the deployment and click on “Deploy” to deploy the model. Wait a few moments for the model to deploy. Once it successfully deployed, please save the “Target URI” and the API Key. Step 2: Deploy LiteLLM Proxy via Docker Container Before pulling the LiteLLM Image into the host environment, create a file named “litellm_config.yaml” and list down the models you deployed on Azure AI Foundry, along with the API endpoints and keys. Replace "API_Endpoint" and "API_Key" with “Target URI” and “Key” found from Azure AI Foundry respectively. Template for the “litellm_config.yaml” file. model_list: - model_name: [model_name] litellm_params: model: azure/[model_name_on_azure] api_base: "[API_ENDPOINT/Target_URI]" api_key: "[API_Key]" api_version: "[API_Version]" Tips: You can find the API version info at the end of the Target URI of the model's endpoint: Sample Endpoint - https://example.openai.azure.com/openai/deployments/o1-mini/chat/completions?api-version=2024-08-01-preview Run the docker command below to start LiteLLM Proxy with the correct settings: docker run -d \ -v $(pwd)/litellm_config.yaml:/app/config.yaml \ -p 4000:4000 \ --name litellm-proxy-v1 \ --restart always \ ghcr.io/berriai/litellm:main-latest \ --config /app/config.yaml --detailed_debug Make sure to run the docker command inside the directory where you created the “litellm_config.yaml” file just now. The port used to listen for LiteLLM Proxy traffic is port 4000. Now that LiteLLM proxy had been deployed on port 4000, lets change the OpenAI API settings on Ollama WebUI. Navigate to Ollama WebUI’s Admin Panel settings > Settings > Connections > Under the OpenAI API section, write http://127.0.0.1:4000 as the API endpoint and set any key (You must write anything to make it work!). Click on “Save” button to reflect the changes. Refresh the browser and you should be able to see the AI models deployed on the Azure AI Foundry listed in the Ollama WebUI. Now let’s test the chat completion + Web Search capability using the "o1-mini" model on Ollama WebUI. Conclusion Hosting Ollama WebUI on an Azure VM and integrating it with OpenAI’s API via LiteLLM offers a powerful, flexible approach to AI deployment, combining the cost-efficiency of open-source models with the advanced capabilities of managed cloud services. While Ollama itself doesn’t support Azure OpenAI endpoints, the hybrid architecture empowers IT teams to balance data privacy (via self-hosted models on Azure AI Foundry) and cutting-edge performance (using Azure OpenAI API), all within Azure’s scalable ecosystem. This guide covers every step required to deploy your OpenAI models on Azure AI Foundry, set up the required resources, deploy LiteLLM Proxy on your host machine and configure Ollama WebUI to support Azure AI endpoints. You can test and improve your AI model even more with the Ollama WebUI interface with Web Search, Text-to-Image Generation, etc. all in one place.Deploy Open Web UI on Azure VM via Docker: A Step-by-Step Guide with Custom Domain Setup.
Introductions Open Web UI (often referred to as "Ollama Web UI" in the context of LLM frameworks like Ollama) is an open-source, self-hostable interface designed to simplify interactions with large language models (LLMs) such as GPT-4, Llama 3, Mistral, and others. It provides a user-friendly, browser-based environment for deploying, managing, and experimenting with AI models, making advanced language model capabilities accessible to developers, researchers, and enthusiasts without requiring deep technical expertise. This article will delve into the step-by-step configurations on hosting OpenWeb UI on Azure. Requirements: Azure Portal Account - For students you can claim $USD100 Azure Cloud credits from this URL. Azure Virtual Machine - with a Linux of any distributions installed. Domain Name and Domain Host Caddy Open WebUI Image Step One: Deploy a Linux – Ubuntu VM from Azure Portal Search and Click on “Virtual Machine” on the Azure portal search bar and create a new VM by clicking on the “+ Create” button > “Azure Virtual Machine”. Fill out the form and select any Linux Distribution image – In this demo, we will deploy Open WebUI on Ubuntu Pro 24.04. Click “Review + Create” > “Create” to create the Virtual Machine. Tips: If you plan to locally download and host open source AI models via Open on your VM, you could save time by increasing the size of the OS disk / attach a large disk to the VM. You may also need a higher performance VM specification since large resources are needed to run the Large Language Model (LLM) locally. Once the VM has been successfully created, click on the “Go to resource” button. You will be redirected to the VM’s overview page. Jot down the public IP Address and access the VM using the ssh credentials you have setup just now. Step Two: Deploy the Open WebUI on the VM via Docker Once you are logged into the VM via SSH, run the Docker Command below: docker run -d --name open-webui --network=host --add-host=host.docker.internal:host-gateway -e PORT=8080 -v open-webui:/app/backend/data --restart always ghcr.io/open-webui/open-webui:dev This Docker command will download the Open WebUI Image into the VM and will listen for Open Web UI traffic on port 8080. Wait for a few minutes and the Web UI should be up and running. If you had setup an inbound Network Security Group on Azure to allow port 8080 on your VM from the public Internet, you can access them by typing into the browser: [PUBLIC_IP_ADDRESS]:8080 Step Three: Setup custom domain using Caddy Now, we can setup a reverse proxy to map a custom domain to [PUBLIC_IP_ADDRESS]:8080 using Caddy. The reason why Caddy is useful here is because they provide automated HTTPS solutions – you don’t have to worry about expiring SSL certificate anymore, and it’s free! You must download all Caddy’s dependencies and set up the requirements to install it using this command: sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list sudo apt update && sudo apt install caddy Once Caddy is installed, edit Caddy’s configuration file at: /etc/caddy/Caddyfile , delete everything else in the file and add the following lines: yourdomainname.com { reverse_proxy localhost:8080 } Restart Caddy using this command: sudo systemctl restart caddy Next, create an A record on your DNS Host and point them to the public IP of the server. Step Four: Update the Network Security Group (NSG) To allow public access into the VM via HTTPS, you need to ensure the NSG/Firewall of the VM allow for port 80 and 443. Let’s add these rules into Azure by heading to the VM resources page you created for Open WebUI. Under the “Networking” Section > “Network Settings” > “+ Create port rule” > “Inbound port rule” On the “Destination port ranges” field, type in 443 and Click “Add”. Repeat these steps with port 80. Additionally, to enhance security, you should avoid external users from directly interacting with Open Web UI’s port - port 8080. You should add an inbound deny rule to that port. With that, you should be able to access the Open Web UI from the domain name you setup earlier. Conclusion And just like that, you’ve turned a blank Azure VM into a sleek, secure home for your Open Web UI, no magic required! By combining Docker’s simplicity with Caddy’s “set it and forget it” HTTPS magic, you’ve not only made your app accessible via a custom domain but also locked down security by closing off risky ports and keeping traffic encrypted. Azure’s cloud muscle handles the heavy lifting, while you get to enjoy the perks of a pro setup without the headache. If you are interested in using AI models deployed on Azure AI Foundry on OpenWeb UI via API, kindly read my other article: Step-by-step: Integrate Ollama Web UI to use Azure Open AI API with LiteLLM ProxyUnleashing Innovation: AI Agent Development with Azure AI Foundry
Creating AI agents using Azure AI Foundry is a game-changer for businesses and developers looking to harness the power of artificial intelligence. These AI agents can automate complex tasks, provide insightful data analysis, and enhance customer interactions, leading to increased efficiency and productivity. By leveraging Azure AI Foundry, organizations can build, deploy, and manage AI solutions with ease, ensuring they stay competitive in an ever-evolving technological landscape. The importance of creating AI agents lies in their ability to transform operations, drive innovation, and deliver personalized experiences, making them an invaluable asset in today's digital age. Let's take a look at how to create an agent on Azure AI Foundry. We'll explore some of the features and experiment with its capabilities in the playground. I recommend by creating a new resource group with a new Azure OpenAI resource. Once the Azure OpenAI Resource is created, follow these steps to get started with Azure AI Foundry Agents. Implementation Overview Open Azure AI Foundry and click on the Azure AI Foundry link at the top right to get to the home page where you'll see all your projects. Click on + Create project then click on Create new hub Give it a name then click Next and Create New resources will be created with your new project. Once inside your new project you should see the Agents preview option on the left menu Select your Azure OpenAI Service resource and click Let's go We can now get started with implementation. A model needs to be deployed. However, it's important to consider which models can be used and their regions for creating these agents. Below is a quick summary of what's currently available. Supported models in Azure AI Agent Service - Azure AI services | Microsoft Learn Other models supported include Meta-Llama-405B-Instruct, Mistral-large-2407, Cohere-command-r-plus, and Cohere-command-r. I've deployed gpt-4 as Global Standard and can now create a new agent. Click on +New agent. A new Agent will be created and details such as the agent instructions, model deployment, Knowledge and Action configurations, and model settings are shown. Incorporating knowledge into AI agents is to enhance their ability to provide accurate, relevant, and context-specific responses. This makes them more effective in automating tasks, answering complex queries, and supporting decision-making processes. Actions enable AI agents to perform specific tasks and interact with various service and data sources. Here we can leverage these abilities by adding a Custom Function, OpenAPI 3.0 specified tool, or an Azure function to help run tasks. The Code Interpreter feature within Actions empowers the agent to read and analyze datasets, generate code, and create visualizations such as graphs and charts. In the next section we'll go deeper with code interpreters' abilities. Code Interpreter For this next step I'll leverage weatherHistory.csv file from Weather Dataset for code interpreter to perform on. Next Actions click on + Add then click on Code interpreter and add the csv file. Update the Instructions to "You are a Weather Data Expert Agent, designed to provide accurate, up-to-date, and detailed weather information." Lets explore what Code interpreter can do. Click on Try in playground on the top right. I'll start by asking "can you tell me which month had the most rain?", code interpreter already knows that I'm asking a question in reference to the data file I just gave it and will breakdown the question into multiple steps to provide the best possible answer. We can see that based on the dataset, August 2010 has the most where 768 instances of rainfall were recorded. We'll take it a step further and create a graph using a different question. Let's ask the agent "ok, can you create a bar chart that shows the amount of rain fall from each year using the provided dataset?" in which the agent will respond with the following: This is just a quick demonstration of how powerful code interpreter can be. Code interpreter allows for efficient data interpretation and presentation as shown above, making it easier to derive insights and make informed decisions. We'll create and add a Bing Grounding Resource which will allow an agent to include real-time public web data into their responses. Bing Grounding Resource A Bing Grounding Resource is a powerful tool that enables AI agents to access and incorporate real-time data from the web into their responses and also ensures that the information provided by the agents is accurate, current, and relevant. An agent will be able to perform Bing searches when needed, fetching up-to-date information and enhancing the overall reliability and transparency of its responses. By leveraging Bing Grounding, AI agents can deliver more precise and contextually appropriate answers, significantly improving user satisfaction and trust. To add a Bing Ground Resource to the agent: Create the Resource: Navigate to the Azure AI Foundry portal and create a new Bing Grounding resource. Add Knowledge: Go to your agent in Azure AI Foundry, click on + Add next to Knowledge on the right side, select Grounding with Big Search, + Create connection. Add connection with API key. The Bing Grounding resource is now added to your agent. In the playground I'll add first ask "Is it raining over downtown New York today?". I will get a live response that also includes the links to the sources where the information was retrieved from. The agent responds as shown below: Next i'll ask the agent "How's should I prepare for the weather in New York this week? Any clothing recommendations?" in which the agent responds: The agent is able to breakdown the question using gpt-4 in detail by leveraging the source information from Bing and providing appropriate information to the user. Other the capabilities of custom functions, OpenAPI 3.0 specified tools, and Azure Functions significantly enhance the versatility and power of Azure AI agents. Custom functions allow agents to perform specialized tasks tailored to specific business needs, while OpenAPI 3.0 specified tools enable seamless integration with a wide range of external services and APIs. Azure Functions further extend the agent's capabilities by allowing it to execute serverless code, automating complex workflows and processes. Together, these features empower developers to build highly functional and adaptable AI agents that can efficiently handle diverse tasks, drive innovation, and deliver exceptional value to users. Conclusion Developing an AI Agent on Azure AI Foundry is a swift and efficient process, thanks to its robust features and comprehensive tools. The platform's Bing Grounding Resource ensures that your AI models are well-informed and contextually accurate, leveraging vast amounts real-time of data to enhance performance. Additionally, the Code Interpreter simplifies the integration and execution of solving complex tasks involving data analysis. By utilizing these powerful resources, you can accelerate the development of intelligent agents that are not only capable of understanding and responding to user inputs but also continuously improving through iterative learning. Azure AI Foundry provides a solid foundation for creating innovative AI solutions that can drive significant value across various applications. Additional Resources: Quickstart - Create a new Azure AI Agent Service project - Azure AI services | Microsoft Learn How to use Grounding with Bing Search in Azure AI Agent Service - Azure OpenAI | Microsoft Learn
Prompt Engineering for OpenAI’s O1 and O3-mini Reasoning Models
Important Attempting to extract the model's internal reasoning is prohibited, as it violates the acceptable use guidelines. This section explores how O1 and O3-mini differ from GPT-4o in input handling, reasoning capabilities, and response behavior, and outlines prompt engineering best practices to maximize their performance. Finally, we apply these best practices to a legal case analysis scenario. Differences Between O1/O3-mini and GPT-4o Input Structure and Context Handling Built-in Reasoning vs. Prompted Reasoning: O1-series models have built-in chain-of-thought reasoning, meaning they internally reason through steps without needing explicit coaxing from the prompt. In contrast, GPT-4o often benefits from external instructions like “Let’s think step by step” to solve complex problems, since it doesn’t automatically engage in multi-step reasoning to the same extent. With O1/O3, you can present the problem directly; the model will analyze it deeply on its own. Need for External Information: GPT-4o has a broad knowledge base and access to tools (e.g. browsing, plugins, vision) in certain deployments, which helps it handle a wide range of topics. By comparison, the O1 models have a narrower knowledge base outside their training focus. For example, O1-preview excelled at reasoning tasks but couldn’t answer questions about itself due to limited knowledge context. This means when using O1/O3-mini, important background information or context should be included in the prompt if the task is outside common knowledge – do not assume the model knows niche facts. GPT-4o might already know a legal precedent or obscure detail, whereas O1 might require you to provide that text or data. Context Length: The reasoning models come with very large context windows. O1 supports up to 128k tokens of input, and O3-mini accepts up to 200k tokens (with up to 100k tokens output), exceeding GPT-4o’s context length. This allows you to feed extensive case files or datasets directly into O1/O3. For prompt engineering, structure large inputs clearly (use sections, bullet points, or headings) so the model can navigate the information. Both GPT-4o and O1 can handle long prompts, but O1/O3’s higher capacity means you can include more detailed context in one go, which is useful in complex analyses. Reasoning Capabilities and Logical Deduction Depth of Reasoning: O1 and O3-mini are optimized for methodical, multi-step reasoning. They literally “think longer” before answering, which yields more accurate solutions on complex tasks. For instance, O1-preview solved 83% of problems on a challenging math exam (AIME), compared to GPT-4o’s 13% – a testament to its superior logical deduction in specialized domains. These models internally perform chain-of-thought and even self-check their work. GPT-4o is also strong but tends to produce answers more directly; without explicit prompting, it might not analyze as exhaustively, leading to errors in very complex cases that O1 could catch. Handling of Complex vs. Simple Tasks: Because O1-series models default to heavy reasoning, they truly shine on complex problems that have many reasoning steps (e.g. multi-faceted analyses, long proofs). In fact, on tasks requiring five or more reasoning steps, a reasoning model like O1-mini or O3 outperforms GPT-4 by a significant margin (16%+ higher accuracy). However, this also means that for very simple queries, O1 may “overthink.” Research found that on straightforward tasks (fewer than 3 reasoning steps), O1’s extra analytical process can become a disadvantage – it underperformed GPT-4 in a significant portion of such cases due to excessive reasoning. GPT-4o might answer a simple question more directly and swiftly, whereas O1 might generate unnecessary analysis. The key difference is O1 is calibrated for complexity, so it may be less efficient for trivial Q&A. Logical Deduction Style: When it comes to puzzles, deductive reasoning, or step-by-step problems, GPT-4o usually requires prompt engineering to go stepwise (otherwise it might jump to an answer). O1/O3 handle logical deduction differently: they simulate an internal dialogue or scratchpad. For the user, this means O1’s final answers tend to be well-justified and less prone to logical gaps. It will have effectively done a “chain-of-thought” internally to double-check consistency. From a prompt perspective, you generally don’t need to tell O1 to explain or check its logic – it does so automatically before presenting the answer. With GPT-4o, you might include instructions like “first list the assumptions, then conclude” to ensure rigorous logic; with O1, such instructions are often redundant or even counterproductive. Response Characteristics and Output Optimization Detail and Verbosity: Because of their intensive reasoning, O1 and O3-mini often produce detailed, structured answers for complex queries. For example, O1 might break down a math solution into multiple steps or provide a rationale for each part of a strategy plan. GPT-4o, on the other hand, may give a more concise answer by default or a high-level summary, unless prompted to elaborate. In terms of prompt engineering, this means O1’s responses might be longer or more technical. You have more control over this verbosity through instructions. If you want O1 to be concise, you must explicitly tell it (just as you would GPT-4) – otherwise, it might err on the side of thoroughness. Conversely, if you want a step-by-step explanation in the output, GPT-4o might need to be told to include one, whereas O1 will happily provide one if asked (and has likely done the reasoning internally regardless). Accuracy and Self-Checking: The reasoning models exhibit a form of self-fact-checking. OpenAI notes that O1 is better at catching its mistakes during the response generation, leading to improved factual accuracy in complex responses. GPT-4o is generally accurate, but it can occasionally be confidently wrong or hallucinate facts if not guided. O1’s architecture reduces this risk by verifying details as it “thinks.” In practice, users have observed that O1 produces fewer incorrect or nonsensical answers on tricky problems, whereas GPT-4o might require prompt techniques (like asking it to critique or verify its answer) to reach the same level of confidence. This means you can often trust O1/O3 to get complex questions right with a straightforward prompt, whereas with GPT-4 you might add instructions like “check your answer for consistency with the facts above.” Still, neither model is infallible, so critical factual outputs should always be reviewed. Speed and Cost: A notable difference is that O1 models are slower and more expensive in exchange for their deeper reasoning. O1 Pro even includes a progress bar for long queries. GPT-4o tends to respond faster for typical queries. O3-mini was introduced to offer a faster, cost-efficient reasoning model – it’s much cheaper per token than O1 or GPT-4o and has lower latency. However, O3-mini is a smaller model, so while it’s strong in STEM reasoning, it might not match full O1 or GPT-4 in general knowledge or extremely complex reasoning. When prompt engineering for optimal response performance, you need to balance depth vs. speed: O1 might take longer to answer thoroughly. If latency is a concern and the task isn’t maximal complexity, O3-mini (or even GPT-4o) could be a better choice. OpenAI’s guidance is that GPT-4o “is still the best option for most prompts,” using O1 primarily for truly hard problems in domains like strategy, math, and coding. In short, use the right tool for the job – and if you use O1, anticipate longer responses and plan for its slower output (possibly by informing the user or adjusting system timeouts). Prompt Engineering Techniques to Maximize Performance Leveraging O1 and O3-mini effectively requires a slightly different prompting approach than GPT-4o. Below are key prompt engineering techniques and best practices to get the best results from these reasoning models: Keep Prompts Clear and Minimal Be concise and direct with your ask. Because O1 and O3 perform intensive internal reasoning, they respond best to focused questions or instructions without extraneous text. OpenAI and recent research suggest avoiding overly complex or leading prompts for these models. In practice, this means you should state the problem or task plainly and provide only necessary details. There is no need to add “fluff” or multiple rephrasing of the query. For example, instead of writing: “In this challenging puzzle, I’d like you to carefully reason through each step to reach the correct solution. Let’s break it down step by step...”, simply ask: “Solve the following puzzle [include puzzle details]. Explain your reasoning.” The model will naturally do the step-by-step thinking internally and give an explanation. Excess instructions can actually overcomplicate things – one study found that adding too much prompt context or too many examples worsened O1’s performance, essentially overwhelming its reasoning process. Tip: For complex tasks, start with a zero-shot prompt (just the task description) and only add more instruction if you find the output isn’t meeting your needs. Often, minimal prompts yield the best results with these reasoning models. Avoid Unnecessary Few-Shot Examples Traditional prompt engineering for GPT-3/4 often uses few-shot examples or demonstrations to guide the model. With O1/O3, however, less is more. The O1 series was explicitly trained to not require example-laden prompts. In fact, using multiple examples can hurt performance. Research on O1-preview and O1-mini showed that few-shot prompting consistently degraded their performance – even carefully chosen examples made them do worse than a simple prompt in many cases. The internal reasoning seems to get distracted or constrained by the examples. OpenAI’s own guidance aligns with this: they recommend limiting additional context or examples for reasoning models to avoid confusing their internal logic. Best practice: use zero-shot or at most one example if absolutely needed. If you include an example, make it highly relevant and simple. For instance, in a legal analysis prompt, you generally would not prepend a full example case analysis; instead, just ask directly about the new case. The only time you might use a demonstration is if the task format is very specific and the model isn’t following instructions – then show one brief example of the desired format. Otherwise, trust the model to figure it out from a direct query. Leverage System/Developer Instructions for Role and Format Setting a clear instructional context can help steer the model’s responses. With the API (or within a conversation’s system message), define the model’s role or style succinctly. For example, a system message might say: “You are an expert scientific researcher who explains solutions step-by-step”. O1 and O3-mini respond well to such role instructions and will incorporate them in their reasoning. However, remember that they already excel at understanding complex tasks, so your instructions should focus on what kind of output you want, not how to think. Good uses of system/developer instructions include: Defining the task scope or persona: e.g. “Act as a legal analyst” or “Solve the problem as a math teacher explaining to a student.” This can influence tone and the level of detail. Specifying the output format: If you need the answer in a structured form (bullet points, a table, JSON, etc.), explicitly say so. O1 and especially O3-mini support structured output modes and will adhere to format requests. For instance: “Provide your findings as a list of key bullet points.” Given their logical nature, they tend to follow format instructions accurately, which helps maintain consistency in responses Setting boundaries: If you want to control verbosity or focus, you can include something like “Provide a brief conclusion after the detailed analysis” or “Only use the information given without outside assumptions.” The reasoning models will respect these boundaries, and it can prevent them from going on tangents or hallucinating facts. This is important since O1 might otherwise produce a very exhaustive analysis – which is often great, but not if you explicitly need just a summary. Ensure any guidance around tone, role, format is included each time. Control Verbosity and Depth Through Instructions While O1 and O3-mini will naturally engage in deep reasoning, you have control over how much of that reasoning is reflected in the output. If you want a detailed explanation, prompt for it (e.g. “Show your step-by-step reasoning in the answer”). They won’t need the nudge to do the reasoning, but they do need to be told if you want to see it. Conversely, if you find the model’s answers too verbose or technical for your purposes, instruct it to be more concise or to focus only on certain aspects. For example: “In 2-3 paragraphs, summarize the analysis with only the most critical points.” The models are generally obedient to such instructions about length or focus. Keep in mind that O1’s default behavior is to be thorough – it’s optimized for correctness over brevity – so it may err on the side of giving more details. A direct request for brevity will override this tendency in most cases. For O3-mini, OpenAI provides an additional tool to manage depth: the “reasoning effort” parameter (low, medium, high). This setting lets the model know how hard to “think.” In prompt terms, if using the API or a system that exposes this feature, you can dial it up for very complex tasks (ensuring maximum reasoning, at the cost of longer answers and latency) or dial it down for simpler tasks (faster, more streamlined answers). This is essentially another way to control verbosity and thoroughness. If you don’t have direct access to that parameter, you can mimic a low effort mode by explicitly saying “Give a quick answer without deep analysis” for cases where speed matters more than perfect accuracy. Conversely, to mimic high effort, you might say “Take all necessary steps to arrive at a correct answer, even if the explanation is long.” These cues align with how the model’s internal setting would operate. Ensure Accuracy in Complex Tasks To get the most accurate responses on difficult problems, take advantage of the reasoning model’s strengths in your prompt. Since O1 can self-check and even catch contradictions, you can ask it to utilize that: e.g. “Analyze all the facts and double-check your conclusion for consistency.” Often it will do so unprompted, but reinforcing that instruction can signal the model to be extra careful. Interestingly, because O1 already self-fact-checks, you rarely need to prompt it with something like “verify each step” (that’s more helpful for GPT-4o). Instead, focus on providing complete and unambiguous information. If the question or task has potential ambiguities, clarify them in the prompt or instruct the model to list any assumptions. This prevents the model from guessing wrongly. Handling sources and data: If your task involves analyzing given data (like summarizing a document or computing an answer from provided numbers), make sure that data is clearly presented. O1/O3 will diligently use it. You can even break data into bullet points or a table for clarity. If the model must not hallucinate (say, in a legal context it shouldn’t make up laws), explicitly state “base your answer only on the information provided and common knowledge; do not fabricate any details.” The reasoning models are generally good at sticking to known facts, and such an instruction further reduces the chance of hallucinationIterate and verify: If the task is critical (for example, complex legal reasoning or a high-stakes engineering calculation), a prompt engineering technique is to ensemble the model’s responses. This isn’t a single prompt, but a strategy: you could run the query multiple times (or ask the model to consider alternative solutions) and then compare answers. O1’s stochastic nature means it might explore different reasoning paths each time. By comparing outputs or asking the model to “reflect if there are alternative interpretations” in a follow-up prompt, you can increase confidence in the result. While GPT-4o also benefits from this approach, it’s especially useful for O1 when absolute accuracy is paramount – essentially leveraging the model’s own depth by cross-verifying. Finally, remember that model selection is part of prompt engineering: If a question doesn’t actually require O1-level reasoning, using GPT-4o might be more efficient and just as accurate. OpenAI recommends saving O1 for the hard cases and using GPT-4o for the rest. So a meta-tip: assess task complexity first. If it’s simple, either prompt O1 very straightforwardly to avoid overthinking, or switch to GPT-4o. If it’s complex, lean into O1’s abilities with the techniques above. How O1/O3 Handle Logical Deduction vs. GPT-4o The way these reasoning models approach logical problems differs fundamentally from GPT-4o, and your prompt strategy should adapt accordingly: Handling Ambiguities: In logical deduction tasks, if there’s missing info or ambiguity, GPT-4o might make an assumption on the fly. O1 is more likely to flag the ambiguity or consider multiple possibilities because of its reflective approach. To leverage this, your prompt to O1 can directly ask: “If there are any uncertainties, state your assumptions before solving.” GPT-4 might need that nudge more. O1 might do it naturally or at least is less prone to assuming facts not given. So in comparing the two, O1’s deduction is cautious and thorough, whereas GPT-4o’s is swift and broad. Tailor your prompt accordingly – with GPT-4o, guide it to be careful; with O1, you mainly need to supply the information and let it do its thing. Step-by-Step Outputs: Sometimes you actually want the logical steps in the output (for teaching or transparency). With GPT-4o, you must explicitly request this (“please show your work”). O1 might include a structured rationale by default if the question is complex enough, but often it will present a well-reasoned answer without explicitly enumerating every step unless asked. If you want O1 to output the chain of logic, simply instruct it to — it will have no trouble doing so. In fact, O1-mini was noted to be capable of providing stepwise breakdowns (e.g., in coding problems) when prompted. Meanwhile, if you don’t want a long logical exposition from O1 (maybe you just want the final answer), you should say “Give the final answer directly” to skip the verbose explanation. Logical Rigor vs. Creativity: One more difference: GPT-4 (and 4o) has a streak of creativity and generative strength. Sometimes in logic problems, this can lead it to “imagine” scenarios or analogies, which isn’t always desired. O1 is more rigor-focused and will stick to logical analysis. If your prompt involves a scenario requiring both deduction and a bit of creativity (say, solving a mystery by piecing clues and adding a narrative), GPT-4 might handle the narrative better, while O1 will strictly focus on deduction. In prompt engineering, you might combine their strengths: use O1 to get the logical solution, then use GPT-4 to polish the presentation. If sticking to O1/O3 only, be aware that you might need to explicitly ask it for creative flourishes or more imaginative responses – they will prioritize logic and correctness by design. Key adjustment: In summary, to leverage O1/O3’s logical strengths, give them the toughest reasoning tasks as a single well-defined prompt. Let them internally grind through the logic (they’re built for it) without micromanaging their thought process. For GPT-4o, continue using classic prompt engineering (decompose the problem, ask for step-by-step reasoning, etc.) to coax out the same level of deduction. And always match the prompt style to the model – what confuses GPT-4o might be just right for O1, and vice versa, due to their different reasoning approaches. Crafting Effective Prompts: Best Practices Summary To consolidate the above into actionable guidelines, here’s a checklist of best practices when prompting O1 or O3-mini: Use Clear, Specific Instructions: Clearly state what you want the model to do or answer. Avoid irrelevant details. For complex questions, a straightforward ask often suffices (no need for elaborate role-play or multi-question prompts). Provide Necessary Context, Omit the Rest: Include any domain information the model will need (facts of a case, data for a math problem, etc.), since the model might not have up-to-date or niche knowledge. But don’t overload the prompt with unrelated text or too many examples – extra fluff can dilute the model’s focus Minimal or No Few-Shot Examples: By default, start with zero-shot prompts. If the model misinterprets the task or format, you can add one simple example as guidance, but never add long chains of examples for O1/O3. They don’t need it, and it can even degrade performance. Set the Role or Tone if Needed: Use a system message or a brief prefix to put the model in the right mindset (e.g. “You are a senior law clerk analyzing a case.”). This helps especially with tone (formal vs. casual) and ensures domain-appropriate language. Specify Output Format: If you expect the answer in a particular structure (list, outline, JSON, etc.), tell the model explicitly. The reasoning models will follow format instructions reliably. For instance: “Give your answer as an ordered list of steps.” Control Length and Detail via Instructions: If you want a brief answer, say so (“answer in one paragraph” or “just give a yes/no with one sentence explanation”). If you want an in-depth analysis, encourage it (“provide a detailed explanation”). Don’t assume the model knows your desired level of detail by default – instruct it. Leverage O3-mini’s Reasoning Effort Setting: When using O3-mini via API, choose the appropriate reasoning effort (low/medium/high) for the task. High gives more thorough answers (good for complex legal reasoning or tough math), low gives faster, shorter answers (good for quick checks or simpler queries). This is a unique way to tune the prompt behavior for O3-mini. Avoid Redundant “Think Step-by-Step” Prompts: Do not add phrases like “let’s think this through” or chain-of-thought directives for O1/O3; the model already does this internally. Save those tokens and only use such prompts on GPT-4o, where they have impact. Test and Iterate: Because these models can be sensitive to phrasing, if you don’t get a good answer, try rephrasing the question or tightening the instructions. You might find that a slight change (e.g. asking a direct question vs. an open-ended prompt) yields a significantly better response. Fortunately, O1/O3’s need for iteration is less than older models (they usually get complex tasks right in one go), but prompt tweaking can still help optimize clarity or format. Validate Important Outputs: For critical use-cases, don’t rely on a single prompt-answer cycle. Use follow-up prompts to ask the model to verify or justify its answer (“Are you confident in that conclusion? Explain why.”), or run the prompt again to see if you get consistent results. Consistency and well-justified answers indicate the model’s reasoning is solid. By following these techniques, you can harness O1 and O3-mini’s full capabilities and get highly optimized responses that play to their strengths. Applying Best Practices to a Legal Case Analysis Finally, let’s consider how these prompt engineering guidelines translate to a legal case analysis scenario (as mentioned earlier). Legal analysis is a perfect example of a complex reasoning task where O1 can be very effective, provided we craft the prompt well: Structure the Input: Start by clearly outlining the key facts of the case and the legal questions to be answered. For example, list the background facts as bullet points or a brief paragraph, then explicitly ask the legal question: “Given the above facts, determine whether Party A is liable for breach of contract under U.S. law.” Structuring the prompt this way makes it easier for the model to parse the scenario. It also ensures no crucial detail is buried or overlooked. Provide Relevant Context or Law: If specific statutes, case precedents, or definitions are relevant, include them (or summaries of them) in the prompt. O1 doesn’t have browsing and might not recall a niche law from memory, so if your analysis hinges on, say, the text of a particular law, give it to the model. For instance: “According to [Statute X excerpt], [provide text]… Apply this statute to the case.” This way, the model has the necessary tools to reason accurately. Set the Role in the System Message: A system instruction like “You are a legal analyst who explains the application of law to facts in a clear, step-by-step manner.” will cue the model to produce a formal, reasoned analysis. While O1 will already attempt careful reasoning, this instruction aligns its tone and structure with what we expect in legal discourse (e.g. citing facts, applying law, drawing conclusions). No Need for Multiple Examples: Don’t supply a full example case analysis as a prompt (which you might consider doing with GPT-4o). O1 doesn’t need an example to follow – it can perform the analysis from scratch.. You might, however, briefly mention the desired format: “Provide your answer in an IRAC format (Issue, Rule, Analysis, Conclusion).” This format instruction gives a template without having to show a lengthy sample, and O1 will organize the output accordingly. Control Verbosity as Needed: If you want a thorough analysis of the case, let O1 output its comprehensive reasoning. The result may be several paragraphs covering each issue in depth. If you find the output too verbose or if you specifically need a succinct brief (for example, a quick advisory opinion), instruct the model: “Keep the analysis to a few key paragraphs focusing on the core issue.” This ensures you get just the main points. On the other hand, if the initial answer seems too brief or superficial, you can prompt again: “Explain in more detail, especially how you applied the law to the facts.” O1 will gladly elaborate because it has already done the heavy reasoning internally. Accuracy and Logical Consistency: Legal analysis demands accuracy in applying rules to facts. With O1, you can trust it to logically work through the problem, but it’s wise to double-check any legal citations or specific claims it makes (since its training data might not have every detail). You can even add a prompt at the end like, “Double-check that all facts have been addressed and that the conclusion follows the law.” Given O1’s self-checking tendency, it may itself point out if something doesn’t add up or if additional assumptions were needed. This is a useful safety net in a domain where subtle distinctions matter. Use Follow-Up Queries: In a legal scenario, it’s common to have follow-up questions. For instance, if O1 gives an analysis, you might ask, “What if the contract had a different clause about termination? How would that change the analysis?” O1 can handle these iterative questions well, carrying over its reasoning. Just remember that, if the project you ar working on, the interface doesn’t have long-term memory beyond the current conversation context (and no browsing), each follow-up should either rely on the context provided or include any new information needed. Keep the conversation focused on the case facts at hand to prevent confusion. By applying these best practices, your prompts will guide O1 or O3-mini to deliver high-quality legal analysis. In summary, clearly present the case, specify the task, and let the reasoning model do the heavy lifting. The result should be a well-reasoned, step-by-step legal discussion that leverages O1’s logical prowess, all optimized through effective prompt construction. Using OpenAI’s reasoning models in this way allows you to tap into their strength in complex problem-solving while maintaining control over the style and clarity of the output. As OpenAI’s own documentation notes, the O1 series excels at deep reasoning tasks in domains like research and strategy– legal analysis similarly benefits from this capability. By understanding the differences from GPT-4o and adjusting your prompt approach accordingly, you can maximize the performance of O1 and O3-mini and obtain accurate, well-structured answers even for the most challenging reasoning tasks.Learn about Azure AI during the Global AI Bootcamp 2025
The Global AI Bootcamp starting next week, and it’s more exciting than ever! With 135 bootcamps in 44 countries, this is your chance to be part of a global movement in AI innovation. 🤖🌍 From Germany to India, Nigeria to Canada, and beyond, join us for hands-on workshops, expert talks, and networking opportunities that will boost your AI skills and career. Whether you’re a seasoned pro or just starting out, there’s something for everyone! 🚀 Why Attend? 🛠️ Hands-on Workshops: Build and deploy AI models. 🎤 Expert Talks: Learn the latest trends from industry leaders. 🤝 Network: Connect with peers, mentors, and potential collaborators. 📈 Career Growth: Discover new career paths in AI. Don't miss this incredible opportunity to learn, connect, and grow! Check out the event in your city or join virtually. Let's shape the future of AI together! 🌟 👉 Explore All Bootcamps
Built-in Enterprise Readiness with Azure AI Agent Service
Ensure enterprise-grade security and compliance with Private Network Isolation (BYO VNet) in Azure AI Agent Service. This feature allows AI agents to operate within a private, isolated network, giving organizations full control over data and networking configurations. Learn how Private Network Isolation enhances security, scalability, and compliance for mission-critical AI workloads.Announcing Provisioned Deployment for Azure OpenAI Service Fine-tuning
You've fine-tuned your models to make your agents behave and speak how you'd like. You've scaled up your RAG application to meet customer demand. You've now got a good problem: users love the service but want it snappier and more responsive. Azure OpenAI Service now offers provisioned deployments for fine-tuned models, giving your applications predictable performance with predictable costs! 💡 What is Provisioned Throughput? If you're unfamiliar with Provisioned Throughput, it allows Azure OpenAI Service customers to purchase capacity in terms of performance needs instead of per-token. With fine-tuned deployments, it replaces both the hosting fee and the token-based billing of Standard and Global Standard (now in Public Preview) with a throughput-based capacity unit called provisioned through units (PTU). Every PTU corresponds to a commitment of both latency and throughput in Tokens per Minute (TPM). This differs from Standard and Global Standard which only provide availability guarantees and best-effort performance. With fine-tuned deployments, it replaces both the hosting fee and the token-based billing of Standard and Global Standard (now in Public Preview) with a throughput-based capacity unit called a PTU. 🤔 Is this the same PTU I'm already using? You might already be using Provisioned Throughput Units with base models and with fine-tuned models they work the same way. In fact, they're completely interchangeable! Already have quota in North Central US for 800 PTU and an annual Azure reservation rate? PTUs are interchangeable and model independent meaning you can get started with using them for fine-tuning immediately without any additional steps. Just select Provisioned Managed (Public Preview) from the model deployment dialog and set your PTU allotment. 📋 What's available in Public Preview? We're offering provisioned deployment in two regions for both gpt-4o (2024-08-06) and gpt-4o-mini (2024-07-18) to support Azure OpenAI Service customers: North Central US Switzerland West If your workload requires regions other than the above, please make sure to submit a request so we can consider it for General Availability. 🙏 🚀 How do I get started? If you don't already have PTU quota from base models, the easiest way to get started and shifting your fine-tuned deployments to provisioned is: Understand your workload needs. Is it spiky but with a baseline demand? Review some of our previous materials on right-sizing PTUs (or have CoPilot summarize it for you 😆). Estimate the PTUs you need for your workload by using the calculator. Increase your regional PTU quota, if required. Deploy your fine-tuned models to secure your Provisioned Throughput capacity. Make sure to purchase an Azure Reservation to cover your PTU usage to save big. Have a spiky workload? Combine PTU and Standard/Global Standard and configure your architecture for spillover. Have feedback as you continue on your PTU journey with Azure OpenAI Service? Let us know how we can make it better!Capacity's AI Answer Engine® leveraged Phi to deliver better results for their customers, faster
Capacity an all-in-one Support Automation Platform, provides organizations with the ultimate Answer Engine®. They needed a way to help unify diverse datasets across tens of millions of search results and billions of interactions and make information more easily accessible and understandable for their customers. By leveraging Phi—Microsoft’s family of powerful small language models offering groundbreaking performance at low cost and low latency—Capacity provides the enterprise with an effective AI knowledge management solution that democratizes knowledge on large teams securely and in a way that maximizes value to the customer. With Phi, Capacity’s Answer Engine® improved results quality and scale, so customers save both time and money by more quickly finding the rich information they invested in to do their best work. What was the challenge? Enterprise employees struggle to find the data they need searching through isolated, untagged content, leading to frustration and wasted time. To address this, Capacity’s Answer Engine® retrieves information across diverse enterprise systems, repositories and sources, instantly delivering the exact answers needed to inform work and make faster decisions. At the same time, AI can only go so far to unify and enrich this data. Capacity addressed the challenge by leveraging Phi using Azure Serverless API to experiment on the effectiveness of Language Model-based tagging infrastructure. They applied prompt engineering, adherence workflows, and at-scale testing to better prepare Answers for search and create a more universal Answer Engine®. Why did Capacity choose Phi? Capacity chose Phi-3.5-mini for its speed, cost-effectiveness, and deployment flexibility. With Azure Models as a Service (MaaS), Capacity was able to use the Phi family models without having to provision GPUs or manage back-end operations, saving their team time, effort, and cost. They used prompt engineering and metadata tagging to optimize search results, ultimately improving development speed and query processing efficiency. Additionally, the favorable MIT Open Source licensing of the Phi family provided a strong long-term strategy for their private cloud deployment, vectorization, and query routing activities. "From our initial experiments, what truly impressed us about the Phi was its remarkable accuracy and the ease of deployment, even before customization. Since then, we've been able to enhance both accuracy and reliability, all while maintaining the cost-effectiveness and scalability we valued from the start." ~ Steve Frederickson, Head of Product, Answer Engine How did they solve for it? To achieve their goal, Capacity implemented Phi-3-mini and Phi-3.5-mini Model-as-a-Service, using both 4k and 128K variants with some prompt engineering. This allowed them to accelerate development on their AI-powered Answer Engine and help their enterprise customers deliver the right information to their end users quickly and accurately. When presenting an Answer to their customer’s end user, Capacity wanted their AI Answer engine to instantly present the full Answer along with all the content metadata around it, so the end user could feel confident in their search results. To accomplish this, Capacity engineers split the tasks for Phi into preprocessing and real-time flows. In preprocessing, they generated metadata such as title summaries for answers, keyword tags for search, and other information to the index. This pre-work was done offline and ahead of time. Depending on the tagging task required for each Answer, they calculated the needed token size then rerouted the query to the appropriate Phi model. At query time, Phi models pre-process the query to retrieve the most relevant content. The split tasks for Phi enabled repeatable performance, keeping the responsive query times users expect while enhancing results with new functionality and increased retrieval relevance. At the same time, the cost-efficiency of Phi was able to produce the same or better qualitative results for preprocessing with a 4.2x cost savings as compared to the competing workflow. The considerable cost savings on the preprocessing allows Capacity to scale to ever-growing datasets. While the increased retrieval relevance fosters sustained growth and enhances user satisfaction. After integrating Phi, Capacity observed significant improvements in both performance and customer satisfaction. The AI-powered solutions provided faster and more accurate information retrieval, which reduced time users spent searching for information. Additionally, the seamless integration of datasets with the Phi-3.5-mini model as a service significantly empowered Capacity to address a wide range of use cases with enhanced efficacy, ultimately elevating the user experience. Steve Frederickson, Capacity's Head of Product, Answer Engine, noted, “Integrating our datasets with the Phi-3.5-mini model was effortless. We have found new opportunity in its speed, and the enriched customer experience of GenAI enables us to resolve customer issues more effectively, delivering a superior user experience." Capacity also shared some valuable tips for other organizations looking to implement similar AI solutions. They recommended designing the system to optimize for query performance and retrieval accuracy, including adding metadata and keyword tags to optimize search efficiency. They also emphasize the importance of choosing the right AI model based on the capability and scalability, to balance speed and cost-effectiveness. The next step Implementing Phi has revolutionized Capacity’s approach to knowledge management, providing their enterprise customers with efficient and accurate information retrieval solutions. Their success highlights the potential of the Phi model family to transform enterprise operations and improve user experiences. Looking ahead, Capacity plans to explore additional state-of-the-art models such as Phi-4-multimodal and Phi-4-mini for more complex reasoning tasks like multilingual support and image understanding scenarios. They also aim to fine-tune their solutions to enhance their knowledge graph and improve interoperability among different institutional knowledge bases. By continuously innovating and leveraging advanced AI technology, the Capacity Answer Engine® is well-positioned to remain at the forefront of knowledge management solutions, helping organizations do their best work with the complexities of information retrieval and discovery. Learn more about the Phi family of models here: About Phi Learn about the latest updates Download the models
