Exploring Azure AI Agent Service: A Leap in Conversational AI

The Azure OpenAI Assistants API introduced powerful features such as:

• Conversation State Management: Efficiently handling token limits and state.
• Automated Tool Actions: Automatically interpreting user inputs to execute actions, such as writing Python code (Code interpreter) or performing vector searches.
• Function Calling: Dynamically identifying and suggesting appropriate custom functions for execution.

Azure AI Agent Service enhances these capabilities by integrating advanced tools and actions, streamlining the developer experience even further:

1. REST APIs: Supports REST APIs compliant with Open API 3.0. It automatically identifies and invokes the appropriate API based on user intent using Swagger definitions.
2. Azure Function Apps: Azure Function Apps can be automatically invoked based on user intent, as with REST APIs mentioned above.
3. Knowledge Tools: Includes Bing Search and Azure AI Search, for data grounding.
4. Azure Logic Apps: Tooling support for Azure Logic Apps is expected but not available at the time of this writing. Currently, function calling must be used to identify and invoke the appropriate Logic App manually.
5. Choice of Language Models: Apart from gpt-4o, gpt-4o-mini models, it also supports the use of Llama 3.1-70B-instruct, Mistral-large-2407, Cohere command R+

Working with the Azure AI Agent Service

I have explored the Azure AI Agent Service’s capabilities through the sample Contoso Retail Shopping Assistant, a simple conversational AI bot.

It implements a combination of tool actions, mentioned below

• REST API Integration: Users can search for orders by category, by category and price, or place orders. Using Swagger definitions, the Azure AI Agent service dynamically identifies and calls APIs based on user input without custom code.
• Azure Logic Apps: Handles shipment order creation post-purchase. Manual function calling is used for creating shipment orders.

Apart from that, the App uses:

• Natural Language Processing: Leverages the gpt-4o-mini model to understand and process user queries effectively.
• Microsoft Bot Framework is used to build the sample App. 

Creating the Agent

Create the AI Agent with access to the necessary tools.

1. OpenApiTool - for REST API invocation
2. FunctionTool - for function calling (to trigger Azure Logic App)

functions = FunctionTool(functions=user_functions)

def create_agent():

        project_client = AIProjectClient.from_connection_string(
            credential=DefaultAzureCredential(),
            conn_str=config.az_agentic_ai_service_connection_string,
        )
        
        # read the swagger file for the Contoso Retail Fashion API definition
        with open("./data-files/swagger.json", "r") as f:
            openapi_spec = jsonref.loads(f.read())
        auth = OpenApiAnonymousAuthDetails()

        # Initialize agent OpenApi tool using the read in OpenAPI spec
        api_tool = OpenApiTool(
            name="contoso_retail_fashion_api",
            spec=openapi_spec,
            description="help users order a product based on id and quantity, search products by category, and search for products based on their category and price.",
            auth=auth,
        )

        # Initialize agent toolset with user functions
        toolset = ToolSet()
        toolset.add(functions)
        toolset.add(api_tool)

        # print("toolsets definition", toolset.definitions)

        agent = project_client.agents.create_agent(
            model="gpt-4o-mini",
            name="contoso-retail-fashions-ai-agent",
            instructions="You are an AI Assistant tasked with helping the customers of Contoso retail fashions with their shopping requirements. You have access to the APIs in contoso_retail_fashion_api that you need to call to respond to the user queriest",
            tools=toolset.definitions,
        )

        print(f"created agent with id {agent.id}")
        

The function that makes the call to Azure Logic Apps to create the delivery order:

def create_delivery_order(order_id: str, destination: str) -> str:
    """
    creates a consignment delivery order (i.e. a shipment order) for the given order_id and destination location

    :param order_id (str): The order number of the purchase made by the user.
    :param destination (str): The location where the order is to be delivered.
    :return: generated delivery order number.
    :rtype: Any
    """

    api_url = config.az_logic_app_url
    print("making the Logic app call.................")
    # make a HTTP POST API call with json payload
    response = requests.post(
        api_url,
        json={"order_id": order_id, "destination": destination},
        headers={"Content-Type": "application/json"},
    )

    print("response from logic app", response.text)
    return json.dumps(response.text)

For simplicity, the HTTP Callable endpoint from the Azure Portal is used directly, complete with the SAS Token. The function 'create_delivery_order' calls this URL directly to trigger the Logic App.

The agent needs to be created only once. The id of the created agent must be set in the '.env' config of the Bot App.

Using the Agent in the Bot App

Retrieve the agent created in the previous step, based on the agent id.

        # Create Azure OpenAI client
        self.project_client = AIProjectClient.from_connection_string(
            credential=DefaultAzureCredential(),
            conn_str=self.config.az_agentic_ai_service_connection_string,
        )

        
        # retrieve the agent already created
        self.agent = self.project_client.agents.get_agent(DefaultConfig.az_assistant_id)
        print("retrieved agent with id ", self.agent.id)

Create a thread for the User Session, which the Bot App persists across conversation exchanges with the user. Every message from the user during the session gets added to this thread, and 'run' called to execute the user request.

           l_thread = conversation_data.thread

            if l_thread is None:
                # Create a thread
                conversation_data.thread = self.project_client.agents.create_thread()
                l_thread = conversation_data.thread
            # Create message to thread
            message = self.project_client.agents.create_message(
                thread_id=l_thread.id, role="user", content=turn_context.activity.text
            )

            run = self.project_client.agents.create_run(
                thread_id=l_thread.id, assistant_id=self.agent.id
            )
            print(f"Created thread run, ID: {run.id}")

            while run.status in ["queued", "in_progress", "requires_action"]:
                time.sleep(1)
                run = self.project_client.agents.get_run(
                    thread_id=l_thread.id, run_id=run.id
                )

                if run.status == "requires_action" and isinstance(
                    run.required_action, SubmitToolOutputsAction
                ):
                    print("Run requires function call to be done..")
                    tool_calls = run.required_action.submit_tool_outputs.tool_calls
                    if not tool_calls:
                        print("No tool calls provided - cancelling run")
                        self.project_client.agents.cancel_run(
                            thread_id=l_thread.id, run_id=run.id
                        )
                        break

                    tool_outputs = []
                    for tool_call in tool_calls:
                        if isinstance(tool_call, RequiredFunctionToolCall):
                            try:
                                print(f"Executing tool call: {tool_call}")
                                output = functions.execute(tool_call)
                                tool_outputs.append(
                                    ToolOutput(
                                        tool_call_id=tool_call.id,
                                        output=output,
                                    )
                                )
                            except Exception as e:
                                print(f"Error executing tool_call {tool_call.id}: {e}")

                    print(f"Tool outputs: {tool_outputs}")
                    if tool_outputs:
                        self.project_client.agents.submit_tool_outputs_to_run(
                            thread_id=l_thread.id,
                            run_id=run.id,
                            tool_outputs=tool_outputs,
                        )

                    print(f"Current run status: {run.status}")

Note that there is no code to be written to make the REST API calls. The OpenApiTool calls the appropriate API automatically during the thread run. Only in the scenario where the delivery order needs to be created, the execution flows into the function calling block, where the identified function is manually executed.

The source code ofthe Bot App discussed in this post is available here

See a demo

Watch a demo of the sample app, below.

Additional Resources

For more details, check out:

• Azure AI Agent Service documentation.
• Microsoft Bot Framework Documentation
• Azure Logic Apps Documentation