Agentic P2P Automation: Harnessing the Power of OpenAI's Responses API

The Procure-to-Pay (P2P) process is traditionally error-prone and labor-intensive, requiring someone to manually open each purchase invoice, look up contract details in a separate system, and painstakingly compare the two to identify anomalies—a task prone to oversight and inconsistency.

About the sample Application

The 'Agentic' characteristics demonstrated here using the Responses API are:

• The client application makes a single call to the Responses API that internally handles all the actions autonomously, processes the information and returns the response. In other words, the client application does not have to perform those actions itself.
• These actions that the Responses API uses, are Hosted tools like (file search, vision-based reasoning).
• Function calling is used to invoke custom action not available in the Hosted tools (i.e. calling Azure Logic App in this case). The Responses API delegates control to the client application that executes the identified Function, hands over the response to the Responses API to complete the rest of the steps in the business process
• Handling of state across all the tool calls and orchestrating them in the right sequence are all handled by the Responses API.
• It autonomously takes the output from each Tool call and uses it to prepare the request for the next one.
• There is no Workflow logic implemented in the code to perform these steps. It is all done through natural language instructions passed when calling the Responses API, and through the Tool actions.

The P2P Anomaly Detection system follows this workflow:

1. Processes purchase invoice images using computer vision capabilities of gpt-4o
2. Extracts critical information like Contract ID, Supplier ID, and line items from it
3. Retrieves corresponding contract details from an external system via Azure Logic App, through Function Calling capabilities in Responses API
4. Performs a vector Search for the business rules in the OpenAI vector store, for detection of anomalies in Procure to Pay processes
5. Applies the Business rules on the Invoice details and validates them against the details in the Contract data, using gpt-4o for reasoning
6. Generates a detailed report of violations and anomalies using gpt-4o

Architecture of the Solution

Code Walkthrough

1. Tools

The Agent (i.e. the application) uses the configuration for File search, and for the Function Call to invoke the Azure Logic App.

# These are the tools that will be used by the Responses API.
tools_list =  [
        {
            "type": "file_search",
            "vector_store_ids": [config.vector_store_id],
            "max_num_results": 20,
        },
        {
            "type": "function",
            "name": "retrieve_contract",
            "description": "fetch contract details for the given contract_id and supplier_id",
            "parameters": {
                "type": "object",
                "properties": {
                    "contract_id": {
                        "type": "string",
                        "description": "The contract id registered for the Supplier in the System",
                    },
                    "supplier_id": {
                        "type": "string",
                        "description": "The Supplier ID registered in the System",
                    },
                },
                "required": ["contract_id", "supplier_id"],
            },
        },
    ]

2. Instructions to the Agent

Unlike Chat Completions End points that use System Prompts, the Responses API uses Instructions. This contains the prompt that describes how the Agent should go about implementing the use case in its entirety.

instructions="""
This is a Procure to Pay process. You will be provided with the Purchase Invoice image as input.
Note that Step 3 can be performed only after Step 1 and Step 2 are completed.
Step 1: As a first step, you will extract the Contract ID and Supplier ID from the Invoice and also all the line items from the Invoice in the form of a table.
Step 2: You will then use the function tool to call the Logic app with the Contract ID and Supplier ID to get the contract details.
Step 3: You will then use the file search tool to retrieve the business rules applicable to detection of anomalies in the Procure to Pay process.
Step 4: Then, apply the retrieved business rules to match the invoice line items with the contract details fetched from the system, and detect anomalies if any.
Provide the list of anomalies detected in the Invoice, and the business rules that were violated.
"""

3. User input to Responses API

Load the Invoice image as an encoded base64 string, and add that to user input payload.

For simplicity the user input is passed as 'user_prompt'  as a string literal in the code, just for demonstration purposes.

user_prompt = """
here are the Purchase Invoice image(s) as input. Detect anomalies in the procure to pay process and give me a detailed report
"""

# read the Purchase Invoice image(s) to be sent as input to the model
image_paths = ["data_files/Invoice-002.png"]

def encode_image_to_base64(image_path):
    with open(image_path, "rb") as image_file:
        return base64.b64encode(image_file.read()).decode("utf-8")

# Encode images
base64_images = [encode_image_to_base64(image_path) for image_path in image_paths]

input_messages = [
    {
        "role": "user",
        "content": [
            {"type": "input_text", "text": user_prompt},
            *[
                {
                    "type": "input_image",
                    "image_url": f"data:image/jpeg;base64,{base64_image}",
                    "detail": "high",
                }
                for base64_image in base64_images
            ],
        ],
    }
]

4. Invoking the Responses API

The single call below performs all the different steps required to complete the anomaly detection end to end. Note that all the actions like Image based reasoning over the Invoice, vector search to retrieve the Business rules, reasoning over every tool call output and preparing the input for the next tool call, all happens directly within the API, in the cloud.

# The following code is to call the Responses API with the input messages and tools
response = client.responses.create(
    model=config.model,
    instructions=instructions,
    input=input_messages,
    tools=tools_list,
    tool_choice="auto",
    parallel_tool_calls=False,
)
tool_call = response.output[0]

There is only one step, related to Function call, that needs to run the custom function locally in the Application. The Responses API response indicates that a Function Call invocation has to happen before it can complete the process. It provides the Function name and the arguments required to make that call.

We then make that function call, locally in the application, to Azure Logic Apps. We get the response back from the Function call, and that that to the payload of input message to the Responses API. It then completes the rest of the steps in the workflow.

# We know this needs a function call, that needs to be executed from here in the application code. # Lets get hold of the function name and arguments from the Responses API response. function_response = None function_to_call = None function_name = None # When a function call is entailed, Responses API gives us control so that we can make the call from our application. # Note that this is because function call is to run our own custom code, it is not a hosted tool that Responses API can directly access and run. if response.output[0].type == "function_call": function_name = response.output[0].name function_to_call = available_functions[function_name] function_args = json.loads(response.output[0].arguments) # Lets call the Logic app with the function arguments to get the contract details. function_response = function_to_call(**function_args) # append the response message to the input messages, and proceed with the next call to the Responses API. input_messages.append(tool_call) # append model's function call message input_messages.append({ # append result message "type": "function_call_output", "call_id": tool_call.call_id, "output": str(function_response) }) # This is the final call to the Responses API with the input messages and tools response_2 = client.responses.create( model=config.model, instructions=instructions, input=input_messages, tools=tools_list, ) print(response_2.output_text)

5. Function Call

Here is the code snippet that invokes the Azure Logic App and returns the relevant contract details from the Azure SQL Database.

if response.output[0].type == "function_call":
    function_name = response.output[0].name
    function_to_call = available_functions[function_name]
    function_args = json.loads(response.output[0].arguments)
    # Lets call the Logic app with the function arguments to get the contract details.
    function_response = function_to_call(**function_args)

# append the response message to the input messages, and proceed with the next call to the Responses API.
input_messages.append(tool_call)  # append model's function call message
input_messages.append({           # append result message
    "type": "function_call_output",
    "call_id": tool_call.call_id,
    "output": str(function_response)
})

# This is the final call to the Responses API with the input messages and tools
response_2 = client.responses.create(
    model=config.model,
    instructions=instructions,
    input=input_messages,
    tools=tools_list,
)
print(response_2.output_text)

Code Run outcome

Here is the output from the run of the Responses API call

## ✅ Contract Line Items (Raw JSON)

```json
[
  {
    "ContractID": "CON000002",
    "LineID": "LINE000003",
    "SupplierID": "SUP0008",
    "ContractDate": "2022-10-19T00:00:00",
    "ExpirationDate": "2023-01-07T00:00:00",
    "TotalAmount": 66543.390625,
    "Currency": "USD",
    "Status": "Expired",
    "ItemID": "ITEM0040",
    "Quantity": 78,
    "UnitPrice": 136.75,
    "TotalPrice": 10666.5,
    "DeliveryDate": "2023-01-01T00:00:00",
    "ItemDescription": "Description for ITEM0040"
  },
  {
    "ContractID": "CON000002",
    "LineID": "LINE000004",
    "SupplierID": "SUP0008",
    "ContractDate": "2022-10-19T00:00:00",
    "ExpirationDate": "2023-01-07T00:00:00",
    "TotalAmount": 66543.390625,
    "Currency": "USD",
    "Status": "Expired",
    "ItemID": "ITEM0082",
    "Quantity": 57,
    "UnitPrice": 479.8699951171875,
    "TotalPrice": 27352.58984375,
    "DeliveryDate": "2022-11-26T00:00:00",
    "ItemDescription": "Description for ITEM0082"
  },
  {
    "ContractID": "CON000002",
    "LineID": "LINE000005",
    "SupplierID": "SUP0008",
    "ContractDate": "2022-10-19T00:00:00",
    "ExpirationDate": "2023-01-07T00:00:00",
    "TotalAmount": 66543.390625,
    "Currency": "USD",
    "Status": "Expired",
    "ItemID": "ITEM0011",
    "Quantity": 21,
    "UnitPrice": 398.0899963378906,
    "TotalPrice": 8359.8896484375,
    "DeliveryDate": "2022-11-29T00:00:00",
    "ItemDescription": "Description for ITEM0011"
  },
  {
    "ContractID": "CON000002",
    "LineID": "LINE000006",
    "SupplierID": "SUP0008",
    "ContractDate": "2022-10-19T00:00:00",
    "ExpirationDate": "2023-01-07T00:00:00",
    "TotalAmount": 66543.390625,
    "Currency": "USD",
    "Status": "Expired",
    "ItemID": "ITEM0031",
    "Quantity": 47,
    "UnitPrice": 429.0299987792969,
    "TotalPrice": 20164.41015625,
    "DeliveryDate": "2022-12-09T00:00:00",
    "ItemDescription": "Description for ITEM0031"
  }
]

## 🧾 Extracted Details from Invoice

- **Contract ID:** CON000002  
- **Supplier ID:** SUP0008  
- **Total Invoice Value:** $113,130.16 USD  
- **Invoice Date:** 2023-06-15  

---

### 📦 Invoice Line Items

| Item ID   | Quantity | Unit Price | Total Price | Description                  |
|-----------|----------|------------|-------------|------------------------------|
| ITEM0040  | 116      | $136.75    | $15,863.00  | Description for ITEM0040     |
| ITEM0082  | 116      | $554.62    | $64,335.92  | Description for ITEM0082     |
| ITEM0011  | 36       | $398.09    | $14,331.24  | Description for ITEM0011     |
| ITEM0031  | 36       | $475.00    | $17,100.00  | Description for ITEM0031     |
| ITEM9999  | 10       | $150.00    | $1,500.00   | Extra item not in contract   |

---

## 📄 Contract Details Retrieved

### ITEM0040
- Quantity: 78  
- Unit Price: $136.75  
- Total Price: $10,666.50  

### ITEM0082
- Quantity: 57  
- Unit Price: $479.87  
- Total Price: $27,352.59  

### ITEM0011
- Quantity: 21  
- Unit Price: $398.09  
- Total Price: $8,359.89  

### ITEM0031
- Quantity: 47  
- Unit Price: $429.03  
- Total Price: $20,164.41  

- **Contract Expiration:** 2023-01-07 (Status: Expired)

---

## ❗ Anomalies Detected

### 🔴 Contract Expiry
- Invoice dated **2023-06-15** refers to an **expired contract** (expired on **2023-01-07**).

### 🔴 Quantity Exceeds Contract
- **ITEM0040:** 116 > 78  
- **ITEM0082:** 116 > 57  
- **ITEM0011:** 36 > 21  
- **ITEM0031:** 36 ≤ 47 (✅ within limit)

### 🔴 Price Discrepancy
- **ITEM0082:** Invoiced @ $554.62 vs Contract @ $479.87  
- **ITEM0031:** Invoiced @ $475.00 vs Contract @ $429.03  

### 🔴 Extra Item
- **ITEM9999** not found in contract records.

---

## 🧩 Conclusion

Multiple business rule violations were found:

- ❌ Contract expired  
- ❌ Quantity overrun  
- ❌ Price discrepancies  
- ❌ Unauthorized items  

> **Recommended:** Detailed investigation and corrective action.

References:

• The source code of the application used in this sample - here
• Read about the Responses API here
• Read about the availability of this API on Azure here

View a video of the demonstration of this sample application below.