Building custom AI Speech models with Phi-3 and Synthetic data
Introduction In today’s landscape, speech recognition technologies play a critical role across various industries—improving customer experiences, streamlining operations, and enabling more intuitive interactions. With Azure AI Speech, developers and organizations can easily harness powerful, fully managed speech functionalities without requiring deep expertise in data science or speech engineering. Core capabilities include: Speech to Text (STT) Text to Speech (TTS) Speech Translation Custom Neural Voice Speaker Recognition Azure AI Speech supports over 100 languages and dialects, making it ideal for global applications. Yet, for certain highly specialized domains—such as industry-specific terminology, specialized technical jargon, or brand-specific nomenclature—off-the-shelf recognition models may fall short. To achieve the best possible performance, you’ll likely need to fine-tune a custom speech recognition model. This fine-tuning process typically requires a considerable amount of high-quality, domain-specific audio data, which can be difficult to acquire. The Data Challenge: When training datasets lack sufficient diversity or volume—especially in niche domains or underrepresented speech patterns—model performance can degrade significantly. This not only impacts transcription accuracy but also hinders the adoption of speech-based applications. For many developers, sourcing enough domain-relevant audio data is one of the most challenging aspects of building high-accuracy, real-world speech solutions. Addressing Data Scarcity with Synthetic Data A powerful solution to data scarcity is the use of synthetic data: audio files generated artificially using TTS models rather than recorded from live speakers. Synthetic data helps you quickly produce large volumes of domain-specific audio for model training and evaluation. By leveraging Microsoft’s Phi-3.5 model and Azure’s pre-trained TTS engines, you can generate target-language, domain-focused synthetic utterances at scale—no professional recording studio or voice actors needed. What is Synthetic Data? Synthetic data is artificial data that replicates patterns found in real-world data without exposing sensitive details. It’s especially beneficial when real data is limited, protected, or expensive to gather. Use cases include: Privacy Compliance: Train models without handling personal or sensitive data. Filling Data Gaps: Quickly create samples for rare scenarios (e.g., specialized medical terms, unusual accents) to improve model accuracy. Balancing Datasets: Add more samples to underrepresented classes, enhancing fairness and performance. Scenario Testing: Simulate rare or costly conditions (e.g., edge cases in autonomous driving) for more robust models. By incorporating synthetic data, you can fine-tune custom STT(Speech to Text) models even when your access to real-world domain recordings is limited. Synthetic data allows models to learn from a broader range of domain-specific utterances, improving accuracy and robustness. Overview of the Process This blog post provides a step-by-step guide—supported by code samples—to quickly generate domain-specific synthetic data with Phi-3.5 and Azure AI Speech TTS, then use that data to fine-tune and evaluate a custom speech-to-text model. We will cover steps 1–4 of the high-level architecture: End-to-End Custom Speech-to-Text Model Fine-Tuning Process Custom Speech with Synthetic data Hands-on Labs: GitHub Repository Step 0: Environment Setup First, configure a .env file based on the provided sample.env template to suit your environment. You’ll need to: Deploy the Phi-3.5 model as a serverless endpoint on Azure AI Foundry. Provision Azure AI Speech and Azure Storage account. Below is a sample configuration focusing on creating a custom Italian model: # this is a sample for keys used in this code repo. # Please rename it to .env before you can use it # Azure Phi3.5 AZURE_PHI3.5_ENDPOINT=https://aoai-services1.services.ai.azure.com/models AZURE_PHI3.5_API_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx AZURE_PHI3.5_DEPLOYMENT_NAME=Phi-3.5-MoE-instruct #Azure AI Speech AZURE_AI_SPEECH_REGION=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx AZURE_AI_SPEECH_API_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx # https://learn.microsoft.com/en-us/azure/ai-services/speech-service/language-support?tabs=stt CUSTOM_SPEECH_LANG=Italian CUSTOM_SPEECH_LOCALE=it-IT # https://speech.microsoft.com/portal?projecttype=voicegallery TTS_FOR_TRAIN=it-IT-BenignoNeural,it-IT-CalimeroNeural,it-IT-CataldoNeural,it-IT-FabiolaNeural,it-IT-FiammaNeural TTS_FOR_EVAL=it-IT-IsabellaMultilingualNeural #Azure Account Storage AZURE_STORAGE_ACCOUNT_NAME=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx AZURE_STORAGE_ACCOUNT_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx AZURE_STORAGE_CONTAINER_NAME=stt-container Key Settings Explained: AZURE_PHI3.5_ENDPOINT / AZURE_PHI3.5_API_KEY / AZURE_PHI3.5_DEPLOYMENT_NAME: Access credentials and the deployment name for the Phi-3.5 model. AZURE_AI_SPEECH_REGION: The Azure region hosting your Speech resources. CUSTOM_SPEECH_LANG / CUSTOM_SPEECH_LOCALE: Specify the language and locale for the custom model. TTS_FOR_TRAIN / TTS_FOR_EVAL: Comma-separated Voice Names (from the Voice Gallery) for generating synthetic speech for training and evaluation. AZURE_STORAGE_ACCOUNT_NAME / KEY / CONTAINER_NAME: Configurations for your Azure Storage account, where training/evaluation data will be stored. > Voice Gallery Step 1: Generating Domain-Specific Text Utterances with Phi-3.5 Use the Phi-3.5 model to generate custom textual utterances in your target language and English. These utterances serve as a seed for synthetic speech creation. By adjusting your prompts, you can produce text tailored to your domain (such as call center Q&A for a tech brand). Code snippet (illustrative): topic = f""" Call center QnA related expected spoken utterances for {CUSTOM_SPEECH_LANG} and English languages. """ question = f""" create 10 lines of jsonl of the topic in {CUSTOM_SPEECH_LANG} and english. jsonl format is required. use 'no' as number and '{CUSTOM_SPEECH_LOCALE}', 'en-US' keys for the languages. only include the lines as the result. Do not include ```jsonl, ``` and blank line in the result. """ response = client.complete( messages=[ SystemMessage(content=""" Generate plain text sentences of #topic# related text to improve the recognition of domain-specific words and phrases. Domain-specific words can be uncommon or made-up words, but their pronunciation must be straightforward to be recognized. Use text data that's close to the expected spoken utterances. The nummber of utterances per line should be 1. """), UserMessage(content=f""" #topic#: {topic} Question: {question} """), ], ... ) content = response.choices[0].message.content print(content) # Prints the generated JSONL with no, locale, and content keys Sample Output (Contoso Electronics in Italian): {"no":1,"it-IT":"Come posso risolvere un problema con il mio televisore Contoso?","en-US":"How can I fix an issue with my Contoso TV?"} {"no":2,"it-IT":"Qual è la garanzia per il mio smartphone Contoso?","en-US":"What is the warranty for my Contoso smartphone?"} {"no":3,"it-IT":"Ho bisogno di assistenza per il mio tablet Contoso, chi posso contattare?","en-US":"I need help with my Contoso tablet, who can I contact?"} {"no":4,"it-IT":"Il mio laptop Contoso non si accende, cosa posso fare?","en-US":"My Contoso laptop won't turn on, what can I do?"} {"no":5,"it-IT":"Posso acquistare accessori per il mio smartwatch Contoso?","en-US":"Can I buy accessories for my Contoso smartwatch?"} {"no":6,"it-IT":"Ho perso la password del mio router Contoso, come posso recuperarla?","en-US":"I forgot my Contoso router password, how can I recover it?"} {"no":7,"it-IT":"Il mio telecomando Contoso non funziona, come posso sostituirlo?","en-US":"My Contoso remote control isn't working, how can I replace it?"} {"no":8,"it-IT":"Ho bisogno di assistenza per il mio altoparlante Contoso, chi posso contattare?","en-US":"I need help with my Contoso speaker, who can I contact?"} {"no":9,"it-IT":"Il mio smartphone Contoso si surriscalda, cosa posso fare?","en-US":"My Contoso smartphone is overheating, what can I do?"} {"no":10,"it-IT":"Posso acquistare una copia di backup del mio smartwatch Contoso?","en-US":"Can I buy a backup copy of my Contoso smartwatch?"} These generated lines give you a domain-oriented textual dataset, ready to be converted into synthetic audio. Step 2: Creating the Synthetic Audio Dataset Using the generated utterances from Step 1, you can now produce synthetic speech WAV files using Azure AI Speech’s TTS service. This bypasses the need for real recordings and allows quick generation of numerous training samples. Core Function: def get_audio_file_by_speech_synthesis(text, file_path, lang, default_tts_voice): ssml = f"""<speak version='1.0' xmlns="https://www.w3.org/2001/10/synthesis" xml:lang='{lang}'> <voice name='{default_tts_voice}'> {html.escape(text)} </voice> </speak>""" speech_sythesis_result = speech_synthesizer.speak_ssml_async(ssml).get() stream = speechsdk.AudioDataStream(speech_sythesis_result) stream.save_to_wav_file(file_path) Execution: For each generated text line, the code produces multiple WAV files (one per specified TTS voice). It also creates a manifest.txt for reference and a zip file containing all the training data. Note: If DELETE_OLD_DATA = True, the training_dataset folder resets each run. If you’re mixing synthetic data with real recorded data, set DELETE_OLD_DATA = False to retain previously curated samples. Code snippet (illustrative): import zipfile import shutil DELETE_OLD_DATA = True train_dataset_dir = "train_dataset" if not os.path.exists(train_dataset_dir): os.makedirs(train_dataset_dir) if(DELETE_OLD_DATA): for file in os.listdir(train_dataset_dir): os.remove(os.path.join(train_dataset_dir, file)) timestamp = datetime.datetime.now().strftime("%Y%m%d%H%M%S") zip_filename = f'train_{lang}_{timestamp}.zip' with zipfile.ZipFile(zip_filename, 'w') as zipf: for file in files: zipf.write(os.path.join(output_dir, file), file) print(f"Created zip file: {zip_filename}") shutil.move(zip_filename, os.path.join(train_dataset_dir, zip_filename)) print(f"Moved zip file to: {os.path.join(train_dataset_dir, zip_filename)}") train_dataset_path = {os.path.join(train_dataset_dir, zip_filename)} %store train_dataset_path You’ll also similarly create evaluation data using a different TTS voice than used for training to ensure a meaningful evaluation scenario. Example Snippet to create the synthetic evaluation data: import datetime print(TTS_FOR_EVAL) languages = [CUSTOM_SPEECH_LOCALE] eval_output_dir = "synthetic_eval_data" DELETE_OLD_DATA = True if not os.path.exists(eval_output_dir): os.makedirs(eval_output_dir) if(DELETE_OLD_DATA): for file in os.listdir(eval_output_dir): os.remove(os.path.join(eval_output_dir, file)) eval_tts_voices = TTS_FOR_EVAL.split(',') for tts_voice in eval_tts_voices: with open(synthetic_text_file, 'r', encoding='utf-8') as f: for line in f: try: expression = json.loads(line) no = expression['no'] for lang in languages: text = expression[lang] timestamp = datetime.datetime.now().strftime("%Y%m%d%H%M%S") file_name = f"{no}_{lang}_{timestamp}.wav" get_audio_file_by_speech_synthesis(text, os.path.join(eval_output_dir,file_name), lang, tts_voice) with open(f'{eval_output_dir}/manifest.txt', 'a', encoding='utf-8') as manifest_file: manifest_file.write(f"{file_name}\t{text}\n") except json.JSONDecodeError as e: print(f"Error decoding JSON on line: {line}") print(e) Step 3: Creating and Training a Custom Speech Model To fine-tune and evaluate your custom model, you’ll interact with Azure’s Speech-to-Text APIs: Upload your dataset (the zip file created in Step 2) to your Azure Storage container. Register your dataset as a Custom Speech dataset. Create a Custom Speech model using that dataset. Create evaluations using that custom model with asynchronous calls until it’s completed. You can also use UI-based approaches to customize a speech model with fine-tuning in the Azure AI Foundry portal, but in this hands-on, we'll use the Azure Speech-to-Text REST APIs to iterate entire processes. Key APIs & References: Azure Speech-to-Text REST APIs (v3.2) The provided common.py in the hands-on repo abstracts API calls for convenience. Example Snippet to create training dataset: uploaded_files, url = upload_dataset_to_storage(data_folder, container_name, account_name, account_key) kind="Acoustic" display_name = "acoustic dataset(zip) for training" description = f"[training] Dataset for fine-tuning the {CUSTOM_SPEECH_LANG} base model" zip_dataset_dict = {} for display_name in uploaded_files: zip_dataset_dict[display_name] = create_dataset(base_url, headers, project_id, url[display_name], kind, display_name, description, CUSTOM_SPEECH_LOCALE) You can monitor training progress using monitor_training_status function which polls the model’s status and updates you once training completes Core Function: def monitor_training_status(custom_model_id): with tqdm(total=3, desc="Running Status", unit="step") as pbar: status = get_custom_model_status(base_url, headers, custom_model_id) if status == "NotStarted": pbar.update(1) while status != "Succeeded" and status != "Failed": if status == "Running" and pbar.n < 2: pbar.update(1) print(f"Current Status: {status}") time.sleep(10) status = get_custom_model_status(base_url, headers, custom_model_id) while(pbar.n < 3): pbar.update(1) print("Training Completed") Step 4: Evaluate Trained Custom Speech After training, create an evaluation job using your synthetic evaluation dataset. With the custom model now trained, compare its performance (measured by Word Error Rate, WER) against the base model’s WER. Key Steps: Use create_evaluation function to evaluate the custom model against your test set. Compare evaluation metrics between base and custom models. Check WER to quantify accuracy improvements. After evaluation, you can view the evaluation results of the base model and the fine-tuning model based on the evaluation dataset created in the 1_text_data_generation.ipynb notebook in either Speech Studio or the AI Foundry Fine-Tuning section, depending on the resource location you specified in the configuration file. Example Snippet to create evaluation: description = f"[{CUSTOM_SPEECH_LOCALE}] Evaluation of the {CUSTOM_SPEECH_LANG} base and custom model" evaluation_ids={} for display_name in uploaded_files: evaluation_ids[display_name] = create_evaluation(base_url, headers, project_id, dataset_ids[display_name], base_model_id, custom_model_with_acoustic_id, f'vi_eval_base_vs_custom_{display_name}', description, CUSTOM_SPEECH_LOCALE) Also, you can see a simple Word Error Rate (WER) number in the code below, which you can utilize in 4_evaluate_custom_model.ipynb. Example Snippet to create WER dateframe: # Collect WER results for each dataset wer_results = [] eval_title = "Evaluation Results for base model and custom model: " for display_name in uploaded_files: eval_info = get_evaluation_results(base_url, headers, evaluation_ids[display_name]) eval_title = eval_title + display_name + " " wer_results.append({ 'Dataset': display_name, 'WER_base_model': eval_info['properties']['wordErrorRate1'], 'WER_custom_model': eval_info['properties']['wordErrorRate2'], }) # Create a DataFrame to display the results print(eval_info) wer_df = pd.DataFrame(wer_results) print(eval_title) print(wer_df) About WER: WER is computed as (Insertions + Deletions + Substitutions) / Total Words. A lower WER signifies better accuracy. Synthetic data can help reduce WER by introducing more domain-specific terms during training. You'll also similarly create a WER result markdown file using the md_table_scoring_result method below. Core Function: # Create a markdown file for table scoring results md_table_scoring_result(base_url, headers, evaluation_ids, uploaded_files) Implementation Considerations The provided code and instructions serve as a baseline for automating the creation of synthetic data and fine-tuning Custom Speech models. The WER numbers you get from model evaluation will also vary depending on the actual domain. Real-world scenarios may require adjustments, such as incorporating real data or customizing the training pipeline for specific domain needs. Feel free to extend or modify this baseline to better match your use case and improve model performance. Conclusion By combining Microsoft’s Phi-3.5 model with Azure AI Speech TTS capabilities, you can overcome data scarcity and accelerate the fine-tuning of domain-specific speech-to-text models. Synthetic data generation makes it possible to: Rapidly produce large volumes of specialized training and evaluation data. Substantially reduce the time and cost associated with recording real audio. Improve speech recognition accuracy for niche domains by augmenting your dataset with diverse synthetic samples. As you continue exploring Azure’s AI and speech services, you’ll find more opportunities to leverage generative AI and synthetic data to build powerful, domain-adapted speech solutions—without the overhead of large-scale data collection efforts. 🙂 Reference Azure AI Speech Overview Microsoft Phi-3 Cookbook Text to Speech Overview Speech to Text Overview Custom Speech Overview Customize a speech model with fine-tuning in the Azure AI Foundry Scaling Speech-Text Pre-Training with Synthetic Interleaved Data (arXiv) Training TTS Systems from Synthetic Data: A Practical Approach for Accent Transfer (arXiv) Generating Data with TTS and LLMs for Conversational Speech Recognition (arXiv)Fine-Tuning Together: How Our Developer Community Is Shaping the Future of Custom AI
In a world where foundation models are increasingly accessible, the real magic happens when developers make them their own. Fine-tuning is no longer a niche capability, it’s becoming a core skill for developers who want to build AI that’s faster, smarter, and more aligned with their users scaling expert knowledge across their organizations. Over the past few months, we’ve seen something remarkable: a growing community of builders, tinkerers, and innovators coming together to push the boundaries of what fine-tuning can do making a powerful difference for everyday organizations. A Community Making a Big Impact: Customer Stories At Build 2025, we saw firsthand how much the landscape has shifted. Just a year ago, many teams were still relying solely on prompt engineering or retrieval-augmented generation (RAG). Today, nearly half of developers say they’re actively exploring fine-tuning. Why? Because it gives them something no off-the-shelf model can: the ability to embed their domain knowledge, tone, and logic directly into the model itself. In our breakout session led by Product Leaders Alicia and Omkar, they dove into fine-tuning and distillation with Azure AI Foundry. We heard from the Oracle Health team and how they are restoring joy in providing patient care by relieving administration burden. The Oracle Health team used fine-tuned GPT-4o-mini models to power a clinical AI agent that responds in under 800 milliseconds—fast enough to keep up with real-time healthcare workflows. Earlier this year, we saw DraftWise use reinforcement fine-tuning on o-series reasoning models within Azure AI Foundry to tailor model behavior for legal-specific tasks. This approach allowed them to sharpen responses based on proprietary legal data, improving the quality of contract drafting and review. The fine-tuned models contributed to a 30% improvement in search result quality enabling faster, more accurate legal drafting and review at scale. And we watched CoStar Group deliver a low-latency, voice-driven home search experience that scales to over 100 million monthly users, while reducing token usage, improving cost-efficiency, and accelerating time to deployment by combining GPT-4o Realtime API audio models and Mistral 3B on Azure AI Foundry. These aren’t just technical wins - they’re community wins. They show what’s possible when developers have the right tools and support to build AI that truly fits their needs. What’s New in Azure AI Foundry: Tools That Empower, Not Overwhelm The last 3 months have brought a wave of updates designed to make fine-tuning more accessible, more affordable, and more powerful—especially for developers who are just getting started. One of the most exciting additions is Reinforcement Fine-Tuning (RFT), now available in public preview with the o4-mini model. Unlike traditional supervised fine-tuning, RFT lets you teach models how to reason through complex tasks using reward signals. It’s ideal for domains where logic and nuance matter—like law, finance, or healthcare—and it’s already helping teams like DraftWise build smarter, more adaptive systems. Watch this o4-mini demo to learn more. We also introduced Global Training, which lets you fine-tune models from any of 24 Azure OpenAI regions. This means no more guessing which region supports which model, significantly lowering the barrier to entry for model customization. One of the most common questions we hear from developers is: “How do I know if my fine-tuned model is actually better?” The new Evaluation API is our answer. This API lets you programmatically evaluate model outputs using model-based graders, custom rubrics, and structured scoring—all from code And for developers who want to experiment without breaking the bank, we launched the Developer Tier. It’s a new way to deploy fine-tuned models for free (for 24 hours), paying only for tokens at the same rate as base models. It’s ideal for A/B testing, distillation experiments, or just kicking the tires on a new idea. Learning Together: From Distillation to Deployment One of the most powerful trends we’ve seen is the rise of distillation. Developers are using larger “teacher” models like GPT-4o to generate high-quality outputs, then fine-tuning smaller “student” models like GPT-4.1-mini or nano to replicate that performance at a fraction of the cost. This is now supported end-to-end in Azure AI Foundry. You can generate completions, store them automatically, fine-tune your student model, and evaluate it using model-based graders—all in one place. And the results speak for themselves. In one demo at Build, we saw a distilled 4o model go from 35% to 90% accuracy just by learning from the outputs of a larger o3 model. Let’s Keep Building: Join Us for Model Mondays | Mondays at 10:30 AM PT We’re excited about what’s next. More models. More techniques. More impact from developers like you. If you’re already fine-tuning, we’d love to hear what you’re working on. And if you’re just getting started, we’re here to help with Model Mondays. Model Mondays is your weekly dose of AI model magic: an hour of livestreamed demos, interactive developer-friendly deep dives, and just enough chaos to keep Mondays interesting. Watch the latest fine-tuning & distillation episode with Dave Voutila, Microsoft PM Checkout these Resources 🧠 Get Started with fine-tuning with Azure AI Foundry on Microsoft Learn Docs ▶️ Watch On-Demand: Fine-tuning and distillation with Azure AI Foundry 👩💻 Fine-tune GPT-4o-mini model with this tutorial 👋 Continue the conversation on Discord
Automate a multi-step business process, using turnkey MCP, Logic App Integration in AI Foundry
This article walks you through an application for Procure-to-Pay anomaly detection using Azure AI Foundry Agent Service. It analyzes purchase invoice images to detect procurement anomalies and compliance issues. The key capabilities of Azure AI Foundry it showcases are: Agent Service that performs a multi-step business process with only natural language instructions. The Application has little to no business logic in the code. Invocation of a variety of automated tool actions required during the multi steps business process. It showcases the recently announced turnkey integration with MCP Server, Azure Logic Apps. In addition, it uses a) visual reasoning of images using gpt-4o models, b) turnkey vector search, and c) reasoning to apply business rules and perform p2p anomaly detection. The ability of Agent Service in Azure AI Foundry to: most importantly, all the steps in the business process are performed through a single call from the Application to the Agent Service. The Agent orchestrates the steps autonomously, based on the instructions in Natural language. The Client application itself contains no business logic in code. handle application state between different tool calls. It determines what to extract from the output of one tool call to pass as input to the next. orchestrating the business process from start to end, sequencing them to achieve the business end. Architecture of the Solution Shown below is the architecture of the Solution. The Client Application is a python-based console App that calls the P2P Anomaly Detection Agent running in the Azure AI Foundry Agent Service. Solution Components Sr. No Entity Entity Purpose or Description 1 Foundry Agent The autonomous Agent that implements the P2P anomaly detection process end to end. It orchestrates all the steps in the business process through a single API call, handling application state between different tool calls and sequencing operations to achieve the business objective. 2 Visual Reasoning Tool Call Analyzes the input Purchase Invoice Image using GPT-4o's vision capabilities to extract the Purchase Invoice Header, Invoice Lines, Supplier ID, and Contract ID. This extracted information is then used in subsequent calls to the Logic App for contract validation. 3 Azure Logic App Tool Call Called by the Agent via HTTP Trigger with the Supplier ID and Contract ID extracted from the invoice. Returns matching contract data by executing dynamic SQL queries on an Azure SQL Database, providing contract header and line item details for comparison with the invoice. 4 Vector Search Tool Call Retrieves the business rules that apply to P2P anomaly detection in an Enterprise environment. Uses the configured vector store to search through the uploaded business rules document (p2p-rules.txt) to find relevant compliance and validation criteria. 5 Reasoning Applies the retrieved business rules to evaluate the Purchase Invoice against the Contract data. Determines if there are anomalies or compliance issues and generates a detailed verification report with verdicts, comparisons, and recommendations. 6 MCP Server Tool Call Catalogs and stores the generated reports in Azure Blob Storage. The MCP Server implements protocol-specific implementations to connect to the Blob Storage service, performing lifecycle actions like creating containers, listing containers, and uploading blobs. The Agent uses the MCP Server to upload the markdown verification report to a designated container. Some of the key aspects of the Solution are: 1) There is only one call from the Client Application to the Agent in Azure AI Foundry. The rest of the steps are performed by the Agent Service autonomously. The code snippet below shows the client application passing the user provided Invoice image and user instruction to the Agent in Azure AI Foundry. 2) Observe that there is no business logic in the code whatsoever, and there is no code written to perform any of the tool actions either. These are autonomously done by the Agent itself. def process_anomaly_detection(self, user_prompt: str, image_path: str, verbose: bool = True) -> str: """ Process the anomaly detection request with proper error handling and retry logic. Args: user_prompt: User's prompt for anomaly detection image_path: Path to the invoice image Returns: Response from the AI agent Raises: Exception: For various processing errors """ try: # Create message content self._show_progress("Creating message content...", verbose) content_blocks = self.create_message_content(user_prompt, image_path) # Create a new thread for this conversation self._show_progress("Creating conversation thread...", verbose) thread = self.client.threads.create() logger.info(f"Created thread, ID: {thread.id}") # Create the message self._show_progress("Sending message to agent...", verbose) message = self.client.messages.create( thread_id=thread.id, role="user", content=content_blocks ) # Run the agent with proper error handling self._show_progress("Processing with AI agent (this may take a moment)...", verbose) run = self.client.runs.create_and_process( thread_id=thread.id, agent_id=self.agent.id ) if run.status == "failed": error_msg = f"Agent run failed: {run.last_error}" logger.error(error_msg) raise Exception(error_msg) # Retrieve and process messages self._show_progress("Retrieving analysis results...", verbose) messages = self.client.messages.list( thread_id=thread.id, order=ListSortOrder.ASCENDING ) # Extract the agent's response agent_response = "" for message in messages: if message.role == "assistant" and message.text_messages: agent_response = message.text_messages[-1].text.value break if not agent_response: raise Exception("No response received from the agent") self._show_progress("Analysis complete!", verbose) logger.info("Successfully processed anomaly detection request") return agent_response except Exception as e: logger.error(f"Error processing anomaly detection: {e}") raise Note: The steps to configure the Solution is covered in the GitHub Repo link shared later in this article, and hence not discussed here. Agent Instructions As mentioned already, the Client application does not contain any Business logic. The latter is provided as natural language instructions to the Foundry Agent. Show below are the instructions configured on the P2P Agent. This is a Procure to Pay process. You will be provided with the Purchase Invoice image as input. Note the sequence of execution you must adhere to strictly: - Step 2 must be performed only after Step 1 is performed - Step 3 below must be performed only after Step 1 and Step 2 are completed. - Step 5 must be performed only after Step 1, Step 2, Step 3 and Step 4 have been performed successfully. - Step 6 must be performed only after Step 5 is performed Step 1: As a first step, you will extract the Contract ID and Supplier ID from the Purchase Invoice image along with all the line items from the Invoice in the form of a table. Step 2: You will then use the function tool by passing the Contract ID and Supplier ID to retrieve the contract details. Step 3: You will then use the file search tool to retrieve the business rules applicable to detection of anomalies in the Procure to Pay process. Step 4: Then, apply the retrieved business rules to match the invoice line items with the contract details fetched from the system, and detect anomalies if any. Step 5: Prepare a detailed 'p2p verification report' in markdown with the following content: - Verdict: Whether the Purchase Invoice complies with the Contract? - Purchase Invoice details: Invoice Header and Invoice Lines, in Markdown Table Format - Contract Details: Contract Header and Contract Lines, in Markdown Table Format - P2P Business Rules: The Rules that were retrieved using the File Search Tool - Reasoning behind the verdict: Provide a detailed reasoning why you think the Invoice aligns with the Contract yes or no. Use a Markdown Table format to compare each item in the Invoice with the Contract and indicate the basis for your judgement. Use icons/images to embellish the report and make it easy for comprehension by Business users. They must be able to quickly give this a once over and commit the Invoice into the System Step 6: You will use the MCP tool available with you to upload the 'p2p verification report' created in Step 5. - Choose a container with name 'p2p-anomaly-detection-outcomes' to upload to. If the container with this name does not exist, create one. - The name of the Report must start with the Invoice Number, appended with a hyphen and appended with a guid. E.g. Invoice001-001.md - Secure the name of the Report document uploaded to the Blob Storage account through the MCP Tool Step 7: Wait till Step 6 completes, then return the content of the Markdown document that was uploaded to Blob Storage. Output from the Run (.venv) PS C:\Users\sansri\agentic-ai-service-samples\p2p-anomaly-detection-agent> python p2pagent.py 2025-07-04 09:04:22,149 - __main__ - INFO - Successfully initialized Azure AI Agents client 2025-07-04 09:04:22,149 - INFO - Successfully initialized Azure AI Agents client === P2P Anomaly Detection Agent === This tool analyzes invoice images for procurement anomalies. Enter the path to your invoice image: data_files/Invoice-002.png Enter your analysis prompt (or press Enter for default): can you perform the procure to pay anomaly detection based on the instructions you have been provided with and give me a detailed response if this Purchase Invoice Image attached aligns with the Contract? Processing image: C:\Users\sansri\agentic-ai-service-samples\p2p-anomaly-detection-agent\data_files\Invoice-002.png Analyzing with Azure AI Agent... 🔄 Creating message content... 2025-07-04 09:04:46,774 - __main__ - INFO - Successfully converted image to base64: C:\Users\sansri\agentic-ai-service-samples\p2p-anomaly-detection-agent\data_files\Invoice-002.png 2025-07-04 09:04:46,774 - INFO - Successfully converted image to base64: C:\Users\sansri\agentic-ai-service-samples\p2p-anomaly-detection-agent\data_files\Invoice-002.png 2025-07-04 09:04:46,777 - __main__ - INFO - Successfully created message content blocks 2025-07-04 09:04:46,777 - INFO - Successfully created message content blocks 🔄 Creating conversation thread... 2025-07-04 09:04:47,231 - __main__ - INFO - Created thread, ID: thread_Lz8ViyvcnDMG9i9TGkgE7jq6 2025-07-04 09:04:47,231 - INFO - Created thread, ID: thread_Lz8ViyvcnDMG9i9TGkgE7jq6 🔄 Sending message to agent... 🔄 Processing with AI agent (this may take a moment)... 🔄 Retrieving analysis results... 🔄 Analysis complete! 2025-07-04 09:05:55,543 - __main__ - INFO - Successfully processed anomaly detection request 2025-07-04 09:05:55,543 - INFO - Successfully processed anomaly detection request ================================================== ANOMALY DETECTION RESULTS ================================================== ### P2P Verification Report #### **Verdict:** The Purchase Invoice **does not comply** with the Contract. --- #### **Purchase Invoice Details** | **Attribute** | **Value** | |-----------------------|------------------------------------------------------------------------------| | **Invoice Number** | INV001002 | | **Contract Reference**| CON000002 | | **Supplier ID** | SUP0008 | | **Total Invoice Value** | 113130.16 USD | | **Invoice Date** | 2023-06-15 | --- #### **Invoice Line Items** | **Item ID** | **Quantity** | **Unit Price (USD)** | **Total Price** | **Description** | |-------------|--------------|-----------------------|-----------------|---------------------------------------| | ITEM0040 | 116 | $136.75 | $15863.00 | Description for ITEM0040 | | ITEM0082 | 116 | $554.62 | $64335.92 | Description for ITEM0082 | | ITEM0011 | 36 | $398.09 | $14331.24 | Description for ITEM0011 | | ITEM0031 | 36 | $475.00 | $17100.00 | Description for ITEM0031 | | ITEM9999 | 10 | $150.00 | $1500.00 | Extra item not in contract | --- #### **Contract Details** | **Attribute** | **Value** | |--------------------------------|------------------------------------------------------------------------------| | **Contract ID** | CON000002 | | **Supplier ID** | SUP0008 | | **Contract Date** | 2022-10-19 | | **Expiration Date** | 2023-01-07 | | **Contract Total Amount** | 66543.39 USD | | **Status** | Expired | | **Currency** | USD | | **Item ID** | **Quantity** | **Unit Price (USD)** | **Total Price** | **Delivery Date** | **Description** | |-------------|--------------|-----------------------|-----------------|-------------------|----------------------------| | ITEM0040 | 78 | $136.75 | $10666.50 | 2023-01-01 | Description for ITEM0040 | | ITEM0082 | 57 | $479.87 | $27352.58 | 2022-11-26 | Description for ITEM0082 | | ITEM0011 | 21 | $398.09 | $8359.89 | 2022-11-29 | Description for ITEM0011 | | ITEM0031 | 47 | $429.03 | $20164.41 | 2022-12-09 | Description for ITEM0031 | --- #### **P2P Business Rules** - Invoice date in Purchase Invoice must be within the Contract term【6:0†source】. - The Contract must be valid; ensure it is not expired【6:0†source】. - Invoice Total Value should stay within the Contract value【6:0†source】. - Items in the Invoice must be strictly from the Contract item list, with correct quantities, descriptions, and unit prices【6:0†source】. - Minor rounding differences in amounts should be ignored【6:0†source】. --- |------------------|-------------------------------------------------|------------------------------------------------------------------------------------------------------------| | ITEM0040 | Quantity: 78, Unit Price: $136.75 | Invoice quantity exceeds Contract quantity. | |------------------|-------------------------------------------------|------------------------------------------------------------------------------------------------------------| | ITEM0040 | Quantity: 78, Unit Price: $136.75 | Invoice quantity exceeds Contract quantity. | | ITEM0082 | Quantity: 57, Unit Price: $479.87 | Invoice unit price differs and quantity exceeds Contract quantity. | | ITEM0011 | Quantity: 21, Unit Price: $398.09 | Invoice quantity exceeds Contract quantity. | | ITEM0031 | Quantity: 47, Unit Price: $429.03 | Invoice quantity exceeds Contract quantity and Contract total is lower than Invoice total. | | ITEM9999 | Not listed in Contract | Extra item not defined in Contract—anomalous. | ### Additional Observations: - **Contract is Expired:** The Contract expired on 2023-01-07. Invoice dated 2023-06-15 falls outside the Contract's validity term. - **Invoice exceeds Contract Value:** Contract value capped at $66543.39 USD, while Invoice totals $113130.16 USD. - **Currency Matches:** Invoice and Contract both indicate prices in USD. --- ### Professional Guidance and Suggested Next Steps: - The anomalies detected prevent this Invoice from fully complying with the Contract terms. - **Recommendation:** Rectify discrepancies (quantities, unit price adjustments, or removal of extra items) and adhere to valid Contract before committing Invoice to the System. --- ### Uploading Report Content Proceeding to upload this Markdown report into Blob Storage. ================================================== MCP Actions As an outcome from the P2P evaluation, a Markdown document is created by the P2P Agent which then gets uploaded to the Azure Blob Storage, using the MCP Tool action. See the screenshot below from the Azure Portal. References Here is the GitHub Repo that contains the code for the solution covered in this article. A Linkedin article related to this topic - hereCreate Stunning AI Videos with Sora on Azure AI Foundry!
Special credit to Rory Preddy for creating the GitHub resource that enable us to learn more about Azure Sora. Reach him out on LinkedIn to say thanks. Introduction Artificial Intelligence (AI) is revolutionizing content creation, and video generation is at the forefront of this transformation. OpenAI's Sora, a groundbreaking text-to-video model, allows creators to generate high-quality videos from simple text prompts. When paired with the powerful infrastructure of Azure AI Foundry, you can harness Sora's capabilities with scalability and efficiency, whether on a local machine or a remote setup. In this blog post, I’ll walk you through the process of generating AI videos using Sora on Azure AI Foundry. We’ll cover the setup for both local and remote environments. Requirements: Azure AI Foundry with sora model access A Linux Machine/VM. Make sure that the machine already has the package below: Java JRE 17 (Recommended) OR later Maven Step Zero – Deploying the Azure Sora model on AI Foundry Navigate to the Azure AI Foundry portal and head to the “Models + Endpoints” section (found on the left side of the Azure AI Foundry portal) > Click on the “Deploy Model” button > “Deploy base model” > Search for Sora > Click on “Confirm”. Give a deployment name and specify the Deployment type > Click “Deploy” to finalize the configuration. You should receive an API endpoint and Key after successful deploying Sora on Azure AI Foundry. Store these in a safe place because we will be using them in the next steps. Step one – Setting up the Sora Video Generator in the local/remote machine. Clone the roryp/sora repository on your machine by running the command below: git clone https://github.com/roryp/sora.git cd sora Then, edit the application.properties file in the src/main/resources/ folder to include your Azure OpenAI Credentials. Change the configuration below: azure.openai.endpoint=https://your-openai-resource.cognitiveservices.azure.com azure.openai.api-key=your_api_key_here If port 8080 is used for another application, and you want to change the port for which the web app will run, change the “server.port” configuration to include the desired port. Allow appropriate permissions to run the “mvnw” script file. chmod +x mvnw Run the application ./mvnw spring-boot:run Open your browser and type in your localhost/remote host IP (format: [host-ip:port]) in the browser search bar. If you are running a remote host, please do not forget to update your firewall/NSG to allow inbound connection to the configured port. You should see the web app to generate video with Sora AI using the API provided on Azure AI Foundry. Now, let’s generate a video with Sora Video Generator. Enter a prompt in the first text field, choose the video pixel resolution, and set the video duration. (Due to technical limitation, Sora can only generate video of a maximum of 20 seconds). Click on the “Generate video” button to proceed. The cost to generate the video should be displayed below the “Generate Video” button, for transparency purposes. You can click on the “View Breakdown” button to learn more about the cost breakdown. The video should be ready to download after a maximum of 5 minutes. You can check the status of the video by clicking on the “Check Status” button on the web app. The web app will inform you once the download is ready and the page should refresh every 10 seconds to fetch real-time update from Sora. Once it is ready, click on the “Download Video” button to download the video. Conclusion Generating AI videos with Sora on Azure AI Foundry is a game-changer for content creators, marketers, and developers. By following the steps outlined in this guide, you can set up your environment, integrate Sora, and start creating stunning AI-generated videos. Experiment with different prompts, optimize your workflow, and let your imagination run wild! Have you tried generating AI videos with Sora or Azure AI Foundry? Share your experiences or questions in the comments below. Don’t forget to subscribe for more AI and cloud computing tutorials!Introducing Azure AI Models: The Practical, Hands-On Course for Real Azure AI Skills
Hello everyone, Today, I’m excited to share something close to my heart. After watching so many developers, including myself—get lost in a maze of scattered docs and endless tutorials, I knew there had to be a better way to learn Azure AI. So, I decided to build a guide from scratch, with a goal to break things down step by step—making it easy for beginners to get started with Azure, My aim was to remove the guesswork and create a resource where anyone could jump in, follow along, and actually see results without feeling overwhelmed. Introducing Azure AI Models Guide. This is a brand new, solo-built, open-source repo aimed at making Azure AI accessible for everyone—whether you’re just getting started or want to build real, production-ready apps using Microsoft’s latest AI tools. The idea is simple: bring all the essentials into one place. You’ll find clear lessons, hands-on projects, and sample code in Python, JavaScript, C#, and REST—all structured so you can learn step by step, at your own pace. I wanted this to be the resource I wish I’d had when I started: straightforward, practical, and friendly to beginners and pros alike. It’s early days for the project, but I’m excited to see it grow. If you’re curious.. Check out the repo at https://github.com/DrHazemAli/Azure-AI-Models Your feedback—and maybe even your contributions—will help shape where it goes next!Deploy Open Web UI on Azure VM via Docker: A Step-by-Step Guide with Custom Domain Setup.
Introductions Open Web UI (often referred to as "Ollama Web UI" in the context of LLM frameworks like Ollama) is an open-source, self-hostable interface designed to simplify interactions with large language models (LLMs) such as GPT-4, Llama 3, Mistral, and others. It provides a user-friendly, browser-based environment for deploying, managing, and experimenting with AI models, making advanced language model capabilities accessible to developers, researchers, and enthusiasts without requiring deep technical expertise. This article will delve into the step-by-step configurations on hosting OpenWeb UI on Azure. Requirements: Azure Portal Account - For students you can claim $USD100 Azure Cloud credits from this URL. Azure Virtual Machine - with a Linux of any distributions installed. Domain Name and Domain Host Caddy Open WebUI Image Step One: Deploy a Linux – Ubuntu VM from Azure Portal Search and Click on “Virtual Machine” on the Azure portal search bar and create a new VM by clicking on the “+ Create” button > “Azure Virtual Machine”. Fill out the form and select any Linux Distribution image – In this demo, we will deploy Open WebUI on Ubuntu Pro 24.04. Click “Review + Create” > “Create” to create the Virtual Machine. Tips: If you plan to locally download and host open source AI models via Open on your VM, you could save time by increasing the size of the OS disk / attach a large disk to the VM. You may also need a higher performance VM specification since large resources are needed to run the Large Language Model (LLM) locally. Once the VM has been successfully created, click on the “Go to resource” button. You will be redirected to the VM’s overview page. Jot down the public IP Address and access the VM using the ssh credentials you have setup just now. Step Two: Deploy the Open WebUI on the VM via Docker Once you are logged into the VM via SSH, run the Docker Command below: docker run -d --name open-webui --network=host --add-host=host.docker.internal:host-gateway -e PORT=8080 -v open-webui:/app/backend/data --restart always ghcr.io/open-webui/open-webui:dev This Docker command will download the Open WebUI Image into the VM and will listen for Open Web UI traffic on port 8080. Wait for a few minutes and the Web UI should be up and running. If you had setup an inbound Network Security Group on Azure to allow port 8080 on your VM from the public Internet, you can access them by typing into the browser: [PUBLIC_IP_ADDRESS]:8080 Step Three: Setup custom domain using Caddy Now, we can setup a reverse proxy to map a custom domain to [PUBLIC_IP_ADDRESS]:8080 using Caddy. The reason why Caddy is useful here is because they provide automated HTTPS solutions – you don’t have to worry about expiring SSL certificate anymore, and it’s free! You must download all Caddy’s dependencies and set up the requirements to install it using this command: sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list sudo apt update && sudo apt install caddy Once Caddy is installed, edit Caddy’s configuration file at: /etc/caddy/Caddyfile , delete everything else in the file and add the following lines: yourdomainname.com { reverse_proxy localhost:8080 } Restart Caddy using this command: sudo systemctl restart caddy Next, create an A record on your DNS Host and point them to the public IP of the server. Step Four: Update the Network Security Group (NSG) To allow public access into the VM via HTTPS, you need to ensure the NSG/Firewall of the VM allow for port 80 and 443. Let’s add these rules into Azure by heading to the VM resources page you created for Open WebUI. Under the “Networking” Section > “Network Settings” > “+ Create port rule” > “Inbound port rule” On the “Destination port ranges” field, type in 443 and Click “Add”. Repeat these steps with port 80. Additionally, to enhance security, you should avoid external users from directly interacting with Open Web UI’s port - port 8080. You should add an inbound deny rule to that port. With that, you should be able to access the Open Web UI from the domain name you setup earlier. Conclusion And just like that, you’ve turned a blank Azure VM into a sleek, secure home for your Open Web UI, no magic required! By combining Docker’s simplicity with Caddy’s “set it and forget it” HTTPS magic, you’ve not only made your app accessible via a custom domain but also locked down security by closing off risky ports and keeping traffic encrypted. Azure’s cloud muscle handles the heavy lifting, while you get to enjoy the perks of a pro setup without the headache. If you are interested in using AI models deployed on Azure AI Foundry on OpenWeb UI via API, kindly read my other article: Step-by-step: Integrate Ollama Web UI to use Azure Open AI API with LiteLLM ProxyIntroduction to OCR Free Vision RAG using Colpali For Complex Documents
Explore the cutting-edge world of document retrieval with "From Pixels to Intelligence: Introduction to OCR Free Vision RAG using ColPali for Complex Documents." This blog post delves into how ColPali revolutionizes the way we interact with documents by leveraging Vision Language Models (VLMs) to enhance Retrieval-Augmented Generation (RAG) processes.AI Automation in Azure Foundry through turnkey MCP Integration and Computer Use Agent Models
The Fashion Trends Discovery Scenario In this walkthrough, we'll explore a sample application that demonstrates the power of combining Computer Use (CUA) models with Playwright browser automation to autonomously compile trend information from the internet, while leveraging MCP integration to intelligently catalog and store insights in Azure Blob Storage. The User Experience A fashion analyst simply provides a query like "latest trends in sustainable fashion" to our command-line interface. What happens next showcases the power of agentic AI—the system requires no further human intervention to: Autonomous Web Navigation: The agent launches Pinterest, intelligently locates search interfaces, and performs targeted queries Intelligent Content Discovery: Systematically identifies and interacts with trend images, navigating to detailed pages Advanced Content Analysis: Applies computer vision to analyze fashion elements, colors, patterns, and design trends Intelligent Compilation: Consolidates findings into comprehensive, professionally formatted markdown reports Contextual Storage: Recognizes the value of preserving insights and autonomously offers cloud storage options Technical capabilities leveraged Behind this seamless experience lies a coordination of AI models: Pinterest Navigation: The CUA model visually understands Pinterest's interface layout, identifying search boxes and navigation elements with pixel-perfect precision Search Results Processing: Rather than relying on traditional DOM parsing, our agent uses visual understanding to identify trend images and calculate precise interaction coordinates Content Analysis: Each discovered trend undergoes detailed analysis using GPT-4o's advanced vision capabilities, extracting insights about fashion elements, seasonal trends, and style patterns Autonomous Decision Making: The agent contextually understands when information should be preserved and automatically engages with cloud storage systems Technology Stack Overview At the heart of this solution lies an orchestration of several AI technologies, each serving a specific purpose in creating a truly autonomous agent. The architecture used ``` ┌─────────────────────────────────────────────────────────────────┐ │ Azure AI Foundry │ │ ┌─────────────────────────────────────────────────────────┐ │ │ │ Responses API │ │ │ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────────┐ │ │ │ │ │ CUA Model │ │ GPT-4o │ │ Built-in MCP │ │ │ │ │ │ (Interface) │ │ (Content) │ │ Client │ │ │ │ │ └─────────────┘ └─────────────┘ └─────────────────┘ │ │ │ └─────────────────────────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────┐ │ Function Calling Layer │ │ (Workflow Orchestration) │ └─────────────────────────────────────────┘ │ ▼ ┌─────────────────┐ ┌──────────────────┐ │ Playwright │◄──────────────► │ Trends Compiler │ │ Automation │ │ Engine │ └─────────────────┘ └──────────────────┘ │ ▼ ┌─────────────────────┐ │ Azure Blob │ │ Storage (MCP) │ └─────────────────────┘ ``` Azure OpenAI Responses API At the core of the agentic architecture in this solution, the Responses API provides intelligent decision-making capabilities that determine when to invoke Computer Use models for web crawling versus when to engage MCP servers for data persistence. This API serves as the brain of our agent, contextually understanding user intent and autonomously choosing the appropriate tools to fulfill complex multi-step workflows. Computer Use (CUA) Model Our specialized CUA model excels at visual understanding of web interfaces, providing precise coordinate mapping for browser interactions, layout analysis, and navigation planning. Unlike general-purpose language models, the CUA model is specifically trained to understand web page structures, identify interactive elements, and provide actionable coordinates for automated browser control. Playwright Browser Automation Acting as the hands of our agent, Playwright executes the precise actions determined by the CUA model. This robust automation framework translates AI insights into real-world browser interactions, handling everything from clicking and typing to screenshot capture and page navigation with pixel-perfect accuracy. GPT-4o Vision Model for Content Analysis While the CUA model handles interface understanding, GPT-4o provides domain-specific content reasoning. This powerful vision model analyzes fashion trends, extracts meaningful insights from images, and provides rich semantic understanding of visual content—capabilities that complement rather than overlap with the CUA model's interface-focused expertise. Model Context Protocol (MCP) Integration The application showcases the power of agentic AI through its autonomous decision-making around data persistence. The agent intelligently recognizes when compiled information needs to be stored and automatically engages with Azure Blob Storage through MCP integration, without requiring explicit user instruction for each storage operation. Unlike traditional function calling patterns where custom applications must relay MCP calls through client libraries, the Responses API includes a built-in MCP client that directly communicates with MCP servers. This eliminates the need for complex relay logic, making MCP integration as simple as defining tool configurations. Function Calling Orchestration Function calling orchestrates the complex workflow between CUA model insights and Playwright actions. Each step is verified and validated before proceeding, ensuring robust autonomous operation without human intervention throughout the entire trend discovery and analysis process. Let me walk you through the code used in the Application. Agentic Decision Making in Action Let's examine how our application demonstrates true agentic behavior through the main orchestrator in `app.py`: async def main() -> str: """Main entry point demonstrating agentic decision making.""" conversation_history = [] generated_reports = [] while True: user_query = input("Enter your query for fashion trends:-> ") # Add user input to conversation context new_user_message = { "role": "user", "content": [{"type": "input_text", "text": user_query}], } conversation_history.append(new_user_message) # The agent analyzes context and decides on appropriate actions response = ai_client.create_app_response( instructions=instructions, conversation_history=conversation_history, mcp_server_url=config.mcp_server_url, available_functions=available_functions, ) # Process autonomous function calls and MCP tool invocations for output in response.output: if output.type == "function_call": # Agent decides to compile trends function_to_call = available_functions[output.name] function_args = json.loads(output.arguments) function_response = await function_to_call(**function_args) elif output.type == "mcp_tool_call": # Agent decides to use MCP tools for storage print(f"MCP tool call: {output.name}") # MCP calls handled automatically by Responses API Key Agentic Behaviors Demonstrated: Contextual Analysis: The agent examines conversation history to understand whether the user wants trend compilation or storage operations Autonomous Tool Selection: Based on context, the agent chooses between function calls (for trend compilation) and MCP tools (for storage) State Management: The agent maintains conversation context across multiple interactions, enabling sophisticated multi-turn workflows Function Calling Orchestration: Autonomous Web Intelligence The `TrendsCompiler` class in `compiler.py` demonstrates sophisticated autonomous workflow orchestration: class TrendsCompiler: """Autonomous trends compilation with multi-step verification.""" async def compile_trends(self, user_query: str) -> str: """Main orchestration loop with autonomous step progression.""" async with LocalPlaywrightComputer() as computer: state = {"trends_compiled": False} step = 0 while not state["trends_compiled"]: try: if step == 0: # Step 1: Autonomous Pinterest navigation await self._launch_pinterest(computer) step += 1 elif step == 1: # Step 2: CUA-driven search and coordinate extraction coordinates = await self._search_and_get_coordinates( computer, user_query ) if coordinates: step += 1 elif step == 2: # Step 3: Autonomous content analysis and compilation await self._process_image_results( computer, coordinates, user_query ) markdown_report = await self._generate_markdown_report( user_query ) state["trends_compiled"] = True except Exception as e: print(f"Autonomous error handling in step {step}: {e}") state["trends_compiled"] = True return markdown_report Autonomous Operation Highlights: Self-Verifying Steps: Each step validates completion before advancing Error Recovery: Autonomous error handling without human intervention State-Driven Progression: The agent maintains its own execution state No User Prompts: Complete automation from query to final report Pinterest's Unique Challenge: Visual Coordinate Intelligence One of the most impressive demonstrations of CUA model capabilities lies in solving Pinterest's hidden URL challenge: async def _detect_search_results(self, computer) -> List[Tuple[int, int, int, int]]: """Use CUA model to extract image coordinates from search results.""" # Take screenshot for CUA analysis screenshot_bytes = await computer.screenshot() screenshot_b64 = base64.b64encode(screenshot_bytes).decode() # CUA model analyzes visual layout and identifies image boundaries prompt = """ Analyze this Pinterest search results page and identify all trend/fashion images displayed. For each image, provide the exact bounding box coordinates in the format: <click>x1,y1,x2,y2</click> Focus on the main content images, not navigation or advertisement elements. """ response = await self.ai_client.create_cua_response( prompt=prompt, screenshot_b64=screenshot_b64 ) # Extract coordinates using specialized parser coordinates = self.coordinate_parser.extract_coordinates(response.content) print(f"CUA model identified {len(coordinates)} image regions") return coordinates The Coordinate Calculation: def calculate_centers(self, coordinates: List[Tuple[int, int, int, int]]) -> List[Tuple[int, int]]: """Calculate center coordinates for precise clicking.""" centers = [] for x1, y1, x2, y2 in coordinates: center_x = (x1 + x2) // 2 center_y = (y1 + y2) // 2 centers.append((center_x, center_y)) return centers key take aways with this approach: No DOM Dependency: Pinterest's hover-based URL revelation becomes irrelevant Visual Understanding: The CUA model sees what humans see—image boundaries Pixel-Perfect Targeting: Calculated center coordinates ensure reliable clicking Robust Navigation: Works regardless of Pinterest's frontend implementation changes Model Specialization: The Right AI for the Right Job Our solution demonstrates sophisticated AI model specialization: async def _analyze_trend_page(self, computer, user_query: str) -> Dict[str, Any]: """Use GPT-4o for domain-specific content analysis.""" # Capture the detailed trend page screenshot_bytes = await computer.screenshot() screenshot_b64 = base64.b64encode(screenshot_bytes).decode() # GPT-4o analyzes fashion content semantically analysis_prompt = f""" Analyze this fashion trend page for the query: "{user_query}" Provide detailed analysis of: 1. Fashion elements and style characteristics 2. Color palettes and patterns 3. Seasonal relevance and trend timing 4. Target demographics and style categories 5. Design inspiration and cultural influences Format as structured markdown with clear sections. """ # Note: Using GPT-4o instead of CUA model for content reasoning response = await self.ai_client.create_vision_response( model=self.config.vision_model_name, # GPT-4o prompt=analysis_prompt, screenshot_b64=screenshot_b64 ) return { "analysis": response.content, "timestamp": datetime.now().isoformat(), "query_context": user_query } Model Selection Rationale: CUA Model: Perfect for understanding "Where to click" and "How to navigate" GPT-4o: Excels at "What does this mean" and "How is this relevant" Specialized Strengths: Each model operates in its domain of expertise Complementary Intelligence: Combined capabilities exceed individual model limitations Compilation and Consolidation async def _generate_markdown_report(self, user_query: str) -> str: """Consolidate all analyses into comprehensive markdown report.""" if not self.image_analyses: return "No trend data collected for analysis." # Intelligent report structuring report_sections = [ f"# Fashion Trends Analysis: {user_query}", f"*Generated on {datetime.now().strftime('%B %d, %Y')}*", "", "## Executive Summary", await self._generate_executive_summary(), "", "## Detailed Trend Analysis" ] # Process each analyzed trend with intelligent categorization for idx, analysis in enumerate(self.image_analyses, 1): trend_section = [ f"### Trend Item {idx}", analysis.get('analysis', 'No analysis available'), f"*Analysis timestamp: {analysis.get('timestamp', 'Unknown')}*", "" ] report_sections.extend(trend_section) # Add intelligent trend synthesis report_sections.extend([ "## Trend Synthesis and Insights", await self._generate_trend_synthesis(), "", "## Recommendations", await self._generate_recommendations() ]) return "\n".join(report_sections) Intelligent Compilation Features: Automatic Structuring: Creates professional report formats automatically Content Synthesis: Combines individual analyses into coherent insights Temporal Context: Maintains timestamp and query context Executive Summaries: Generates high-level insights from detailed data Autonomous Storage Intelligence Note that there is no MCP Client code that needs to be implemented here. The integration is completely turnkey, through configuration alone. # In app_client.py - MCP tool configuration def create_app_tools(self, mcp_server_url: str, available_functions: Dict[str, Any]) -> List[Dict[str, Any]]: """Configure tools with automatic MCP integration.""" tools = [ { "type": "mcp", "server_label": "azure-storage-mcp-server", "server_url": mcp_server_url, "require_approval": "never", # Autonomous operation "allowed_tools": ["create_container", "list_containers", "upload_blob"], } ] return tools # Agent instructions demonstrate contextual intelligence instructions = f""" Step1: Compile trends based on user query using computer use agent. Step2: Prompt user to store trends report in Azure Blob Storage. Use MCP Server tools to perform this action autonomously. IMPORTANT: Maintain context of previously generated reports. If user asks to store a report, use the report generated in this session. """ Turnkey MCP Integration: Direct API Calls: MCP tools called directly by Responses API No Relay Logic: No custom MCP client implementation required Autonomous Tool Selection: Agent chooses appropriate MCP tools based on context Contextual Storage: Agent understands what to store and when Demo and Code reference Here is the GitHub Repo of the Application described in this post. See a demo of this application in action: Conclusion: Entering the Age of Practical Agentic AI The Fashion Trends Compiler Agent represents Agentic AI applications that work autonomously in real-world scenarios. By combining Azure AI Foundry's turnkey MCP integration with specialized AI models and robust automation frameworks, we've created an agent that doesn't just follow instructions but intelligently navigates complex multi-step workflows with minimal human oversight. Ready to build your own agentic AI solutions? Start exploring Azure AI Foundry's MCP integration and Computer Use capabilities to create the next generation of intelligent automation.
