Transitioning from Azure Language Features to Foundry Models

Azure AI Language in Foundry Tools will begin deprecating eight features:

Key Phrase Extraction, Entity Linking, Sentiment Analysis & Opinion Mining, Summarization (document & conversation), Conversational Language Understanding (CLU), Custom Question Answering (CQA), Orchestration Workflow, and Custom Text Classification,

to be ultimately retired by March 2029 (Entity Linking even sooner, by September 20, 2028).

Microsoft’s guidance is to implement these functionalities using Microsoft Foundry-based models. This series of recommendation guides is intended to help you get started as you explore the breadth of Foundry-based options at your fingertips. Each standalone guide includes:

• A brief overview of the Azure AI Language in Foundry Tools feature.
• A recommended Foundry model and deployment for a similar solution.
• A tutorial to implement the new solution via Foundry’s REST API.
• A point-in-time comparison table listing alternate Foundry models you might consider for each feature.

Endpoint conventions used in this guide

The REST examples below use the unified Microsoft Foundry inference endpoint, which is the same Azure OpenAI path used for every model (OpenAI, Mistral, Cohere, Phi, and more): https://<your-resource>.services.ai.azure.com/openai/deployments/<your-deployment>/chat/completions?api-version=2024-10-21. Authenticate with header api-key: <YOUR_API_KEY>. <your-deployment> is the deployment name you assigned when adding the model to your Foundry resource.

For keyless (Microsoft Entra ID) authentication, swap the URL to https://<your-resource>.openai.azure.com/openai/v1/chat/completions and use header Authorization: Bearer <ENTRA_TOKEN> (issue the token with scope https://ai.azure.com/.default).

The older Azure AI Inference beta route (<endpoint>.<region>.models.ai.azure.com/chat/completions) is being retired on August 26, 2026 — use the unified path above for any new code.

New: Microsoft’s first-party MAI models in Foundry. Microsoft’s in-house MAI models are now available in Microsoft Foundry. For the text and language features covered in this guide, the relevant new option is MAI-Thinking-1 — Microsoft’s first reasoning model (sparse Mixture-of-Experts, ~35B active / ~1T total parameters, 256K-token context), introduced at Build 2026 and currently in private preview. It is Chat Completions API-compatible and supports function calling, so it slots into the same call patterns shown below, and it now appears as an alternate option in the comparison tables for the reasoning- and context-heavy features. The MAI media models released in April 2026 — MAI-Transcribe-1 (speech-to-text), MAI-Voice-1 (text-to-speech), and MAI-Image-2 (image generation) — are not covered here, since they address audio and vision rather than the text features being retired.

1. Key Phrase Extraction → Foundry with mistral-small-2503 (24B)

Key Phrase Extraction identifies main ideas or topics in unstructured text. For example, given the sentence “The food was delicious and the staff were wonderful.”, it returns the phrases “food” and “staff” as the key topics. In Foundry, you can replicate this by prompting a knowledge-dense model to extract significant terms from text and return them in JSON format. We recommend trying out mistral-small-2503, a 24-billion-parameter open model that rivals larger proprietary models on comprehension tasks.

Step 1: Deploy the Mistral model in Foundry. In your Foundry project, deploy the Mistral Small 3.1 model (mistral-small-2503). This model is licensed for enterprise use and can process extremely long documents (128,000 tokens) with state-of-the-art accuracy. Once deployed, note its deployment name and region from the Azure portal.

Step 2: Write a prompt for key phrase extraction. Instruct the model to extract key phrases and output them as JSON by writing a prompt for key phrase extraction. For example, your prompt can say: “Extract the key phrases from the given text and output them in JSON as a list.”

Step 3: Call the Foundry REST API with your text. Use your Foundry endpoint and API key to send a POST request to the model’s chat completion API (similar to how you called Azure’s API). Include a system message with extraction instructions, and a user message containing your text. For example:

1     POST https://<your-resource>.services.ai.azure.com/openai/deployments/<your-deployment>/chat/completions?api-version=2024-10-21
2     Headers: { "api-key": "<YOUR_API_KEY>", "Content-Type": "application/json" } 
3     Body: {
4       "messages": [
5         {
6           "role": "system",
7           "content": "Extract the key phrases (main topics) from the given text. Respond in JSON as {\"keyPhrases\": [ ... ]}."
8         },
9         {
10           "role": "user",
11           "content": "Text: \"The food was delicious and there were wonderful staff.\"\nTask: What are the key phrases?"
12         }
13       ]
14     }

Step 4: Receive the JSON output. The Mistral model will parse the text and return the key concepts in JSON form. For this example, it responds with:

1     {
2       "keyPhrases": [
3         "food",
4         "staff"
5       ]
6     }

You can write your prompt to return a desired structured output to ensure that it returns a list of extracted key phrases. Your application can then consume the "keyPhrases" array.

Other recommended models for Key Phrase Extraction:

2. Entity Linking → Foundry with gpt-4o + Retrieval

Azure AI Entity Linking identifies named entities in text (people, places, organizations, etc.) and links each to a reference ID or entry. While Azure AI’s stand-alone Entity Linking will be retired, Microsoft’s officially recommended replacement is Azure AI Language in Foundry Tools’ Named Entity Recognition (NER), which continues to be supported and identifies the same entity types. If you need URL-linked entities, multilingual disambiguation, or richer reasoning beyond NER’s tag set, Foundry offers an alternative using a combination of knowledge retrieval and a generative model. One option is to try out gpt-4o, which excels at reasoning and handling complex instructions. By pairing it with your enterprise or public knowledge sources, you can link entities accurately.

Step 1: Set up an entity knowledge base. Prepare the reference data for your entities. For public knowledge, you might use Wikipedia; for internal terms, index the definitions using Azure AI Search. This knowledge base will allow the LLM to retrieve factual descriptions for any given entity.

Step 2: Deploy gpt-4o in Foundry. In your Foundry resource, deploy gpt-4o from the Foundry model catalog (or choose an equivalent high-accuracy model). This model is designed for broad reasoning and “agentic” tasks – meaning it can use tools and contextual data to solve problems. With strong general knowledge and reasoning, it’s ideal for disambiguating entities.

Step 3: Implement retrieval + linking prompt. When processing text, first retrieve relevant information about each potential entity. For instance, if the text says “Jane moved to Paris to work at Microsoft.”, search your knowledge base or call an API to get info on “Paris” and “Microsoft”. Then, craft a prompt that provides this context to the model and asks it to output a JSON mapping of entities to their IDs/URLs. For example:

1     {
2       "messages": [
3         {"role": "system", "content":
4             "You are an entity linking assistant. Identify entities in text and link each to its official reference ID and URL. Use provided context as needed and respond with JSON."},
5         {"role": "user", "content":
6             "Text: 'Jane moved to Paris to work at Microsoft.'\n\n" +
7             "Context:\n" +
8             "Paris – City in France (ID: Paris, URL: https://en.wikipedia.org/wiki/Paris).\n" +
9             "Microsoft – Technology company (ID: Microsoft, URL: https://en.wikipedia.org/wiki/Microsoft).\n\n" +
10             "Task: List each entity with its name, id, and url in JSON format as {\"entities\": [...] }."
11         }
12       ]
13     }

Step 4: Get the JSON linking output. The GPT-4o model, grounded by the context you provided, will output a JSON block such as:

1     {
2       "entities": [
3         {
4           "name": "Paris",
5           "id": "Paris",
6           "url": "https://en.wikipedia.org/wiki/Paris"
7         },
8         {
9           "name": "Microsoft",
10           "id": "Microsoft",
11           "url": "https://en.wikipedia.org/wiki/Microsoft"
12         }
13       ]
14     }

Other recommended models for Entity Linking:

3. Sentiment Analysis & Opinion Mining → Foundry with phi-3.5-mini (3.8B)

Azure AI Sentiment Analysis evaluates whether text expresses a positive, neutral, or negative tone, while Opinion Mining provides granular sentiment on specific aspects or targets in the text. For example, in a hotel review “The room was great, but the staff was unfriendly,” the overall sentiment might be mixed, with aspect-level results: room: positive; staff: negative. We suggest you try a model, like Phi-3.5-mini model, which supports 128K context and was fine-tuned by Microsoft for quality instruction following. This model handles sentiment classification with high accuracy and speed, rivaling much larger models.

Step 1: Deploy the Phi model. In the Foundry portal model catalog, search for Phi-3.5-mini-instruct (128k) and deploy it. Once deployed, record the deployment name and have your API key ready for the following steps.

Step 2: Prepare your sentiment analysis prompt. Write a prompt that instructs the model to produce a sentiment analysis JSON output. Use a prompt like: "Analyze the following text. Provide 'sentiment' (positive, neutral, negative) and, if applicable, list 'opinions' of specific aspects and their sentiment. Respond in JSON." This explicitly asks for an overall sentiment and aspects that inform the predicted sentiment.

Step 3: Send the text to the model’s REST endpoint. Using your Foundry endpoint and key, call the phi-3.5-mini chat API with the text. For an example input:

1     {
2       "messages": [
3         {"role": "system", "content":
4             "Analyze the sentiment. Output JSON with overall 'sentiment' and 'opinions' for aspects mentioned."},
5         {"role": "user", "content":
6             "Review: 'The room was great, but the staff was unfriendly and rude.'"}
7       ]
8     }

Step 4: Review the output JSON. Once you adjust your prompt to your use case, the model can provide an analysis similar to Azure’s service output:

1     {
2       "sentiment": "mixed",
3       "confidenceScores": {
4         "positive": 0.5,
5         "neutral": 0.0,
6         "negative": 0.5
7       },
8       "opinions": [
9         {
10           "aspect": "room",
11           "sentiment": "positive"
12         },
13         {
14           "aspect": "staff",
15           "sentiment": "negative"
16         }
17       ]
18     }

The Phi model’s output is factual and concise, capturing the correct tone for each element. You can adjust the prompt for more or fewer details (e.g., add confidenceScores or not) as needed.

Other recommended models for Sentiment:

4. Text & Conversation Summarization → Foundry with claude-sonnet-4-6

Azure AI Summarization automatically produces a concise summary of input text. Azure AI’s summarization includes extractive (selecting key sentences) and abstractive (generating new summary text) modes for documents and conversations. To continue summarizing content, we recommend Claude Sonnet 4.6, known for its strength in summarization and very long context support. Claude can ingest large documents or transcripts and generate coherent, human-like summaries.

Step 1: Deploy the Claude model. In Foundry, deploy Claude Sonnet 4.6 as your model. With Claude deployed, note the deployment’s name for use in the API call.

Step 2: Create a summarization prompt. Decide on the summary style (short paragraph, bullet points, etc.). For example: “Summarize the following document in 2-3 sentences.” For conversation transcripts, you might ask: “Summarize the conversation below, highlighting key issues and resolutions.” Also specify that the response should be in JSON (e.g., {"summary": "..."}) for a more structured output style.

Step 3: Call Claude with your content. Use the Foundry API to send the text and prompt. For example, to summarize a document:

1     {
2       "messages": [
3         {"role": "system", "content": "You summarize texts. Output JSON with a 'summary' field."},
4         {"role": "user", "content": "Document: At any given moment, turnaround coordinators at Lufthansa CityLine monitor video feeds of airplanes at gates, ensuring each step of unloading and servicing planes happens on time. Even minutes of delay can add up, costing airlines millions of dollars a year. ...\n\nTask: Summarize the main point of this document in 2-3 sentences."}
5       ]
6     }

Step 4: Inspect the summary output. Claude will reply with a short summary, for example:

1     {
2       "summary": "Lufthansa CityLine uses real-time video monitoring to streamline aircraft turnarounds. By tracking tasks like unloading, refueling, and cleaning, they minimize delays and save significant costs."
3     }
4    

Claude’s summarization quality is excellent for both extractive and abstractive summaries. Additionally, because Claude Sonnet 4.6 can incorporate enormous context, you can feed very long content directly without splitting it.

Conversation summarization: per-aspect starter prompts 

The example above covers plain document summarization. If you are migrating the Conversation Summarization API, note that it let you request one or more summaryAspects: issue, resolution, recap, chapterTitle, and narrative — each with a distinct purpose, so a single generic summary prompt does not reproduce them. In Foundry, reproduce each aspect with its own targeted system prompt on claude-sonnet-4-6, sending the speaker-labeled transcript (for example, Agent: / Customer:) as the user message. The output field names below mirror the Azure API (summaries: [{aspect, text}]) so your integration code barely changes. Request several aspects in parallel calls, or combine them into a single call. 

1. Issue — the customer’s problem (call-center focused)

{"role":"system","content":"You replicate the Azure AI Language conversation summarization 'issue' aspect. Read the agent/customer transcript and summarize ONLY the customer's primary problem — what they contacted support about — from the customer's perspective. Do NOT include any resolution or troubleshooting steps. Respond ONLY with JSON: {\"summaries\":[{\"aspect\":\"issue\",\"text\":\"<concise issue>\"}]}"} 

Example output: 

{"summaries":[{"aspect":"issue","text":"Customer could not set up the Wi-Fi connection for their Smart Brew 300 espresso machine."}]} 

2. Resolution — what fixed it, or didn’t (call-center focused)

{"role":"system","content":"You replicate the Azure AI Language conversation summarization 'resolution' aspect. Summarize ONLY the actions the agent took to resolve the customer's issue and the final outcome, including any steps that failed or remain open. Do NOT restate the issue itself. Respond ONLY with JSON: {\"summaries\":[{\"aspect\":\"resolution\",\"text\":\"<concise resolution>\"}]}"} 

Example output: 

{"summaries":[{"aspect":"resolution","text":"The agent had the customer reset the Wi-Fi button and attempt a factory reset; neither restored the connection, so the issue was escalated for further troubleshooting."}]} 

3. Recap — a brief one-paragraph summary of the whole conversation

{"role":"system","content":"You replicate the Azure AI Language conversation summarization 'recap' aspect. Condense the entire conversation into a single brief paragraph (2–4 sentences) capturing the participants, the main topic, and the outcome. Respond ONLY with JSON: {\"summaries\":[{\"aspect\":\"recap\",\"text\":\"<brief recap>\"}]}"} 

4. Chapter title — topic-based segmentation with titles

{"role":"system","content":"You replicate the Azure AI Language conversation summarization 'chapterTitle' aspect. Segment the conversation into sequential chapters by topic, in chronological order, and give each a short title (max ~10 words). For each chapter, also return the verbatim opening line so offsets can be recomputed client-side. Respond ONLY with JSON: {\"summaries\":[{\"aspect\":\"chapterTitle\",\"text\":\"<chapter title>\",\"startsAt\":\"<first utterance of the chapter, verbatim>\"}]}"} 

Azure returned character offsets per chapter. Because an LLM cannot reliably count characters, this prompt returns the verbatim opening line of each chapter; map it back to an offset with a string search on the client side. 

5. Narrative — detailed call or meeting notes

{"role":"system","content":"You replicate the Azure AI Language conversation summarization 'narrative' aspect. Produce detailed call notes / meeting notes for the conversation — more thorough than a recap — in coherent prose or structured notes, preserving the order events occurred and any decisions or follow-ups. Respond ONLY with JSON: {\"summaries\":[{\"aspect\":\"narrative\",\"text\":\"<detailed notes>\"}]}"} 

Migration note: where you previously passed summaryAspects: ["issue","resolution"] in a single call, now select the matching prompt(s) above. For multiple aspects, make parallel calls (cleanest — keeps each output focused) or combine the instructions and ask for one summaries array containing every requested aspect. 

Other recommended models for Summarization:

5. Conversational Language Understanding (CLU) → Foundry (claude-sonnet-4-6, or fine-tuned model)

Azure AI’s Conversational Language Understanding (CLU) allows chatbots to interpret user utterances by predicting a top intent and extracting key entities. For example, given “Book a flight to Paris tomorrow”, a custom CLU model might return Intent: BookFlight and Entities like Destination = Paris, Date = tomorrow. Foundry offers two solutions:

• Approach A – Prompt-based CLU with an LLM: Use a single advanced model (like Claude Sonnet 4.6) to classify intents zero-shot and format a JSON response with the predicted intent and entities. This avoids any training and leverages the model’s high generalization ability.
• Approach B – Fine-tuned custom model: Fine-tune an open model (e.g., Ministral 3B) on your existing labeled utterances to get a dedicated classifier for your scenario. This requires training (similar to CLU today) but yields maximum domain accuracy and offline deployment.

Approach A: Use claude-sonnet-4-6 as an Intent & Entity Model

Step 1: Gather CLU schema and examples. Assemble your existing list of intents (and what each means), plus any entity types. Include a None intent for irrelevant utterances as CLU does. For example, suppose your bot’s intents are: GetWeather (ask about weather), BookFlight (book a flight), CancelBooking (cancel a booking), and None (no match).

Step 2: Build a detailed prompt. Incorporate the above schema into a prompt for the model. You might create a system message like: “

"You are a conversational NLU classifier that mimics Azure Conversational Language Understanding (CLU). Your job is to read the user's text and return a CLU-like prediction JSON for intents and entities.\n\nTASK\n1) Identify the single best matching intent from the ALLOWED INTENTS list below.\n2) Extract zero or more entities from the text using the ALLOWED ENTITY TYPES list below.\n3) Return ONLY valid JSON that follows the OUTPUT SCHEMA exactly (no markdown, no explanation, no extra keys).\n\nALLOWED INTENTS (choose exactly one)\n- GetWeather: user is asking about weather conditions or forecast.\n- BookFlight: user is requesting to book, reserve, or search for a flight.\n- CancelBooking: user is requesting to cancel an existing reservation/booking (flight/hotel/etc.).\n- None: none of the intents above apply.\n\nALLOWED ENTITY TYPES (extract only these)\n- Location: city, region, airport, or place name that is relevant to the request.\n- Date: a date or date-like phrase (e.g., \"tomorrow\", \"next Friday\", \"2026-03-24\").\n\nSCORING RULES\n- intentConfidence must be a number from 0.00 to 1.00.\n- If the intent is None, set intentConfidence between 0.00 and 0.49 and return an empty entities array.\n- If an intent matches well, set intentConfidence between 0.70 and 1.00.\n- If uncertain between two intents, pick the best one and set intentConfidence between 0.50 and 0.69.\n\nENTITY RULES\n- Each entity object MUST include: category, text, offset, length, confidenceScore.\n- offset is the 0-based character index of the first character of the entity span in the ORIGINAL user text.\n- length is the number of characters in the span.\n- confidenceScore must be a number from 0.00 to 1.00.\n- Do not invent entities that are not explicitly present in the text.\n\nOUTPUT SCHEMA (return exactly this structure)\n{\n \"query\": \"<original user query>\",\n \"prediction\": {\n \"topIntent\": \"<best matching intent name>\",\n \"intents\": {\n \"GetWeather\": {\"confidenceScore\": <0.00-1.00>},\n \"BookFlight\": {\"confidenceScore\": <0.00-1.00>},\n \"CancelBooking\": {\"confidenceScore\": <0.00-1.00>},\n \"None\": {\"confidenceScore\": <0.00-1.00>}\n },\n \"entities\": [\n {\n \"category\": \"<entity type, e.g. Location or Date>\",\n \"text\": \"<entity span text>\",\n \"offset\": <integer>,\n \"length\": <integer>,\n \"confidenceScore\": <0.00-1.00>\n }\n ]\n }\n}\n\nCONSTRAINTS\n- Output MUST be valid JSON and MUST match the schema.\n- Populate confidence scores for ALL intents; the topIntent should have the highest score.\n- If no entities are found, return an empty array for prediction.entities.\n- Do not output any additional text before or after the JSON."},”

This explicit enumeration guides Claude to stick to the known intents only.

Step 3: Call the model with user utterances. Send a chat completion request for each user query. For example, for “Is it going to rain in Seattle tomorrow?”:

1     {
2       "messages": [

3         {"role": "system", "content":
4           "Intents:\n- GetWeather: user asks about weather.\n- BookFlight: user wants to book a flight.\n- CancelBooking: user wants to cancel a booking.\n- None: input doesn't match above.\nEntities to extract: Location, Date.\nOutput JSON with {\"intent\":..., \"entities\":...} for each user query."
5         },
6         {"role": "user", "content": "Is it going to rain in Seattle tomorrow?"}
7       ]
8     }

Step 4: Examine the JSON response. Claude uses your schema to infer the best intent and fills in entities:

1     {
2       "intent": "GetWeather",
3       "entities": [
4         {
5           "category": "Location",
6           "text": "Seattle"
7         },
8         {
9           "category": "Date",
10           "text": "tomorrow"
11         }
12       ]
13     }

Note that the example above is the simplified shape your application typically consumes — the prompt in Step 2 actually requires the model to return the full CLU-compatible schema (query, prediction.topIntent, prediction.intents.<each>.confidenceScore, and entities with offset, length, and confidenceScore). Extract just the fields you need on the client side, or relax the Step 2 schema to match the simplified output. With careful prompting, Claude Sonnet 4.6’s classifier can approach the accuracy of your old CLU model – without any fine-tuning – and is easy to maintain because you can tweak the schema in the prompt itself.

Approach B: Fine-tune a Custom Intent Classification Model

If you have many intents and domain-specific language, training a custom model gives the highest accuracy. Fine-tuning an open model on your utterances yields a tailor-made classifier you control and can be deployed off-cloud if needed.

Step 1: Export CLU training data. Collect the utterances, intent labels, and annotated entities from your CLU project. Ensure each example is labeled correctly. This will be your training dataset.

Step 2: Fine-tune an open model in Foundry. Use Foundry’s fine-tuning UI or API to train one of the supported base models — Ministral-3B, Qwen-32B, or Llama-3.3-70B-Instruct for open-source; gpt-4.1-mini or gpt-4o-mini for OpenAI-managed — with your data. During training, the model learns to classify your intents and tag entities as in CLU. For more detailed information on fine-tuning in Foundry, check out Customize a model with fine-tuning - Azure AI Foundry | Microsoft Learn. After training, you can deploy the fine-tuned model as a new endpoint.

Step 3: Integrate the fine-tuned model. Once deployed, use the Foundry REST API to send user utterances and get predictions. The response format can be customized, but you can tailor it to predict the predicted intent and possibly entity spans. For instance, if you send “I want to cancel my Contoso booking”, the model might output Intent: CancelBooking; Entity: { type: Subscription, value: “Contoso” } (assuming it learned those classes).

Step 4: Evaluate and iterate. Validate the new model against test data or real queries. You should find it highly accurate on your domain-specific utterances, often matching or exceeding CLU’s performance. Fine-tuning yields a model that doesn’t stray beyond your provided intents, ensuring no unexpected outputs. Additionally, you can re-train periodically with new data to continuously improve it.

Why fine-tune? If your chatbot has a large set of intents or very specialized language, a fine-tuned model will be more consistent and precise than prompting a general LLM. It also runs without sending data to third-party APIs (since the model is your custom deployment). This path is ideal for enterprises needing predictable, controllable NLU – you invest once in training and get a dedicated model that aligns exactly with your taxonomy and can be packaged for offline or on-prem use.

Other recommended models for CLU:

6. Custom Question Answering (CQA) → Foundry with gpt-4o or Cohere Rerank + Retrieval

Azure AI’s Custom Question Answering (CQA) allows you to upload your documents or FAQs and answer natural language queries from that content. You can use a Retrieval-Augmented Generation (RAG) pattern: search for relevant info in your data using Azure Search (L1), then use another model to rank and generate an answer based only on your knowledge base (L2).

Approach A: Use Cohere Rerank v4.0 Pro as a Deterministic L2 Ranker

Step 1: Export your existing CQA knowledge base. Export your current Custom Question Answering knowledge base. You should retain all existing question answer pairs. No retraining or rewriting of answers is required.

Step 2: Create an Azure AI Search index. Create a new Azure AI Search index that will store your existing QA pairs. Each QA pair should be uploaded as a document containing the stored question text and stored answer text. This index will replace the internal retrieval layer previously used by the CQA runtime.

Step 3: Deploy the Cohere Rerank v4.0 Pro model in Foundry. In Microsoft Foundry, deploy the Cohere Rerank v4.0 Pro model to your project. This model will replace the confidence scoring functionality that was previously handled by the CQA service and will be used to determine which stored answer best matches the incoming user query.

Step 4: Update your application’s query endpoint. Locate the existing service call in your application that sends user questions to the CQA endpoint. Replace this call so that user queries are now sent to your Azure AI Search index instead. A minimal index schema for QA retrieval is: id (key), question (searchable text), answer (retrievable text), and questionVector (vector field populated by an embedding model such as text-embedding-3-small). Configure Azure AI Search to run a hybrid query — combining BM25 keyword match on the question field with vector similarity on questionVector — and return the top 10 candidate QA pairs for each query. The reranker in Step 5 will reduce those candidates to the single best answer.

Step 5: Pass search results to the Cohere Reranker. Modify your application workflow so that the QA pairs returned from Azure AI Search are passed to the deployed Cohere Rerank model along with the user’s original query. This replaces the scoring logic that your application previously received from the CQA runtime.

Step 6: Select the highest ranked QA pair. Update your application logic to 1) sort the reranked results returned by Cohere and 2) select the highest ranked result. The answer associated with that QA pair can then be returned directly to the user using the same response logic that currently exists in your application.

Step 7: Return the selected answer to the user. Update your application response logic so that the answer returned to the user is taken from the top ranked QA pair produced by the reranker. This preserves the same deterministic answer selection behavior that existed in the original CQA workflow while allowing your application to operate using Foundry hosted models.

Step 8: Optional GPT-Based Fallback. If needed, you can introduce an optional GPT-based validation or fallback step to evaluate the top reranked results or generate a synthesized response when no candidate answer meets your required relevance threshold. This allows you to gradually introduce retrieval augmented generation without changing the primary deterministic selection path used by your application today.

Approach B: Use GPT-4o as a Generative Answerer (RAG)

Step 1: Index your content for retrieval. Take the knowledge bases (documents, manuals, Q&A pairs, etc.) you were using in CQA and load it into an index, such as Azure Search. Enable keyword vector search so that queries can find relevant passages, not just keyword matches.

Step 2: Deploy the GPT-4o model. Use Foundry to deploy gpt-4o as your answer generator. Also consider deploying a smaller embedding model for vector search if needed (e.g., OpenAI Ada or Mistral Embeddings).

Step 3: Formulate a retrieval-augmented prompt. When a user asks a question, your application will first retrieve the top matching content from the index (e.g., a paragraph from an FAQ or a snippet from a manual). Then, you send the question along with that content to GPT-4o, instructing it to use only the provided content to answer. For example: “Content: [insert sample retrieved passage] \n\n Question: [insert sample user question] \n\n If the answer is in the content, answer in one sentence; otherwise say “Answer not found”.”

Step 4: Call GPT-4o via REST API. The user message will include both the Context (retrieved content) and the Question. GPT-4o will then generate an answer grounded in that context. For instance, if a user asks “How do I reset my device?”, you pass in the relevant instruction from your manual as context. The model’s answer would be factual and derived from those instructions.

Step 5: Validate the answer output. The model should return an accurate answer, e.g.: “Press and hold the reset button for 10 seconds until the device restarts.” In scenarios where the context doesn’t cover the question, the model can be configured to say “Answer not found.” This ensures no hallucinated answers. You can also instruct the model to output references or citations to the source content if needed.

By combining search and an LLM, you can effectively build a question answering model on Foundry: the search step finds the answer, and the model produces a user-friendly statement from it.

Other recommended models for CQA:

7. Orchestration Workflow → Foundry Agent (using gpt-4o)

Azure AI’s orchestration workflow directs incoming requests to the correct Language sub-service (CLU, CQA, etc.) by first determining the user’s goal. Instead of maintaining a separate orchestrator, you can now rely on a Foundry agent – essentially a single LLM that can interpret user input and decide how to fulfill it using the available tools. GPT-4o, for instance, can analyze an utterance and either answer it or format an action, as needed.

Step 1: Define the agent’s tasks. Identify what distinct actions or knowledge the orchestrator used to handle. For example, perhaps the bot either answered a question (using QnA) or executed a command (like scheduling). We will rely on the LLM to detect which scenario applies and respond appropriately.

Step 2: Construct an agent prompt. Write a system message that explains the assistant’s capabilities. For example: “You are an AI assistant. If the user asks a knowledge question, answer it using the provided content. If the user makes a request to perform an action, output a JSON specifying the action.” This effectively gives GPT-4o a simple set of tools (answering vs. acting).

Step 3: Provide context or tools per query. In each conversation turn, include any info needed. For instance, if the user query is factual (“What’s our security policy?”), attach a relevant snippet from your data (like in the QnA solution). If the query is an action (“Schedule a meeting with IT”), you might not have extra context, but expect an action-oriented output.

Step 4: Evaluate the outputs. GPT-4o will analyze the input and produce either a direct answer or an action, following the instructions. For the fact question, it might output a paragraph answer; for the command, it could output a JSON like {"action": "ScheduleMeeting", "topic": "IT Support", "time": "tomorrow 10am"}. Because GPT-4o is an “agentic” model, it can reason about the user’s intent and choose the correct format on its own. This means the separate orchestration layer is no longer needed – the model’s reasoning replaces it.

Using an AI agent for orchestration leads to a simpler flow: you only call the LLM (and perhaps a search for facts) rather than maintaining multiple endpoints and routing logic. Foundry’s advanced models like GPT-4o have demonstrated they can handle multi-step tasks and tool usage in a single conversation, making them a powerful replacement for manual orchestration.

Other recommended models for Orchestration:

8. Custom Text Classification → Foundry (fine-tuned Ministral-3B)

Custom Text Classification allowed you to train a model to classify text into your custom categories (e.g., labeling customer reviews as Complaint, Suggestion, or Praise). In Foundry, you will achieve the same by fine-tuning a foundation model with your data. The ideal base model is Ministral-3B, since it is one of the open-source base models Foundry currently supports for fine-tuning, and it is an Apache 2.0 licensed model with top-tier accuracy in language tasks. Fine-tuning this model with your training data will produce a robust classifier that matches or exceeds your previous model’s performance, with the added benefit that you can fully control and deploy it as needed.

Step 1: Prepare training data. Export your labeled text samples and their categories from the Language Studio (or however you stored them). Ensure you have a representative dataset for each label. Clean the data if needed (e.g., remove duplicates, ensure consistent labeling).

Step 2: Fine-tune Ministral-3B in Foundry. In the Foundry UI or using the Python SDK, start a fine-tuning job on Ministral-3B with your dataset. Foundry will handle the training process; you simply provide the data and some configuration (like number of epochs). Once training is completed, you’ll have a new model version specialized to predict your categories.

Step 3: Deploy and test the model. Deploy the fine-tuned model to get an endpoint. Then send test texts to its REST API. For example, if you send the text “The product arrived broken and I’m very unhappy”, the model might return:

1     {
2       "category": "Complaint",
3       "confidenceScore": 0.98
4     }

This indicates the text was classified as a “Complaint” with high confidence. The output format can mimic the original service’s JSON structure (as shown above), so integration changes are minimal.

Step 4: Integrate and iterate. Update your application to call the new Foundry model endpoint for classification requests. Because the model was trained on your exact labels and examples, it will produce one of your predefined classes every time (no extraneous outputs), and often with greater accuracy than before. Over time, you can retrain with more data or adjust the model if needed, giving you full control over the classifier’s evolution.

Other recommended models for Custom Classification:

By following these guides, you’ll smoothly transition each Azure Language feature to a Microsoft Foundry solution. No direct replacements exist, but by leveraging foundation models + prompt engineering, you can achieve equal or better functionality. This approach consolidates tasks onto a unified, powerful platform with models that are continuously improving. The result is continuity for your applications and an upgrade in capabilities – from static APIs to the flexible world of Foundry’s LLMs. Each guide above equips you to implement the change with minimal friction, ensuring your users enjoy the benefits of modern AI with the structurally similar JSON outputs they expect.