Authored by Sanjana Mohan, Carmel Zolkov, and Moran Assaf, Edge RAG Product Management
During Ignite 2024, we explored how Azure’s adaptive cloud approach is reshaping the AI landscape—enabling organizations to build, deploy, and scale AI solutions across hybrid and multicloud environments with consistency and control. That foundation is now evolving with a powerful new capability: Retrieval-Augmented Generation (RAG).
RAG represents a pivotal shift in how enterprises can ground generative AI in their own data. By combining the reasoning power of large language models (LLMs) with real-time access to enterprise content, RAG enables more accurate, context-aware, and trustworthy responses. This is especially critical in hybrid environments where data is distributed across on-premises systems, edge locations, and multiple clouds.
What is RAG?
Retrieval-Augmented Generation (RAG) is a technique in AI that enhances the performance of language models by combining two steps:
- Retrieve: The model first fetches relevant information from external sources (e.g., documents, databases, or vector indexes).
- Generate: It then uses this retrieved content to generate more accurate, grounded, and context-aware responses.
This approach helps reduce hallucinations, improves factual accuracy, and allows models to work with up-to-date or domain-specific data without retraining
We’re excited to further expand RAG capabilities on Azure Local and enable customers to:
- Ground AI in their own data—whether stored in Azure, on-premises, or across multicloud environments—without needing to move or duplicate it.
- Maintain data sovereignty and compliance by keeping sensitive data within jurisdictional boundaries while still enabling AI to reason over it.
- Accelerate time to insight by integrating RAG into existing applications and workflows using Azure Arc.
This evolution is part of our broader vision to make Azure the most open, extensible, and intelligent cloud for AI innovation—where your data, wherever it lives, becomes a strategic asset for transformation.
RAG on Azure Local
Customers can bring their private cloud data to language models to build generative AI applications and create a retrieval system for RAG-based applications. The capability is available as a first-party extension from Azure Arc for Kubernetes, packaging the end-to-end data ingestion and retrieval pipeline. It also includes essential developer features like prompt engineering, evaluation, and monitoring through a local developer portal.
Image 1: The chat interface includes options to control the inference model and several parameters, as well as the system prompt that can be adjusted for the specific use case.
The RAG capabilities on Azure Local enable organizations to bring Generative AI to their on-premises data, eliminating the necessity of transmitting any information to the cloud. This No-Code/Low-Code experience provides an intuitive interface, allowing users to deploy and manage AI models without the need for extensive programming skills while addressing several critical concerns:
- Data Privacy and Compliance: Maintains proprietary data on-premises, ensuring adherence to data protection regulations and internal policies.
- Reduced Latency: Processes data locally, resulting in faster response times essential for real-time applications.
- Bandwidth Efficiency: Eliminates the requirement to transfer large datasets to the cloud, conserving network resources.
- Scalability and Flexibility: Utilizes Azure Arc to manage and scale Kubernetes clusters seamlessly across diverse environments.
Discovering the Advanced Capabilities of RAG on Azure Local
- Support for Hybrid Search, and soon Lazy Graph RAG, allowing robust, fast, low-cost indexing and providing quality and relevant answers regardless of query type.
- Evaluation flows: includes built-in evaluation features to assess the quality and performance of the RAG system. These features support multiple experimentation flows, allowing for concurrent experimentation and evaluation.
- Multi-Modality: supports multi-modal RAG, which includes handling images, documents, and soon videos. It uses the best parsers available for each media type, focusing on unstructured data hosted on Network File System (NFS) shares. This capability allows for comprehensive data analysis across different formats.
- Support for multiple languages: 100+ common languages for document ingestion and question-answer sessions
- Language Models Updates: ensures that language models are kept up to date with each extension update. This means that users will always have access to the latest advancements in language model technology, ensuring optimal performance and accuracy.
- Managed Responsible AI: ensures features to manage security and regulatory compliance, reducing the burden on developers. It ensures content safety and responsible AI practices are followed, helping developers navigate the complexities of regulatory requirements and maintain high standards of security.
Image 2: The capability includes built-in evaluation feature, reducing the operational overhead of building and maintaining custom RAG solutions, based on Phi-4-Multi-Modal.
Key Use Cases and Scenarios
Deploying RAG at the edge on AKS clusters running on Azure Local empowers organizations to leverage generative AI capabilities while maintaining data sovereignty, ensuring compliance, and reducing latency. Here are some key use cases and scenarios:
Financial Services
A financial institution can utilize it to process and analyze sensitive data that must remain on-premises due to regulatory constraints. This enables use cases such as:
- Compliance Checks: Automating the review of transactions and documents to ensure they meet regulatory requirements.
- Customer Assistance: Providing personalized support and recommendations to customers based on their financial data.
- Sales Pitch Generation: Creating tailored sales pitches and marketing materials by analyzing customer data and preferences.
Manufacturing
A manufacturing company can deploy it to enhance operations and support factory floor activities. Key use cases include:
- Issue Resolution: Reducing the time to resolve issues by providing real-time troubleshooting assistance using local data.
- Operational Efficiency: Analyzing production data to optimize processes and improve efficiency.
- Predictive Maintenance: Using historical data to predict equipment failures and schedule maintenance proactively.
Public Sector
Public sectors can leverage it to derive insights from sensitive on-premises data, enabling various applications such as:
- Decision Making: Summarizing large datasets to provide actionable insights for quicker decision-making.
- Training and Education: Creating training materials and educational content by analyzing and summarizing relevant data.
- Public Safety: Enhancing public safety measures by analyzing local data to identify patterns and predict potential threats.
Healthcare
Healthcare providers can benefit from deploying it to manage and analyze patient data securely. Use cases include:
- Patient Care: Providing personalized treatment plans and recommendations based on patient data.
- Medical Research: Analyzing clinical data to support medical research and development.
- Operational Management: Improving hospital operations by analyzing data related to patient flow, resource utilization, and more.
Retail
Retail businesses can use it to enhance customer experiences and optimize operations. Key scenarios include:
- Personalized Marketing: Creating personalized marketing campaigns based on customer purchase history and preferences.
- Inventory Management: Analyzing sales data to optimize inventory levels and reduce stockouts.
- Customer Insights: Gaining insights into customer behavior and preferences to improve product offerings and services.
By deploying RAG on Azure Local, organizations across various industries can harness the power of generative AI while ensuring data remains secure and compliant with local regulations.
Resources
Microsoft