Operationalizing AI powered medical imaging pipeline for cohort building
Authors: Jared Erwin, Senior Software Engineer, HLS Nursing AI and Data Platform, Faculty UW School of Medicine Manoj Kumar, Director, HLS - Data & AI HLS Frontiers AI Alberto Santamaria-Pang, Principal Applied Data Scientist, HLS Frontiers AI and Adjunct Faculty, Johns Hopkins Medicine Overview In Part 1, of this series, we showed how natural language could be used to define medical imaging cohorts and retrieve relevant studies in seconds instead of months. That proof-of-concept demonstrated the value of the idea — but not how to make it repeatable, or production-ready. This post focuses on how we turned that prototype into a production-oriented Azure Machine Learning pipeline — to scale execution and produce clear, versioned artifacts that could drive an interactive cohort exploration UI. If you're building ML pipelines for medical imaging, or any domain where data is large, messy, and locked behind access controls, we hope our experience saves you time. From scripts to a pipeline: Why Azure ML components? The original hackathon implementation consisted of notebooks and scripts that required careful manual execution. To make the system repeatable and auditable, we standardized it using Azure ML pipelines. Azure ML pipelines gave us: Componentized execution — each processing step is a self-contained unit with defined inputs, outputs, and dependencies Parallel branches — steps that don't depend on each other run concurrently Reproducibility — every run is versioned and logged with full lineage Compute flexibility — run on CPU for metadata extraction, GPU for model inference, without manual orchestration The pipeline architecture The pipeline consists of 5 python components arranged in a DAG with two parallel branches: [0]scans a DICOM directory and extracts metadata from headers — study/series UIDs, modality, body part, slice counts. [1]classifies each series by anatomy and orientation using a multi-tier strategy (more on this below). [2] and [3] form the search pipeline: anatomy labels are converted to natural language text templates, then encoded with BiomedCLIP into a FAISS vector index. [4]generates 2D UMAP coordinates from the embeddings for the interactive scatter plot visualization in the UI. The image depicts a flowchart detailing the process of DICOM metadata extraction, anatomy classification, visualization enrichment, and text template generation, followed by the creation of a FAISS vector index. Components 2 and 4 run in parallel after component 1 completes, saving roughly 10-15% of total execution time. It's a modest gain for a single run, but it adds up when iterating on pipeline parameters. [1] Anatomy classification, integrating MedImageInsight The Anatomy classification component in the pipeline relies on MedImageInsight (MI2). MedImageInsight is Microsoft's foundation model for medical image understanding, available through the Azure AI Foundry model catalog. Unlike generative models, MedImageInsight is an embedding model — it maps medical images and text into a shared 1024-dimensional vector space, enabling tasks like classification and similarity search by comparing image embeddings against text label embeddings. Given a DICOM image, we compare its embedding against candidate labels (e.g., "Brain", "Chest", "Abdomen") to determine the body part, scan orientation, and other imaging characteristics through zero-shot classification. We also may get directly annotated anatomy from component 0, the DICOM metadata extractor component. We can combine both data points to build our final search index. [2] [3] FAISS index construction As an input to the FAISS index, we first run component 2, the text template generator. This component takes the metadata and anatomy information from components 0 and 1 and feeds them into 5 different agents with different instructions on how to describe the DICOM study. This results in textual descriptions which some variation, referred to as text templates, which can be indexed in the next component The FAISS index builder (component 3) uses BiomedCLIP to encode all text templates into 512-dimensional vectors: MODEL_NAME = "hf-hub:microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224" @torch.no_grad() def encode(self, texts: List[str], batch_size: int = 256) -> np.ndarray: embeddings = [] for i in range(0, len(texts), batch_size): batch = texts[i:i+batch_size] tokens = self.tokenizer(batch).to(self.device) batch_embeddings = self.model.encode_text(tokens) batch_embeddings = F.normalize(batch_embeddings, dim=-1) # L2 normalize embeddings.append(batch_embeddings.cpu().numpy()) return np.vstack(embeddings) We L2-normalize all vectors and use faiss.IndexFlatIP (inner product), which is equivalent to cosine similarity on normalized vectors. For our current dataset sizes (thousands of series), flat indexing is fast enough. For hospital-scale datasets with millions of images, we might switch to IndexIVFFlat or IndexHNSW for approximate nearest neighbor search. In the cohort explorer app, a user will enter a natural language query, which is then converted to embeddings using the same BiomedCLIP model. This allows a search using the FAISS index to find relevant DICOM studies. [4] Visualization: making embeddings explorable The scatter plot in the UI is often the first thing users interact with. It needs to show meaningful clusters without requiring users to understand dimensionality reduction. Component 4 takes the embeddings from component 1 and projects them to 2D with UMAP: umap = UMAP( n_components=2, n_neighbors=10, # Balances local vs. global structure min_dist=0.5, # Prevents over-clustering metric='cosine', # Matches our embedding similarity metric random_state=42 # Reproducible layouts ) coordinates_2d = umap.fit_transform(features) Each point in the scatter plot corresponds to a single DICOM series produced by the pipeline, with color, grouping, and hover metadata derived directly from the JSON artifacts emitted by components 1 and 4. Each pipeline run produces a small set of well-defined artifacts — metadata tables, embedding vectors, UMAP coordinates, and the FAISS index — which are consumed directly by the cohort exploration UI. The cohort explorer application can reload or switch between datasets. The diagram is a screen capture of an Azure ML pipeline. It includes 5 pipeline components along with connecting arrows showing incoming and outgoing data, including the final outputs of the pipeline. Pipeline execution: time, cost, and what we learned Here's what a typical pipeline run looks like for a dataset of ~4,500 DICOM series: Component Task Approximate Time (CPU) Approximate Time (GPU) 0 - DICOM Metadata Extractor Scan files, extract headers 5-10 min 5-10 min 1 - Anatomy Classification Classify anatomy/orientation 90-120 min 5-10 min 2 - Text Template Generator Generate 5 templates per series 5-10 min 5-10 min 3 - FAISS Index Builder BiomedCLIP encoding + FAISS build 60-90 min 10-15 min 4 - Visualization Enrichment UMAP + color assignment 20-40 min 5-10 min Azure ML overhead Compute provisioning, env setup 5-10 min 5-10 min Total ~200-300 min ~30-50 min Key observations: Azure ML overhead is significant when doing quick iteration and testing. Compute provisioning, conda environment builds, and data mounting add several minutes before any component code runs. We first built each component as python code to run locally and debug before our first Azure ML run. This way we quickly iterated and avoided cost until we were ready. BiomedCLIP encoding dominates on CPU. Component 3 is the bottleneck. Moving to GPU compute for this component cuts encoding time roughly in half, but GPU clusters cost more. For a pipeline you run occasionally, CPU is fine. For frequent re-indexing, GPU pays for itself. Batch size tuning matters. The default BiomedCLIP batch size of 256 balances memory and throughput. On GPU, you can push to 512. On CPU with limited RAM, drop to 128. At Scale: 120,000 Images, CPU vs. GPU We ran the full pipeline against a larger dataset of ~120,000 images to understand how compute choice affects end-to-end time and cost: CPU Pipeline GPU Pipeline Pipeline compute time 4 days, 12 hours (108 hrs) 15 hours Pipeline compute cost ~$0.25/hr × 108 hrs = ~$27 ~$3.00/hr × 15 hrs = ~$45 MedImageInsight endpoint (MaaP on Standard_NC4as_T4_v3) ~$151 ~$21 Total estimated cost ~$178 ~$66 Both pipeline runs make the same ~120,000 classification calls to the MedImageInsight endpoint, but those calls are spread out over different time periods depending on how quickly and efficiently the pipeline can make the calls to MedImageInsight. The hourly cost for MedImageInsight on a Standard_NC4as_T4_v3 VM is ~$1.40/hr. Resulting in the estimated costs for MedImageInsight in the table above. GPU compute was roughly 7× faster at about 0.37× the total cost when endpoint costs are included. This was a key learning and clearly indicates the benefits of the more powerful compute resources. MedImageInsight can be deployed in two ways, depending on dataset size and operational needs. For smaller or infrequently processed datasets, we deploy MedImageInsight as a managed Azure ML online endpoint and invoke it from the pipeline. This keeps the pipeline simpler and avoids managing the MedImageInsight compute directly, while offering comparable performance at modest scale. For larger batch workloads, an alternative approach is to load MedImageInsight directly on the Azure ML pipeline’s GPU-backed compute. In this model, the pipeline handles both model loading and classification, eliminating per-request network round trips and the fixed cost of hosting a persistent endpoint. While this approach requires slightly longer pipeline run time, it becomes more cost‑effective at scale by avoiding endpoint overhead and improving throughput during bulk processing. Possible future enhancements Additional modalities: Extending the pipeline and classification to CT, X-ray, and ultrasound imaging, and build on the pattern for pathology images Image embeddings fusion: Combining MedImageInsight image embeddings with text embeddings for hybrid search Condition-aware search: Enabling queries about findings and conditions, not just imaging parameters The gap between a hackathon demo and a production system is where the real engineering happens. We hope sharing our journey helps others building similar systems. If you’re interested in partnering with us to work toward this goal or need access to the GitHub repo with the pipeline and UI code, contact authors through your Microsoft account team or reach out to Microsoft HLS AI frontier team The healthcare AI models in Microsoft Foundry are intended for research and model development exploration. The models are not designed or intended to be deployed in clinical settings as-is nor for use in the diagnosis or treatment of any health or medical condition, and the individual models' performances for such purposes have not been established. You bear sole responsibility and liability for any use of the healthcare AI models, including verification of outputs and incorporation into any product or service intended for a medical purpose or to inform clinical decision-making, compliance with applicable healthcare laws and regulations, and obtaining any necessary clearances or approvals.
Driving AI‑Powered Healthcare: A Data & AI Webinar and Workshop Series
Across these sessions, you’ll learn how healthcare organizations are using Microsoft Fabric, advanced analytics, and AI to unify fragmented data, modernize analytics, and enable intelligent, scalable solutions, from enterprise reporting to AI‑powered use cases. Whether you’re just getting started or looking to accelerate adoption, these sessions offer practical guidance, real‑world examples, and hands‑on learning to help you build a strong data foundation for AI in healthcare. Date Topic Details Location Registration Link May 6 Webinar: Microsoft Fabric Foundations - A Simple Path to Modern Analytics and AI Discover how Microsoft Fabric consolidates fragmented analytics into a single integrated data platform, making it easier to deliver trusted insights and adopt AI without added complexity. Virtual Register May 13 Webinar: Reduce BI Sprawl, Cut Cost and Build an AI-Ready Analytics Foundation Learn how Power BI enables enterprise BI consolidation, consistent metrics, and secure, scalable analytics that support both operational reporting and emerging AI use cases. Virtual Register May 19-20 In Person Workshop: Driving AI‑Powered Healthcare: Advanced Analytics, AI, and Real‑World Impact Attend this two‑day, in‑person event to learn how healthcare organizations use Microsoft Fabric to unify data, accelerate AI adoption, and deliver measurable clinical and operational value. Day 1 focuses on strategy, architecture, and real‑world healthcare use cases, while Day 2 offers hands‑on workshops to apply those concepts through guided labs and agent‑powered solutions. Chicago Register May 27 Webinar: Unified Data Foundation for AI & Analytics - Leveraging OneLake and Microsoft Fabric This session shows how organizations can simplify fragmented data architectures by using Microsoft Fabric and OneLake as a single, governed foundation for analytics and AI. Virtual Register June 3-4 In Person Workshop: Driving AI‑Powered Healthcare: Advanced Analytics, AI, and Real‑World Impact Attend this two‑day, in‑person event to learn how healthcare organizations use Microsoft Fabric to unify data, accelerate AI adoption, and deliver measurable clinical and operational value. Day 1 focuses on strategy, architecture, and real‑world healthcare use cases, while Day 2 offers hands‑on workshops to apply those concepts through guided labs and agent‑powered solutions. New York Register June 10 Webinar: From Data to Decisions: How AI Data Agents in Microsoft Fabric Redefine Analytics Join us to learn how Fabric Data Agents enable users to interact with enterprise data through AI‑powered, governed agents that understand both data and business context. Virtual Register June 23-24 In Person Workshop: Driving AI‑Powered Healthcare: Advanced Analytics, AI, and Real‑World Impact Attend this two‑day, in‑person event to learn how healthcare organizations use Microsoft Fabric to unify data, accelerate AI adoption, and deliver measurable clinical and operational value. Day 1 focuses on strategy, architecture, and real‑world healthcare use cases, while Day 2 offers hands‑on workshops to apply those concepts through guided labs and agent‑powered solutions. Dallas RegisterUshering in the Next Era of Cloud-Native AI Capabilities for Radiology
Introducing Dragon Copilot, your AI companion for PowerScribe One For radiologists, the reporting workflow of the future is here. At RSNA 2025, in Chicago, we’re showcasing Dragon Copilot, a cloud-native companion for PowerScribe One. Currently in preview, Dragon Copilot builds on the trusted capabilities of PowerScribe One to accelerate innovation and modernize reporting workflows while unlocking extensibility for radiology teams and partners. Why we built it: Technical drivers for a new era With growing demand for imaging services coupled with a workforce shortage, healthcare professionals face increased workloads and burnout while patients experience greater wait times. With our breadth of healthcare industry experience combined with our AI expertise and development at Microsoft, we immediately understood how we could help address these challenges. For radiologists, we sought to plugin into existing reporting workflows with rapid innovation, scalable AI, and open extensibility. How we built it: Modern architecture and extensibility By delivering Dragon Copilot as cloud-native solution built on Azure, we can enable new services globally. We apply the full capabilities of Azure for compute, storage, and security for high availability and compliance. Our modular architecture enables fast delivery of new features with APIs at the core to allow seamless integration, extensibility, and partner innovation. To imbue the workflow with AI through our platform, we harness the latest generative, multimodal, and agentic AI (both internal and through our partners) to support clinical reporting, workflow automation, and decision support. Key architectural highlights: AI services: Integrated large language models (LLMs) and vision-language models (VLMs) for multimodal data processing. API-first design: RESTful APIs expose core functions (draft report content generation, prior summarization, quality checks and chat) enabling partners and developers to build extensions and custom workflows. Extensibility framework: Open platform for 1st- and 3rd-party extensions, supporting everything from custom AI models to workflow agents. Inside the innovation Dragon Copilot alongside PowerScribe provides a unified AI experience. Radiologists can take advantage of the latest AI advancements without disruption to their workflows. They do not need another widget taking up room on their desktop. Instead, they need AI that fits seamlessly into existing workflows connecting their data to the cloud. Our cloud-first approach brings increased reliability, stability, and performance to a radiologists’ workflow. I’m thrilled to highlight the key capabilities of this dynamic duo: PowerScribe One with Dragon Copilot. Prior report summary: Automatically summarizes relevant prior reports, surfacing key findings, and context for the current study. AI-generated draft reports and quality checks: The most transformative aspect of Dragon Copilot is its open, extensible architecture for AI integration. We don’t limit radiology teams to a single set of AI tools. We enable seamless plug-ins for AI apps & agents from both Microsoft and our growing ecosystem of 3rd-parties. We provide a single surface for all your AI needs. This approach will enable radiology departments to discover, acquire, & deploy new AI-powered extensions. We’re enthusiastic about embarking on this journey with partners. We're also excited about collaborations with developers and academic innovators to bring their own AI models and services directly into the Dragon Copilot experience. Integrated chat experience with credible knowledge sources and medical safeguards: This chat interface connects radiologists to credible, clinically validated sources from Radiopedia and Radiology Assistant. It enables agentic orchestration and safeguards provided by Azure's Healthcare Agent Services for PHI and clinical accuracy. In the future, we expect to have a variety of other sources for radiology customers to choose from as well as the ability for organizations to add their own approved policies and protocols. This chat is designed to route questions to the right agent, provide evidence for claims, and filter responses for clinical validity. Over time, it will include extensions with custom agents powered by Copilot Studio. Help us shape what’s next As we continue to evolve Dragon Copilot alongside PowerScribe One, we invite innovators, developer partners, and academics to join us in shaping the future of radiology workflow. Dragon Copilot is more than a product; it’s a solution for rapid, responsible innovation in radiology. By combining cloud-native architecture, advanced AI capabilities, and open extensibility, we’re enabling radiology teams to work smarter, faster, and with greater confidence. Ready to see it in action? Visit us at RSNA 2025 (November 30–December 4), booth #1311 South Hall. Or contact our team to join the journey.Building Secure, Multi-User AI Workflows with the Responses API
With the recent GA (General Availability) of the Responses API, developers and enterprises now have access to a production-ready service purpose-built for stateful, multi-turn, tool-using AI agents. This milestone means you can confidently integrate the Responses API into real-world applications, knowing it’s fully supported, scalable, and designed for enterprise-grade use cases. Unlike traditional stateless APIs like Chat Completions, the Responses API maintains conversation history, supports tool orchestration, and enables multi-modal interactions. It’s ideal for building intelligent agents that need to remember context, call external tools, and interact with users over time. The Challenge: Securing AI Responses in Multi-User Environments As AI becomes more deeply embedded in enterprise apps, a new challenge emerges: response leakage. In multi-user environments, any user with a response ID could potentially access content they didn’t create—posing serious risks to privacy, data ownership, and compliance. By default, the Responses API allows retrieval of any response if you have the response ID. While this is convenient for prototyping, it’s not secure for production. There’s no built-in mechanism to verify who is making the request or whether they’re authorized to access that response. In this lab, I set out to solve that problem using Azure API Management (APIM). The goal? To ensure that only the user who created a response can retrieve or add to it, even if someone else has the response ID. This is especially important in scenarios where AI-generated content may include sensitive or proprietary information. The Problem: Response IDs Aren’t Enough The default behavior of the Responses API is simple: if you have a response ID, you can fetch the response. That’s convenient, but it’s also risky. There’s no built-in check to verify who is making the request. The Responses API is designed to be stateful, combining capabilities from chat completions and assistants into a unified experience. It’s powerful—but without additional safeguards, it can expose sensitive content to unintended users. This lab introduces a way to wrap the Responses API with APIM policies that enforce user-level access control. It’s a lightweight but powerful approach to securing AI-generated content. The Solution: APIM as a Gatekeeper Here’s how it works: A user sends a request to retrieve or update a response. APIM intercepts the request and extracts the user ID—either from the authentication token or, for testing purposes, from a custom header. APIM compares the user ID with the one associated with the response. If they match, the request proceeds. If not, it’s blocked. This ensures that only the original creator of a response can access or modify it. What’s in the Lab The lab in the AI Gateway repo includes: A sample API that mimics AI-generated responses. APIM policies that enforce user-level access. A test harness that lets you simulate requests with different user IDs. Header-based user ID injection for easier testing (ideal for labs and demos). This setup gives you a repeatable pattern for securing AI responses in production environments. Sample APIM Policy Snippet Here’s a simplified version of the APIM inbound policy that enforces user-level access: This policy checks the x-user-id header against the stored owner ID of the response. If they don’t match, the request is blocked with a 403 error. In a production scenario, you would want to use something other than just a userid in the header, for this I might suggest the userid from the authentication token. Why This Matters As AI becomes more embedded in our apps, we need to think beyond just securing the model—we need to secure the responses too. This lab shows how APIM can be used to: Enforce ownership of AI-generated content. Prevent unauthorized access to sensitive responses. Build trust into your AI workflows. Final Thoughts This lab is a great starting point for anyone building AI APIs in a multi-user environment. It’s simple, effective, and leverages tools you already know—like APIM. If you’re interested in extending this to token validation, role-based access, or integrating with Entra ID, let’s talk. I’d love to hear how you’re securing your AI stack.
Monitoring and Evaluating LLMs in Clinical Contexts with Azure AI Foundry
👀 Missed Session 02? Don’t worry—you can still catch up. But first, here’s what AI HLS Ignited is all about: What is AI HLS Ignited? AI HLS Ignited is a Microsoft-led technical series for healthcare innovators, solution architects, and AI engineers. Each session brings to life real-world AI solutions that are reshaping the Healthcare and Life Sciences (HLS) industry. Through live demos, architectural deep dives, and GitHub-hosted code, we equip you with the tools and knowledge to build with confidence. Session 02 Recap: In this session, we introduced MedEvals, an end-to-end evaluation framework for medical AI applications built on Azure AI Foundry. Inspired by Stanford’s MedHELM benchmark, MedEvals enables providers and payers to systematically validate performance, safety, and compliance of AI solutions across clinical decision support, documentation, patient communication, and more. 🧠 Why Scalable Evaluation Is Critical for Medical AI "Large language models (LLMs) hold promise for tasks ranging from clinical decision support to patient education. However, evaluating the performance of LLMs in medical contexts presents unique challenges due to the complex and critical nature of medical information." — Evaluating large language models in medical applications: a survey As AI systems become deeply embedded in healthcare workflows, the need for rigorous evaluation frameworks intensifies. Although large language models (LLMs) can augment tasks ranging from clinical documentation to decision support, their deployment in patient-facing settings demands systematic validation to guarantee safety, fidelity, and robustness. Benchmarks such as MedHELM address this requirement by subjecting models to a comprehensive battery of clinically derived tasks built on dataset (ground truth), enabling fine-grained, multi-metric performance assessment across the full spectrum of clinical use cases. However, shipping a medical LLM is only step one. Without a repeatable, metrics-driven evaluation loop, quality erodes, regulatory gaps widen, and patient safety is put at risk. This project accelerates your ability to operationalize trustworthy LLMs by delivering plug-and-play medical benchmarks, configurable evaluators, and CI/CD templates—so every model update triggers an automated, domain-specific “health check” that flags drift, surfaces bias, and validates clinical accuracy before it ever reaches production. 🚀 How to Get Started with MedEvals Kick off your MedEvals journey by following our curated labs. Newcomers to Azure AI Foundry can start with the foundational workflow; seasoned practitioners can dive into advanced evaluation pipelines and CI/CD integration. 🧪 Labs 🧪 Foundry Basics & Custom Evaluations: 🧾 Notebook Authenticate, initialize a Foundry project, run built-in metrics, and build custom evaluators with EvalAI and PromptEval. 🧪 Search & Retrieval Evaluations: 🧾 Notebook Prepare datasets, execute search metrics (precision, recall, NDCG), visualize results, and register evaluators in Foundry. 🧪 Repeatable Evaluations & CI/CD: 🧾 Notebook Define evaluation schemas, build deterministic pipelines with PyTest, and automate drift detection using GitHub Actions. 🏥 Use Cases 📝 Creating Your Clinical Evaluation with RevCycle Determinations Select a model and metric that best supports the determination behind the rationale made on AI-assisted prior authorizations based on real payor policy. This notebook use case includes: Selecting multiple candidate LLMs (e.g., gpt-4o, o1) Breaking down determinations both in deterministic results (approved vs rejected) and the supporting rationale and logic. Running evaluations across multiple dimensions Combining deterministic evaluators and LLM-as-a-Judge methods Evaluating the differential impacts of evaluators on the rationale across scenarios 🧾Get Started with the Notebook Why it matters: Enables data-driven metric selection for clinical workflows, ensures transparent benchmarking, and accelerates safe AI adoption in healthcare. 📝 Evaluating AI Medical Notes Summarization Applications Systematically assess how different foundation models and prompting strategies perform on clinical summarization tasks, following the MedHELM framework. This notebook use case includes: Preparing real-world datasets of clinical notes and summaries Benchmarking summarization quality using relevance, coherence, factuality, and harmfulness metrics Testing prompting techniques (zero-shot, few-shot, chain-of-thought prompting) Evaluating outputs using both automated metrics and human-in-the-loop scoring 🧾Get Started with the Notebook Why it matters: Ensures responsible deployment of AI applications for clinical summarization, guaranteeing high standards of quality, trustworthiness, and usability. 📣 Join Us for the Next Session Help shape the future of healthcare by sharing AI HLS Ignited with your network—and don’t miss what’s coming next! 📅 Register for the upcoming session → AI HLS Ignited Event Page 💻 Explore the code, demos, and architecture → AI HLS Ignited GitHub Repository
Orchestrate multimodal AI insights within your healthcare data estate (Public Preview)
In today’s healthcare landscape, there is an increasing emphasis on leveraging artificial intelligence (AI) to extract meaningful insights from diverse datasets to improve patient care and drive clinical research. However, incorporating AI into your healthcare data estate often brings significant costs and challenges, especially when dealing with siloed and unstructured data. Healthcare organizations produce and consume data that is not only vast but also varied in format—ranging from structured EHR entries to unstructured clinical notes and imaging data. Traditional methods require manual effort to prepare and harmonize this data for AI, specify the AI output format, set up API calls, store the AI outputs, integrate the AI outputs, and analyze the AI outputs for each AI model or service you decide to use. Orchestrate multimodal AI insights is designed to streamline and scale healthcare AI within your data estate by building off of the data transformations in healthcare data solutions in Microsoft Fabric. This capability provides a framework to generate AI insights by connecting your multimodal healthcare data to an ecosystem of AI services and models and integrating structured AI-generated insights back into your data estate. When you combine these AI-generated insights with the existing healthcare data in your data estate, you can power advanced analytics scenarios for your organization and patient population. Key features: Metadata store lakehouse acts as a central repository for the metadata for AI orchestration to effectively capture and manage enrichment definitions, view definitions, and contextual information for traceability purposes. Execution notebooks define the enrichment view and enrichment definition based on the model configuration and input mappings. They also specify the model processor and transformer. The model processor calls the model API, and the transformer produces the standardized output while saving the output in the bronze lakehouse in the Ingest folder. Transformation pipeline to ingest AI-generated insights through the healthcare data solutions medallion lakehouse layers and persist the insights in an enrichment store within the silver layer. Conceptual architecture: The data transformations in healthcare data solutions in Microsoft Fabric allow you ingest, store, and analyze multimodal data. With the orchestrate multimodal AI insights capability, this standardized data serves as the input for healthcare AI models. The model results are stored in a standardized format and provide new insights from your data. The diagram below shows the flow of integrating AI generated insights into the data estate, starting as raw data in the bronze lakehouse and being transformed to delta tables in the silver lakehouse. This capability simplifies AI integration across modalities for data-driven research and care, currently supporting: Text Analytics for health in Azure AI Language to extract medical entities such as conditions and medications from unstructured clinical notes. This utilizes the data in the DocumentReference FHIR resource. MedImageInsight healthcare AI model in Azure AI Foundry to generate medical image embeddings from imaging data. This model leverages the data in the ImagingStudy FHIR resource. MedImageParse healthcare AI model in Azure AI Foundry to enable segmentation, detection, and recognition from imaging data across numerous object types and imaging modalities. This model uses the data in the ImagingStudy FHIR resource. By using orchestrate multimodal AI insights to leverage the data in healthcare data solutions for these models and integrate the results into the data estate, you can analyze your existing data alongside AI enrichments. This allows you to explore use cases such as creating image segmentations and combining with your existing imaging metadata and clinical data to enable quick insights and disease progression trends for clinical research at the patient level. Get started today! This capability is now available in public preview, and you can use the in-product sample data to test this feature with any of the three models listed above. For more information and to learn how to deploy the capability, please refer to the product documentation. We will dive deeper into more detailed aspects of the capability, such as the enrichment store and custom AI use cases, in upcoming blogs. Medical device disclaimer: Microsoft products and services (1) are not designed, intended or made available as a medical device, and (2) are not designed or intended to be a substitute for professional medical advice, diagnosis, treatment, or judgment and should not be used to replace or as a substitute for professional medical advice, diagnosis, treatment, or judgment. Customers/partners are responsible for ensuring solutions comply with applicable laws and regulations. FHIR® is the registered trademark of HL7 and is used with permission of HL7.Reshape Business Processes
* To see all 4 pillars and links to the associated blog posts see AI Transformation with Microsoft 365 Copilot | Microsoft Community Hub Below is the list of Healthcare & Lifesciences Blog posts focused on Reshaping Business Processes: AGS Health: Call audits simplified Apollo Hospitals: Transforming clinical documentation audits Azure AI Foundry: Your AI App and agent factory | Microsoft Azure Blog Empower clinicians with trusted content and GenAI: MSD Manuals added to healthcare agent service Enhancing Genomics Annotation with GraphRAG How generative AI is reshaping business applications - Microsoft Dynamics 365 Blog SolutionHealth builds patient-focused workflows with Dragon for two health systems | Microsoft Customer Stories Syneos Health reduces time for clinical trial site activation by about 10% with Azure OpenAI Service | Microsoft Customer StoriesBuilding AI-Powered Clinical Knowledge Stores with Azure AI Search
👀 Missed Session 01? Don’t worry—you can still catch up. But first, here’s what AI HLS Ignited is all about: What is AI HLS Ignited? AI HLS Ignited is a Microsoft-led technical series for healthcare innovators, solution architects, and AI engineers. Each session brings to life real-world AI solutions that are reshaping the Healthcare and Life Sciences (HLS) industry. Through live demos, architectural deep dives, and GitHub-hosted code, we equip you with the tools and knowledge to build with confidence. Session 01 Recap: In our first session, we introduced the accelerator MedIndexer - which is an indexing framework designed for the automated creation of structured knowledge bases from unstructured clinical sources. Whether you're dealing with X-rays, clinical notes, or scanned documents, MedIndexer converts these inputs into a schema-driven format optimized for Azure AI Search. This will allow your applications to leverage state-of-the-art retrieval methodologies, including vector search and re-ranking. Moreover, by applying a well-defined schema and vectorizing the data into high-dimensional representations, MedIndexer empowers AI applications to retrieve more precise and context-aware information... The result? AI systems that surface more relevant, accurate, and context-aware insights—faster. 🔍 Turning Your Unstructured Data into Value "About 80% of medical data remains unstructured and untapped after it is created (e.g., text, image, signal, etc.)" — Healthcare Informatics Research, Chungnam National University In the era of AI, the rise of AI copilots and assistants has led to a shift in how we access knowledge. But retrieving clinical data that lives in disparate formats is no trivial task. Building retrieval systems takes effort—and how you structure your knowledge store matters. It’s a cyclic, iterative, and constantly evolving process. That’s why we believe in leveraging enterprise-ready retrieval platforms like Azure AI Search—designed to power intelligent search experiences across structured and unstructured data. It serves as the foundation for building advanced retrieval systems in healthcare. However, implementing Azure AI Search alone is not enough. Mastering its capabilities and applying well-defined patterns can significantly enhance your ability to address repetitive tasks and complex retrieval scenarios. This project aims to accelerate your ability to transform raw clinical data into high-fidelity, high-value knowledge structures that can power your next-generation AI healthcare applications. 🚀 How to Get Started with MedIndexer New to Azure AI Search? Begin with our guided labs to build a strong foundation and get hands-on with the core capabilities. Already familiar with the tech? Jump ahead to the real-world use cases—learn how to build Coded Policy Knowledge Stores and X-ray Knowledge Stores. 🧪 Labs 🧪 Building Your Azure AI Search Index: 🧾 Notebook - Building your first Index Learn how to create and configure an Azure AI Search index to enable intelligent search capabilities for your applications. 🧪 Indexing Data into Azure AI Search: 🧾 Notebook - Ingest and Index Clinical Data Understand how to ingest, preprocess, and index clinical data into Azure AI Search using schema-first principles. 🧪 Retrieval Methods for Azure AI Search: 🧾 Notebook - Exploring Vector Search and Hybrid Retrieval Dive into retrieval techniques such as vector search, hybrid retrieval, and reranking to enhance the accuracy and relevance of search results. 🧪 Evaluation Methods for Azure AI Search: 🧾 Notebook - Evaluating Search Quality and Relevance Learn how to evaluate the performance of your search index using relevance metrics and ground truth datasets to ensure high-quality search results. 🏥 Use Cases 📝 Creating Coded Policy Knowledge Stores In many healthcare systems, policy documents such as pre-authorization guidelines are still trapped in static, scanned PDFs. These documents are critical—they contain ICD codes, drug name coverage, and payer-specific logic—but are rarely structured or accessible in real-time. To solve this, we built a pipeline that transforms these documents into intelligent, searchable knowledge stores. This diagram shows how pre-auth policy PDFs are ingested via blob storage, passed through an OCR and embedding skillset, and then indexed into Azure AI Search. The result: fast access to coded policy data for AI apps. 🧾 Notebook - Creating Coded Policies Knowledge Stores Transform payer policies into machine-readable formats. This use case includes: Preprocessing and cleaning PDF documents Building custom OCR skills Leveraging out-of-the-box Indexer capabilities and embedding skills Enabling real-time AI-assisted querying for ICDs, payer names, drug names, and policy logic Why it matters: This streamlines prior authorization and coding workflows for providers and payors, reducing manual effort and increasing transparency. 🩻 Creating X-ray Knowledge Stores In radiology workflows, X-ray reports and image metadata contain valuable clinical insights—but these are often underutilized. Traditionally, they’re stored as static entries in PACS systems or loosely connected databases. The goal of this use case is to turn those X-ray reports into a searchable, intelligent asset that clinicians can explore and interact with in meaningful ways. This diagram illustrates a full retrieval pipeline where radiology reports are uploaded, enriched through foundational models, embedded, and indexed. The output powers an AI-driven web app for similarity search and decision support. 🧾 Notebook - Creating X-rays Knowledge Stores Turn imaging reports and metadata into a searchable knowledge base. This includes: Leveraging push APIs with custom event-driven indexing pipeline triggered on new X-ray uploads Generating embeddings using Microsoft Healthcare foundation models Providing an AI-powered front-end for X-ray similarity search Why it matters: Supports clinical decision-making by retrieving similar past cases, aiding diagnosis and treatment planning with contextual relevance. 📣 Join Us for the Next Session Help shape the future of healthcare by sharing AI HLS Ignited with your network—and don’t miss what’s coming next! 📅 Register for the upcoming session → AI HLS Ignited Event Page 💻 Explore the code, demos, and architecture → AI HLS Ignited GitHub Repository
