Enter new era of enterprise communication with Microsoft Translator Pro & document image translation
Microsoft Translator Pro: standalone, native mobile experience We are thrilled to unveil the gated public preview of Microsoft Translator Pro, our robust solution designed for enterprises seeking to dismantle language barriers in the workplace. Available on iOS, Microsoft Translator Pro offers a standalone, native experience, enabling speech-to-speech translated conversations among coworkers, users, or clients within your enterprise ecosystem. Watch how Microsoft Translator Pro transforms a hotel check-in experience by breaking down language barriers. In this video, a hotel receptionist speaks in English, and the app translates and plays the message aloud in Chinese for the traveler. The traveler responds in Chinese, and the app translates and plays the message aloud in English for the receptionist. Key features of the public preview Our enterprise version of the app is packed with features tailored to meet the stringent demands of enterprises: Core feature - speech-to-speech translation: Break language barriers: Real-time speech-to-speech translation allows you to have seamless communication with individuals speaking different languages. Unified experience: View or hear both transcription and translation simultaneously on a single device, ensuring smooth and efficient conversations. On-device translation: Harness the app's speech-to-speech translation capability without an internet connection in limited languages, ensuring your productivity remains unhampered. Full administrator control: Enterprise IT Administrators wield extensive control over the app's deployment and usage within your organization. They can fine-tune settings to manage conversation history, audit, and diagnostic logs, with the ability to disable history or configure automatic exportation of the history to cloud storage. Uncompromised privacy and security: Microsoft Translator Pro provides enterprises with a high level of translation quality and robust security. We know that Privacy and security are top priorities for you. Once granted access by your organization's admin, you can sign in the app with your organizational credentials. Your conversational data remains strictly yours, safeguarded within your Azure tenant. Neither Microsoft nor any external entities have access to your data. Join the Preview To embark on this journey with us, please complete the gating form . Upon meeting the criteria, we will grant your organization access to the paid version of the Microsoft Translator Pro app, which is now available in the US. Learn more and get started: Microsoft Translator Pro documentation. Document translation translates text embedded in images Our commitment to advancing cross-language communication takes a major step forward with a new enhancement in Azure AI Translator’s Document Translation (DT) feature. Previously, Document Translation supported fully digital documents and scanned PDFs. Starting January 2025, with this latest update, the service can also process mixed-content documents, translating both digital text and text embedded within images. Sample document translated from English to Spanish: (Frames in order: Source document, translated output document (image not translated), translated output document with image translation) How It Works To enable this feature, the Document Translation service now leverages Microsoft Azure AI Vision API to detect, extract, and translate text from images within documents. This capability is especially useful for scenarios where documents contain a mix of digital text and image-based text, ensuring complete translations without manual intervention. Getting Started To take advantage of this feature, customers can use the new optional parameter when setting up a translation request: Request A new parameter under "options" called "translateTextWithinImage" has been introduced. This parameter is of type Boolean, accepting "true" or "false." The default value is "false," so you’ll need to set it to "true" to activate the image text translation capability. Response: When this feature is enabled, the response will include additional details for transparency on image processing: totalImageScansSucceeded: The count of successfully translated image scans. totalImageScansFailed: The count of image scans that encountered processing issues. Usage and cost For this feature, customers will need to use the Azure AI Services resource, as this new feature leverages Azure AI Vision services along with Azure AI Translator. The OCR service incurs additional charges based on usage. Pricing details for the OCR service can be found here: Pricing details Learn more and get started (starting January 2025): Translator Documentation These new advancements reflect our dedication to pushing boundaries in Document Translation, empowering enterprises to connect and collaborate more effectively, regardless of language. Stay tuned for more innovations as we continue to expand the reach and capabilities of Microsoft Azure AI Translator.Ground your AI agents with knowledge from Bing Search, Microsoft Fabric, SharePoint and more
Today, we are thrilled to announce the upcoming preview of Azure AI Agent Service, a comprehensive suite of capabilities designed to empower developers to securely build, deploy, and scale high-quality, extensible, and reliable AI agents. By leveraging an extensive ecosystem of models, tools, and capabilities from OpenAI, Microsoft, and industry-leading partners such as Meta, Azure AI Agent Service enables developers to efficiently create agents for a wide range of generative AI use cases. In this blog, we will explore the Knowledge Integration capabilities of Azure AI Agent Service, designed not only to streamline the creation of Retrieval-Augmented Generation (RAG) workflow, but also to empower developers to build intelligent, knowledge-driven AI agents. Grounding AI Agents with Knowledge Knowledge is the foundation of generating accurate, grounded responses, allowing Azure AI Agent Service to make informed decisions with confidence. By integrating comprehensive and accurate data, Azure AI Agent Service enhances precision and provides effective solutions, elevating the overall customer experience. With the preview of Azure AI Agent Service, you can ground your agent’s responses using data from Bing Search, Microsoft Fabric, SharePoint, Azure AI Search, Azure Blob Storage, your local files, and even your own licensed data. These data sources enable grounding with diverse data types, from enterprise private data and public web data to your own licensed data, structured or unstructured. Enterprise-grade security features, such as On-Behalf-Of (OBO) authorization, ensures your data is stored, retrieved and accessed, meeting the highest standards of privacy and protection. Key Capabilities Leverage Real-Time Public Web Data with Grounding with Bing Search LLMs can sometimes generate outdated content. By grounding your agent with Bing Search, you can overcome this limitation and create more reliable and trustworthy applications. Grounding with Bing Search allows your agents to integrate real-time public web data, ensuring their response is accurate and up to date. By including supporting URLs and search query links, Grounding with Bing Search enhances trust and transparency, empowering the users to verify responses with the original sources. Empower Data-Driven Decisions with Microsoft Fabric Integrate your Azure AI Agent with Fabric AI Skill to unlock powerful data analysis capabilities. Fabric AI Skill transforms enterprise data into conversational Q&A systems, allowing users to interact with the data through chat and uncover data-driven and actionable insights effortlessly. With OBO authorization, this integration simplifies access to enterprise data in Fabric while maintaining robust security, ensuring proper access control and enterprise-grade protection. Connect Private Data Securely with SharePoint Azure AI Agent Service supports grounding response with your data in SharePoint (coming soon). This integration makes your SharePoint content more accessible to your end users. Enterprise-grade security features, such as OBO authorization for SharePoint, ensure secure and controlled access for end users. Ground Private Data with Azure AI Search, Azure Blob Storage and Your Local Files Azure AI Agent Service supports connecting private data from various sources, such as Azure AI Search, Azure Blob Storage, and local files, to enhance responses. Bring your existing Azure AI Search index or create a new one using the improved File Search tool. This tool leverages a built-in ingestion pipeline to process files from your local system or Azure Blob Storage. With the new File Search tool, your files remain in your own storage, and your Azure AI Search resource is used to ingest them, ensuring you maintain complete control over your data. Enrich Responses with Your Licensed Data Azure AI Agent Service also integrates your own licensed data from specialized data providers, such as Tripadvisor. Enhance the quality of your agent’s responses with high-quality, fresh data, such as travel guidance and reviews. These insights empower your agents to deliver nuanced, informed solutions tailored to specific use cases. “We’re excited to partner with Microsoft as the first data and intelligence provider for its Azure AI Agent Service," said Rahul Todkar, Vice President, Head of Data and AI at Tripadvisor. “At Tripadvisor, we are focused on leveraging the power of Data and Generative AI to benefit all travelers and partners across the globe. With this new partnership we are making available to developers a set of APIs that provide access to granular Tripadvisor data, content and intelligence. This will allow developers and AI engineers/scientists to use our robust travel data across a broad array of AI and ML use cases including building AI agents, contextually relevant recommendations and drive increased personalization." What’s Next Sign up for the private preview: Contact your account executive Learn More: Watch our breakout session on Azure AI Agent Service. See it in action: Check out our demo session on building custom agents with models and tools. Start building: Explore single and multi-agent solutions with our Azure AI Agent Service code samples.Introducing Azure AI Agent Service
Introducing Azure AI Agent Service at Microsoft Ignite 2024 Discover how Azure AI Agent Service is revolutionizing the development and deployment of AI agents. This service empowers developers to build, deploy, and scale high-quality AI agents tailored to business needs within hours. With features like rapid development, extensive data connections, flexible model selection, and enterprise-grade security, Azure AI Agent Service sets a new standard in AI automationDiscover the Azure AI Training Profiler: Transforming Large-Scale AI Jobs
Meet the AI Training Profiler Large-scale AI training can be complicated, especially in distributed environments like healthcare, finance, and e-commerce, where the need for accuracy, speed, and massive data processing is crucial. Efficiently managing hardware resources, ensuring smooth parallelism, and minimizing bottlenecks are crucial for optimal performance. The AI Training Profiler powered by PyTorch Profiler inAzure Machine Learning is here to help! By giving you detailed visibility into hardware and software metrics, this tool helps you spot inefficiencies, make the best use of resources, and scale your training workflows like a pro. Why Choose the AI Training Profiler? Running large AI training jobs on distributed infrastructure is inherently complex, and inefficiencies can quickly escalate into increased costs and delays in deploying models. The AI Training Profiler addresses these issues by providing a comprehensive breakdown of compute resource usage throughout the training lifecycle. This enables users to fine-tune and streamline their AI workflows, yielding several key benefits: Improved Performance: Identify bottlenecks and inefficiencies, such as slow data loading or underutilized GPUs, to enhance training throughput. Reduced Costs: Detect idle or underused resources, thereby minimizing compute time and hardware expenses. Faster Debugging: Leverage real-time monitoring and intuitive visualizations to troubleshoot performance issues swiftly. Key Features of the AI Training Profiler GPU Core and Tensor Core Utilization The profiler meticulously tracks GPU kernel execution, reporting utilization metrics such as time spent on forward and backward passes, tensor core operations, and other computation-heavy tasks. This detailed breakdown enables users to pinpoint under-utilized resources and optimize kernel execution patterns. Memory Profiling Memory Allocation and Peak Usage: Monitors GPU memory usage throughout the training process, offering insights into underutilized or over-allocated memory. CUDA Memory Footprint: Visualizes memory consumption during forward/backward propagation and optimizer steps to identify bottlenecks or fragmentation. Page Fault and Out-of-Memory Events: Detects critical events that could slow training or cause job failures due to insufficient memory allocation. Kernel Execution Metrics Kernel Execution Time: Provides per-kernel timing, breaking down execution into compute-bound and memory-bound operations, allowing users to discern whether performance bottlenecks stem from inefficient kernel launches or memory access patterns. Instruction-level Performance: Measures IPC (Instructions Per Cycle) to understand kernel-level performance and identify inefficient operations. Distributed Training Communication Primitives: Captures inter-GPU and inter-node communication patterns, focusing on the performance of primitives like AllReduce, AllGather, and Broadcast in multi-GPU training. This helps users identify communication bottlenecks such as imbalanced data distribution or excessive communication overhead. Synchronization Events: Measures the time spent on synchronization barriers between GPUs, highlighting where parallel execution is slowed by synchronization. Getting Started with the Profiling Process Using the AI Training Profiler is a breeze! Activate it when you launch a job, either through the CLI or our platform’s user-friendly interface. Here are the three environment variables you need to set: Enable/Disable the Profiler: ENABLE_AZUREML_TRAINING_PROFILER: 'true' Configure Trace Capture Duration: AZUREML_PROFILER_RUN_DURATION_MILLISECOND: '50000' Delay the Start of Trace Capturing: AZUREML_PROFILER_WAIT_DURATION_SECOND: '1200' Once your training job is running, the profiler collects metrics and stores them centrally. After the run, this data is analyzed to give you visual insights into critical metrics like kernel execution times. Use Cases The AI Training Profiler is a game-changer for fine-tuning large language models and other extensive architectures. By ensuring efficient GPU utilization and minimizing distributed training costs, this tool helps organizations get the most out of their infrastructure, whether they're working on cutting-edge models or refining existing workflows. In conclusion, the AI Training Profiler is a must-have for teams running large-scale AI training jobs. It offers the visibility and control needed to optimize resource utilization, reduce costs, and accelerate time to results. Embrace the future of AI training optimization with the AI Training Profiler and unlock the full potential of your AI endeavors. How to Get Started? The feature is available as a preview, you can just set up the environment variables and start using the profiler! Stay tuned for future repository with many samples that you can use as well!Announcing management center and other tools to secure and govern Azure AI Foundry
We’re pleased to share new security and IT governance capabilities in Azure AI Foundry that can help organizations build and scale GenAI solutions that are secure by default, including a new management center, granular networking controls, and the general availability of data and service connections.The Evolution of AI Frameworks: Understanding Microsoft's Latest Multi-Agent Systems
The landscape of artificial intelligence is undergoing a fundamental transformation in late 2024. Microsoft has unveiled three groundbreaking frameworks—AutoGen 0.4, Magentic-One, and TinyTroupe—that are revolutionizing how we approach AI development. Moving beyond single-model systems, these frameworks represent a shift toward collaborative AI, where multiple specialized agents work together to solve complex problems. Think of these frameworks as different but complementary systems, much like how a city needs infrastructure, service providers, and community organizations to function effectively. AutoGen 0.4 provides the robust foundation, Magentic-One orchestrates complex tasks through specialized agents, and TinyTroupe simulates human behavior for business insights. Together, they form a comprehensive ecosystem for building the next generation of intelligent systems. As we explore each framework in detail, we'll see how this coordinated approach is opening new possibilities in AI development, from enterprise-scale applications to sophisticated business simulations. Framework Comparison: A Deep Dive Before we explore each framework in detail, let's understand how they compare across key dimensions. These comparisons will help us understand where each framework excels and how they complement each other. Core Capabilities and Design Focus Aspect AutoGen 0.4 Magentic-One TinyTroupe Primary Architecture Layered & Event-driven Orchestrator-based Persona-based Core Strength Infrastructure & Scalability Task Orchestration Human Simulation Development Stage Beta Preview Early Release Target Users Enterprise Developers Automation Teams Business Analysts Key Innovation Cross-language Support Dual-loop Orchestration Persona Modeling Deployment Model Cloud/On-premise Container-based Local Main Use Case Enterprise Systems Task Automation Business Insights AutoGen 0.4: The Digital Infrastructure Builder Imagine building a modern city. Before any services can operate, you need robust infrastructure – roads, power grids, water systems, and communication networks. AutoGen 0.4 serves a similar foundational role in the AI ecosystem. It provides the essential infrastructure that allows Agentic systems to operate at enterprise scale. The framework's brilliance lies in its three-layer architecture: The Core Layer acts as the fundamental infrastructure, handling basic communication and resource management, much like a city's utility systems. The AgentChat Layer provides high-level interaction capabilities, similar to how city services interface with residents. The Extensions Layer enables specialized functionalities, comparable to how cities can add new services based on specific needs. What truly sets AutoGen 0.4 apart is its understanding of real-world enterprise needs. Modern organizations rarely operate with a single technology stack – they might use Python for data science, .NET for backend services, and other languages for specific needs. AutoGen 0.4 embraces this reality through its multi-language support, ensuring different components can communicate effectively while maintaining strict type safety to prevent errors. from autogen_agentchat.agents import AssistantAgent from autogen_agentchat.task import Console from autogen_ext.models import OpenAIChatCompletionClient async def enterprise_example(): # Create an enterprise agent with specific configuration agent = AssistantAgent( name="enterprise_system", model_client=OpenAIChatCompletionClient( model="gpt-4o-2024-08-06", api_key="YOUR_API_KEY" ) ) # Define a complex enterprise task task = { "objective": "Analyze sales data and generate insights", "data_source": "sales_database", "output_format": "report" } # Execute task with streaming output stream = agent.run_stream(task=task) await Console(stream) # Example usage: # asyncio.run(enterprise_example()) Magentic-One: The Master Orchestra Conductor If AutoGen 0.4 builds the city's infrastructure, Magentic-One acts as its management system. Think of it as a highly skilled orchestra conductor, coordinating various musicians (specialized agents) to create a harmonious performance (completed tasks). The framework's innovative dual-loop architecture demonstrates this orchestration: The Task Ledger works like a conductor's score, planning out what needs to be done. The Progress Ledger functions as the conductor's real-time monitoring, ensuring each section performs its part correctly. Magentic-One's specialized agents exemplify this orchestra metaphor: WebSurfer: Like the string section, handling intricate web interactions FileSurfer: Similar to the percussion section, managing rhythmic file operations Coder: Comparable to the brass section, producing powerful code outputs ComputerTerminal: Like the woodwinds, executing precise commands This specialization has proven its worth through impressive benchmark performances across GAIA, AssistantBench, and WebArena, showing that specialized expertise, when properly coordinated, produces superior results. from magentic_one import ( Orchestrator, WebSurfer, FileSurfer, Coder, ComputerTerminal ) def automation_example(): # Initialize specialized agents agents = { 'web': WebSurfer(), 'file': FileSurfer(), 'code': Coder(), 'terminal': ComputerTerminal() } # Create orchestrator with task and progress ledgers orchestrator = Orchestrator(agents) # Define complex automation task task = { "type": "web_automation", "steps": [ {"action": "browse", "url": "example.com"}, {"action": "extract", "data": "pricing_info"}, {"action": "save", "format": "csv"} ] } # Execute orchestrated task result = orchestrator.execute_task(task) return result # Example usage: # result = automation_example() TinyTroupe: The Social Behavior Laboratory TinyTroupe takes a fundamentally different approach, more akin to a sophisticated social simulation laboratory than a traditional AI framework. Instead of focusing on task completion, it seeks to understand and replicate human behavior, much like how social scientists study human interactions and decision-making. The framework creates detailed artificial personas (TinyPersons) with rich backgrounds, personalities, and behaviors. Think of it as creating a miniature society where researchers can observe how different personality types interact with products, services, or each other. These personas exist within controlled environments (TinyWorlds), allowing for systematic observation and analysis. Consider a real-world parallel: When automotive companies design new vehicles, they often create detailed driver personas to understand different user needs. TinyTroupe automates and scales this approach, allowing businesses to simulate thousands of interactions with different personality types, providing insights that would be impractical or impossible to gather through traditional focus groups. The beauty of TinyTroupe lies in its ability to capture the nuances of human behavior. Just as no two people are exactly alike, each TinyPerson brings its unique perspective, shaped by its programmed background, experiences, and preferences. This diversity enables more realistic and valuable insights for business decision-making. from tinytroupe import TinyPerson, TinyWorld, TinyPersonFactory from tinytroupe.utils import ResultsExtractor def simulation_example(): # Create simulation environment world = TinyWorld("E-commerce Platform") # Generate diverse personas factory = TinyPersonFactory() personas = [ factory.generate_person( "Create a tech-savvy professional who values efficiency" ), factory.generate_person( "Create a budget-conscious parent who prioritizes safety" ), factory.generate_person( "Create a senior citizen who prefers simplicity" ) ] # Add personas to simulation world for persona in personas: world.add_person(persona) # Define simulation scenario scenario = { "type": "product_evaluation", "product": "Smart Home Device", "interaction_points": ["discovery", "purchase", "setup"] } # Run simulation and extract insights results = world.run_simulation(scenario) insights = ResultsExtractor().analyze(results) return insights # Example usage: # insights = simulation_example() Framework Selection Guide To help you make an informed decision, here's a comprehensive selection matrix based on specific needs: Need Best Choice Reason Alternative Enterprise Scale AutoGen 0.4 Built for distributed systems Magentic-One Task Automation Magentic-One Specialized agents AutoGen 0.4 User Research TinyTroupe Persona simulation None High Performance AutoGen 0.4 Optimized architecture Magentic-One Quick Deployment TinyTroupe Minimal setup Magentic-One Complex Workflows Magentic-One Strong orchestration AutoGen 0.4 Practical Implications For organizations looking to implement these frameworks, consider the following guidance: For Enterprise Applications: Use AutoGen 0.4 as your foundation. Its robust infrastructure and cross-language support make it ideal for building scalable, production-ready systems. For Complex Automation: Implement Magentic-One for tasks requiring sophisticated orchestration. Its specialized agents and safety features make it perfect for automated workflows. For Business Intelligence: Deploy TinyTroupe for market research and user behavior analysis. Its unique simulation capabilities provide valuable insights for business decision-making. Conclusion Microsoft's three-pronged approach to multi-agent AI systems represents a significant leap forward in artificial intelligence. By addressing different aspects of the AI development landscape – infrastructure (AutoGen 0.4), task execution (Magentic-One), and human simulation (TinyTroupe) – these frameworks provide a comprehensive toolkit for building the next generation of AI applications. As these frameworks continue to evolve, we can expect to see even more sophisticated capabilities and tighter integration between them. Organizations that understand and leverage the strengths of each framework will be well-positioned to build powerful, scalable, and intelligent systems that drive real business value. Appendix Technical Implementation Details Feature AutoGen 0.4 Magentic-One TinyTroupe Language Support Python, .NET Python Python State Management Distributed Centralized Environment-based Message Passing Async Event-driven Task-based Simulation-based Error Handling Comprehensive Task-specific Simulation-bound Monitoring Enterprise-grade Task-focused Analysis-oriented Extensibility High Medium Framework-bound Performance and Scalability Metrics Metric AutoGen 0.4 Magentic-One TinyTroupe Response Time Milliseconds Seconds Variable Concurrent Users Thousands Hundreds Dozens Resource Usage Optimized Task-dependent Simulation-dependent Horizontal Scaling Yes Limited No State Persistence Distributed Cache Container Storage Local Files Recovery Capabilities Advanced Basic Manual Security and Safety Features Security Aspect AutoGen 0.4 Magentic-One TinyTroupe Access Control Role-based Container-based Environment-based Content Filtering Enterprise-grade Active Monitoring Simulation Bounds Audit Logging Comprehensive Action-based Simulation Logs Isolation Level Service Container Process Risk Assessment Dynamic Pre-execution Scenario-based Recovery Options Automated Semi-automated Manual Integration and Ecosystem Support Integration Type AutoGen 0.4 Magentic-One TinyTroupe API Support REST, gRPC REST Python API External Services Extensive Web-focused Limited Database Support Multiple Basic Simulation Only Cloud Services Full Support Container Services Local Only Custom Extensions Yes Limited Framework-bound Third-party Tools Wide Support Moderate MinimalAnnouncing Model Fine-Tuning Collaborations: Weights & Biases, Scale AI, Gretel and Statsig
As AI continues to transform industries, the ability to fine-tune models and customize them for specific use cases has become more critical than ever. Fine-tuning can enable companies to align models with their unique business goals, ensuring that AI solutions deliver results with greater precision However, organizations face several hurdles in their model customization journey: Lack of end-to-end tooling: Organizations struggle with fine-tuning foundation models due to complex processes, and the absence of tracking and evaluation tools for modifications. Data scarcity and quality: Limited access to large, high-quality datasets, along with privacy issues and high costs, complicate model training and fine-tuning. Shortage of fine-tuning expertise and pre-trained models: Many companies lack specialized knowledge and access to refined models for fine-tuning. Insufficient experimentation tools: A lack of tools for ongoing experimentation in production limits optimization of key variables like model diversity and operational efficiency. To address these challenges, Azure AI Foundry is pleased to announce new collaborations with Weights & Biases, Scale AI, Gretel and Statsig to streamline the process of model fine-tuning and experimentation through advanced tools, synthetic data and specialized expertise. Weights & Biases integration with Azure OpenAI Service: Making end-to-end fine-tuning accessible with tooling The integration of Weights & Biases with Azure OpenAI Service offers a comprehensive end-to-end solution for enterprises aiming to fine-tune foundation models such as GPT-4, GPT-4o, and GPT-4o mini. This collaboration provides a seamless connection between Azure OpenAI Service and Weights and Biases Models which offers powerful capabilities for experiment tracking, visualization, model management, and collaboration. With the integration, users can also utilize Weights and Biases Weave to evaluate, monitor, and iterate on the performance of their fine-tuned models powered AI applications in real-time. Azure's scalable infrastructure allows organizations to handle the computational demands of fine-tuning, while Weights and Biases offers robust capabilities for fine-tuning experimentation and evaluation of LLM-powered applications. Whether optimizing GPT-4o for complex reasoning tasks or using the lightweight GPT-4o mini for real-time applications, the integration simplifies the customization of models to meet enterprise-specific needs. This collaboration addresses the growing demand for tailored AI models in industries such as retail and finance, where fine-tuning can significantly improve customer service chatbots or complex financial analysis. Azure Open AI Service and Weights & Biases integration is now available in public preview. For further details on Azure OpenAI Service and Weights & Biases integration including real-world use-cases and a demo, refer to the blog here. Scale AI and Azure Collaboration: Confidently Implement Agentic GenAI Solutions in Production Scale AI collaborates with Azure AI Foundry to offer advanced fine-tuning and model customization for enterprise use cases. It enhances the performance of Azure AI Foundry models by providing high-quality data transformation, fine-tuning and customization services, end-to-end solution development and specialized Generative AI expertise. This collaboration helps improve the performance of AI-driven applications and Azure AI services such as Azure AI Agent in Azure AI Foundry, while reducing production time and driving business impact. "Scale is excited to partner with Azure to help our customers transform their proprietary data into real business value with end-to-end GenAI Solutions, including model fine-tuning and customization in Azure." Vijay Karunamurthy, Field CTO, Scale AI Checkout a demo in BRK116 session showcasing how Scale AI’s fine-tuned models can improve agents in Azure AI Foundry and Copilot Studio. In the coming months, Scale AI will offer fine-tuning services for Azure AI Agents in Azure AI Foundry. For more details, please refer to this blog and start transforming your AI initiatives by exploring Scale AI on the Azure Marketplace. Gretel and Azure OpenAI Service Collaboration: Revolutionizing data pipeline for custom AI models Azure AI Foundry is collaborating with Gretel, a pioneer in synthetic data and privacy technology, to remove data bottlenecks and bring advanced AI development capabilities to our customers. Gretel's platform enables Azure users to generate high-quality datasets for ML and AI through multiple approaches - from prompts and seed examples to differential privacy-preserved synthetic data. This technology helps organizations overcome key challenges in AI development including data availability, privacy requirements, and high development costs with support for structured, unstructured, and hybrid text data formats. Through this collaboration, customers can seamlessly generate datasets tailored to their specific use cases and industry needs using Gretel, then use them directly in Azure OpenAI Service for fine-tuning. This integration greatly reduces both costs and time compared to traditional data labeling methods, while maintaining strong privacy and compliance standards. The collaboration enables new use cases for Azure AI Foundry customers who can now easily use synthetic data generated by Gretel for training and fine-tuning models. Some of the new use cases include cost-effective improvements for Small Language Models (SLMs), improved reasoning abilities of Large Language Models (LLMs), and scalable data generation from limited real-world examples. This value is already being realized by leading enterprises. “EY is leveraging the privacy-protected synthetic data to fine-tune Azure OpenAI Service models in the financial domain," said John Thompson, Global Client Technology AI Lead at EY. "Using this technology with differential privacy guarantees, we generate highly accurate synthetic datasets—within 1% of real data accuracy—that safeguard sensitive financial information and prevent PII exposure. This approach ensures model safety through privacy attack simulations and robust data quality reporting. With this integration, we can safely fine-tune models for our specific financial use cases while upholding the highest compliance and regulatory standards.” The Gretel integration with Azure OpenAI Service is available now through Gretel SDK. Explore this blog describing a finance industry case study and checkout details in technical documentation for fine-tuning Azure OpenAI Service models with synthetic data from Gretel. Visit this page to learn more Statsig and Azure Collaboration: Enabling Experimentation in AI Applications Statsig is a platform for feature management and experimentation that helps teams manage releases, run powerful experiments, and measure the performance of their products. Statsig and Azure AI Foundry are collaborating to enable customers to easily configure and run experiments (A/B tests) in Azure AI-powered applications, using Statsig SDKs in Python, NodeJS and .NET. With these Statsig SDKs, customers can manage the configuration of their AI applications, manage the release of new configurations, run A/B tests to optimize model and application performance, and automatically collect metrics at the model and application level. Please check out this page to learn more about the collaboration and get detailed documentation here. Conclusion The new collaborations between Azure and Weights & Biases, Scale AI, Gretel and Statsig represent a significant step forward in simplifying the process of AI model customization. These collaborations aim to address the common pain points associated with fine-tuning models, including lack of end-to-end tooling, data scarcity and privacy concerns, lack of expertise and experimentation tooling. Through these collaborations, Azure AI Foundry will empower organizations to fine-tune and customize models more efficiently, ultimately enabling faster, more accurate AI deployments. Whether it’s through better model tracking, access to synthetic data, or scalable data preparation services, these collaborations will help businesses unlock the full potential of AI.A Roadmap to Microsoft Ignite 2024 for AI Developers at Startups and Cloud-First Companies
Microsoft Ignite starts November 19th. Register now to attend virtually. Microsoft Ignite, the annual conference where customers, partners, and tech professionals come together to learn about Microsoft technology and AI solutions kicks off Nov 19 th in Chicago. If you don’t already have a pass to attend in-person, you can register as a virtual attendee and get access to all the keynotes and breakout sessions. Generative AI is rapidly evolving. Take for example, the recent launch of OpenAI’s newest models o1-preview and o1-mini. These models have enhanced reasoning capabilities with multistep reasoning. While earlier models excelled at language tasks such as writing and editing, the new models are designed to handle complex reasoning tasks such as advanced mathematics, coding, and STEM-based questions. They use a “chain of thought” technique that leads to increased accuracy versus earlier language-based models. Each generation of models continues to provide considerable performance improvement with lower costs. And that’s just OpenAI. There is a plethora of cutting-edge solutions providing models, frameworks, vector databases, LLM observability platforms, and many more tools for developers building GenAI apps. GenAI has spurred a record share of startup funding this year with 35% of US startup investment going to AI-related companies per Crunchbase data. All these advancements are making GenAI easier to build, more affordable and suitable for a wide range of use cases. While a large majority of the companies are jumping on the genAI bandwagon, experimenting with use cases and technologies, many are not running genAI apps at scale in production just yet. Microsoft has been an early innovator in the GenAI wave, and has been recognized as a leader in the 2024 Gartner Magic Quadrant for Cloud AI Developer Services as well as the latest Gartner Magic Quadrant for Cloud Application Platform. Microsoft Ignite promises to feature exciting announcements and learning opportunities to help you build, deploy, and run your GenAI apps at scale securely and responsibly. More importantly, you’ll also hear from peers at leading startups and at-scale cloud-first companies on how they are innovating and disrupting with GenAI. Here are the top must-watch sessions: Microsoft Ignite opening keynote (Nov 19, 6-8.30am PST). Reason to watch: Satya Nadella, CEO Microsoft, will be joined by Scott Guthrie, EVP Cloud and AI, Charlie Bell, EVP Microsoft Security, and Rajesh Jha, EVP Experiences and Devices, to talk about how this era of AI will unlock new opportunities, transform how developers work and drive business productivity across industries. Implementing AI Responsibly and Securely (Nov 19, 12.45-1.30pm PST). Reason to watch: The session addresses startup-specific issues like limited resources, ethical considerations, and building scalable AI solutions, offering practical tips. Annie Pearl, CVP/GM of Azure Experiences and Ecosystems leads this session featuring Adriano Koshiyama, Co-Founder Holistic AI and Sam Dover, Global AI Strategy Lead at Unilever. How Microsoft can help you build and sell differentiated AI apps (Nov 19, 2-2.45pm PST). Reason to watch: Startups and cloud-first companies are looking not just for a cloud platform to build their genAI apps on, but also a reliable technology partner that can help them grow their business. Jason Graefe, CVP ISV and Digital Natives is joined by Microsoft leaders for AI, developer services, and application platform as well as Adam Binnie, Chief Innovation Officer at Visier and Jeff Zobrist, VP global Partner Network and GTM at Siemens Digital Industries Software, to dive deep into building on the Microsoft Cloud and partnering with Microsoft to grow your business. Strategies for enterprises to fast-track AI adoption (Nov 21, 6.30-7.15 am PST). Reason to watch: This fireside chat, led by Tom Davis, Partner at Microsoft for Startups features Shlomi Avigdor, CPO Dataloop AI, Surojit Chatterjee, Founder and CEO Ema Unlimited, and Gil Perry, CEO and Co-founder D-ID. It delves into strategies and methods from industry leading startups who have worked hand in hand with global Enterprises to expedite the integration and widespread use of artificial intelligence across various industries. Inspire AI enterprise revolution with innovations from NVIDIA startups (Nov 21, 7.45-8.30 am PST). Reason to watch: NVIDIA Inception Head of Cloud Partnerships, Jennifer Hoskins, hosts a power-packed panel including Dr Vibhor Gupta, CEO Pangaea Data, Lauren Nemeth, COO Pinecone, and Andrew Gostine, CEO Artisight to Gain insights into how AI is driving groundbreaking innovations, enhancing efficiency, and creating new business opportunities across sectors. Unicorns Unleashed: Scaling Innovation with Microsoft (Nov 21, 12.30-1 pm PST). Reason to watch: If you are going to be attending in-person in Chicago, this panel led by Ross Kennedy, VP Digital Natives features visionary leaders from leading startups across the globe including Marc Jones CTO Shift Technology, Mike Murchison CEO and Co-founder Ada, and Sergio Passos CTO Blip. Best practices for building and monetizing your AI applications (Nov 21, 1.45-2.30pm PST). Reason to watch: This session will offer best practices and hands-on guidance to help maximize your AI product’s success, including product strategy, considerations for tech stack, GTM and even pricing. It features Dave Uppal, VP of ecosystem at Moveworks, and Alois Reitbauer, Chief Technology Strategist at Dynatrace. Disrupt and grow with Azure AI (On Demand). Reason to watch: If you are new to Azure, watch this on-demand session to learn about the full stack of Azure services that help you build, run, and scale your GenAI apps. Anand Kulkarni, CEO Crowdbotics will talk about their AI-powered app development platform built on Azure. Azure OpenAI for Startups and Digital-First Companies (On Demand). Reason to watch: While there is a ton of content on Azure OpenAI, this one specifically talks about challenges facing startups and at-scale cloud-first companies and how Azure OpenAI can help accelerate innovation. Mike Murchison, CEO Ada will discuss how Ada’s AI-powered customer service automation platform built on Azure OpenAI delivers streamlined and differentiated customer service experience to enterprises. These are just a few sessions featuring thought leaders from Microsoft and the startup industry. If you are a developer, there is a lot of content across the Azure AI, application platform, database, developer services, and infrastructure stack you can put on your watch list based on the models, use cases, and tools you use. I highly recommend checking out the session catalog to discover sessions that are most relevant to you. Popular AI toolchain solutions such as Arize, LlamaIndex, Pinecone, Weights & Biases, and Hugging Face are also featured in the content. Thousands of startups and at-scale software development companies are innovating with Microsoft. D-ID, for example, provides a Natural User Interface (NUI)-based immersive content platform for businesses specializing in customer experience, marketing, and sales. Enveda, a biotechnology company, uses AI-powered tools to identify and characterize a wide range of molecules produced by living organisms—the vast majority of which have never been explored by science—creating a database of chemical biodiversity. Cribl, the Data Engine for IT and Security, partners with Microsoft to make its data management capabilities available to enterprises to help improve security posture and enhance visibility. This is truly the era of AI and cloud-first companies are at the forefront of this wave. Join us at Microsoft Ignite 2024 and let’s build the future together! Btw, if you are looking to quickly get started with building apps on Azure for common AI use cases in as little as 5 mins, I highly recommend checking out the AI App Template Gallery. Resources: Microsoft Azure Microsoft for Startups Founders Hub Digital Natives on Azure Build AI applications with the new AI App Template Gallery Try Azure for free
Announcing an accelerator for GenAI-powered assistants using Azure AI Language and Azure OpenAI
We’re thrilled to introduce a new accelerator solution in GitHub Azure-Samples library designed specifically for creating and enhancing your GenAI-based conversational assistants with robust, human-controllable workflows. This accelerator uses key services from Azure AI Language in addition to Azure OpenAI, including PII detection to protect sensitive information, Conversational Language Understanding (CLU) to predict top users’ intents, Custom Question Answering (CQA) to respond to top questions with deterministic answers. Together with Azure OpenAI and Large Language Models (LLMs), the solution is designed to orchestrate and deliver a smooth, human-guided, controllable and deterministic conversational experience. The integration with LLMs will come soon. It’s perfect for developers and organizations looking to build assistants that can handle complex queries, route tasks, and provide reliable answers, all with a controlled, scalable architecture. Why This Accelerator While LLMs have been appreciated by many customers to build conversational assistants for natural, engaging, and context-aware interactions, there are challenges such as the significant efforts required in prompt engineering, document chunking, and reducing hallucinations to improve the quality of their Retrieval-Augmented Generation (RAG) solutions. If an AI quality issue is discovered in production, customers need to find an effective way to address it promptly. This solution aims to help customers utilize offerings in the Azure AI portfolio and address key challenges when building Generative AI (GenAI) assistants. Designed for flexibility and reliability, this accelerator enables human-controllable workflows that meet real-world customer needs. It minimizes the need for extensive prompt engineering by using a structured workflow to prioritize top questions with exact answers and custom intents that are critical to your business and use LLM to handle topics in a conversation that have lower priorities. This architecture not only enhances answer quality and control but also ensures that complex queries are handled efficiently. If you want to fix quickly an incorrect answer for your chatbot built with RAG, you can also attach this accelerator solution to your existing RAG solution and quickly add a QA pair with the correct response in CQA to fix the issue for your users. What This Accelerator Delivers This accelerator provides and demonstrates an end-to-end orchestration using a few capabilities in Azure AI Language and Azure OpenAI for conversational assistants. It can be applied in various scenarios where control over assistant behavior and response quality is essential, like call centers, help desks, and other customer support applications. Below is a reference architecture of the solutions: Key components of this solution include (components in dash boxes coming soon): Client-Side User Interface for Demonstration (coming soon) A web-based client-side interface is included in the accelerator solution, to showcase the accelerator solution in an interactive, user-friendly format. This web UI allows you to quickly explore and test this solution, such as its orchestration routing behavior and functionalities. Workflow Orchestration for Human-Controllable Conversations By combining services like CLU, CQA, and LLMs, the accelerator allows for a dynamic, adaptable workflow. CLU can recognize and route customer-defined intents, while CQA provides exact answers from predefined QA pairs. If a question falls outside the pre-defined scope, the workflow can seamlessly fall back to LLMs, which is enhanced with RAG for contextually relevant, accurate responses. This workflow ensures human-like adaptability while maintaining control over assistant responses. Conversational Language Understanding (CLU) for Intent Routing The CLU service allows you to define the top intents you want the assistants to handle. The top intents can be those critical to your business and/or those most users ask your assistants. This component plays a central role in directing conversations by interpreting user intents and routing them to the right action or AI agents. Whether completing a task or addressing specific customer needs, CLU provides the mechanism to ensure the assistant accurately understands and executes the process of handling custom-defined intents. Custom Question Answering (CQA) for Exact Answers and with No Hallucinations CQA allows you to create and manage predefined QA pairs to deliver precise responses, reducing ambiguity and ensuring that the assistant aligns closely with defined answers. This controlled response approach maintains consistency in interactions, improving reliability, particularly for high-stake or regulatory-sensitive conversations. You can also attach CQA to your existing RAG solution to quickly fix incorrect answers. PII Detection and Redaction for Privacy Protection (coming soon) Protecting user privacy is a top priority, especially in conversational AI. This accelerator showcases an optional integration of Azure AI Language’s Personally Identifiable Information (PII) to automatically identify and redact sensitive information, if compliance with privacy standards and regulations is required LLM with RAG to Handle Everything Else (coming soon) In this accelerator, we are using a RAG solution to handle missed intents or user queries on lower-priority topics. This RAG solution can be replaced with your existing one. The predefined intents and question-answer pairs can be appended and updated over time based on evolving business needs and DSATs (dissatisfaction) discovered in the RAG responses. This approach ensures controlled and deterministic experiences for high-value or high-priority topics while maintaining flexibility and extensibility for lower-priority interactions. Components Configuration for "Plug-and-Play" One of the standout features of this accelerator is its flexibility through a "plug-and-play" component configuration. The architecture is designed to allow you to easily swap, add, or remove components to tailor the solution to your specific needs. Whether you want to add custom intents, adjust fallback mechanisms, or incorporate additional data sources, the modular nature of the accelerator makes it simple to configure. Get Started Building Your GenAI-Powered Assistant Today Our new accelerator is available on GitHub, ready for developers to deploy, customize, and use as a foundation for your own needs. Join us as we move towards a future where GenAI can empower organizations to meet business needs with intelligent, adaptable, and human-controllable assistants. What’s more: Other New Azure AI Language Releases This Ignite Beyond these, Azure AI Language provides additional capabilities to support GenAI customers in more scenarios to ensure quality, privacy and flexible deployment in any types of environments, either clouds or on premises. We are also excited to announce the following new features launching at Ignite. Azure AI Language in Azure AI Studio: Azure AI Language is moving to AI Studio. Extract PII from text, Extract PII from conversation, Summarize text, Summarize conversation, Summarize for call center, and Text Analytics for health are now available in AI Studio playground. More skills follow. Conversational Language Understanding (CLU): Today, customers use CLU to build custom natural language understanding models hosted by Azure to predict the overall intention of an incoming utterance and extract important information from it. However, some customers have specific needs that require an on-premise connection. We are excited to announce runtime containers for CLU for these specific use cases. PII Detection and Redaction: Azure AI Language offers Text PII and Conversational PII services to extract personally identifiable information from input text and conversation to enhance privacy and security, oftentimes before sending data to the cloud or an LLM. We are excited to announce new improvements to these services - the preview API (version 2024-11-15-preview) now supports the option to mask detected sensitive entities with a label (i.e. “John Doe received a call from 424-878-9192” can now be masked with an entity label, i.e. . “[PERSON_1] received a call from [PHONENUMBER_1]”. More on how to specify the redaction policy style for your outputs can be found in our documentation. Native document support: The gating control is removed with the latest API version, 2024-11-15-preview, allowing customers to access native document support for PII redaction and summarization. Key updates in this version include: - Increased Maximum File Size Limits (from 1 MB to 10 MB). - Enhanced PII Redaction Customization: Customers can now specify whether they want only the redacted document or both the redacted document and a JSON file containing the detected entities. Language detection: Language detection is a preconfigured feature that can detect the language a document is written in and returns a language code for a wide range of languages, variants, dialects, and some regional/cultural languages. We are happy to announce today the general availability of scription detection capability, and 16 more languages support, which adds up to 139 total supported languages. Named entity recognition (NER): The Named Entity Recognition (NER) service supports customer scenarios for identifying and analyzing entities such as addresses, names, and phone numbers from inputs text. NER’s Generally Available API (version 2024-11-01) now supports several optional input parameters (inclusionList, exclusionList, inferenceOptions, and overlapPolicy) as well as an updated output structure (with new fields tags, type, and metadata) to ensure enhanced user customization and deeper analysis. More on how to use these parameters can be found in our documentation. Text analytics for health: Text analytics for health (TA4H) is a preconfigured feature that extracts and labels relevant medical information from unstructured texts such as doctor's notes, discharge summaries, clinical documents, and electronic health records. Today, we released support for Fast Healthcare Interoperability Resources (FHIR) structuring and temporal assertion detection in the Generally Available API.
