Organizations around the world are excited by the opportunity to leverage data to improve productivity and efficiency while also creating new revenue streams and competitive differentiation.  With the increasing availability of AI tools to reason over the massive quantity of data produced each day, these goals have never been closer to reality.  However, the ability to effectively harness AI at scale across large, distributed data estate requires a new approach.

At Ignite 2024, Microsoft is making a variety of exciting announcements regarding how customers can collect data from any source, process and analyze that data locally and globally for AI-enabled insights anywhere, and securely share those insights with employees and the ecosystem.  These announcements include the General Availability of Azure IoT Operations, the General Availability of Fabric Real-Time Intelligence, new capabilities for Azure Video Indexer, enabled by Arc, and a new Arc-enabled service focused on enabling generative AI applications at the edge.

These announcements build upon Azure's adaptive cloud approach, a new paradigm to accelerate adoption of cloud innovations in scenarios extending beyond our hyperscale datacenters. The adaptive cloud approach unifies hybrid, multi-cloud, and edge infrastructure with a common operations, innovation, and insights platform, enabled by Azure Arc.  Organizations that choose to embrace this methodology can drive consistency in how they manage and secure infrastructure, applications and data, as well as in how they collect and contextualize data.  The ability to reduce operational complexity and improve visibility can serve as a powerful combination to help organizations accelerate digital transformation outcomes.   

Distributed organizations have distributed data

Today, it is common for organizations to have a data estate that is distributed across a variety of physical locations.  According to global research firm Gartner:

"Gartner® predicts that, by 2025, more than 50% of enterprise managed-data will be created and processed outside the data center or cloud.”*

There are a variety of reasons why distributed data is the norm versus an exception. Many organizations have a global footprint or have operations that span multiple physical locations. Additionally, organizations in any industry may go through acquisitions, introducing new physical and digital technology infrastructure.  They may do business in countries with specific data privacy or security requirements.  They may also be leveraging data for real-time or near real-time business processes that require very low latency.  While there are valid reasons data lives in multiple locations and systems, this reality can also impede the ability to derive value from that data.

Imagine a manufacturer with a goal to improve its yield across sites, an energy company aiming to reduce its global carbon footprint - or a retailer that would like to improve the in-store experience by leveraging consumer data across brands. None of these scenarios can be accomplished with a fragmented enterprise data management strategy.  In addition, many organizations are operating within complex ecosystems where the ability to share data and insights is paramount. Data sitting in multiple places and under the governance of multiple systems can also act as a barrier to valuable ecosystem participation.

Leveraging an adaptive cloud approach to unify data and AI across a distributed estate

As we spend time with customers around the globe with distributed data estates, it is clear a more holistic approach to data is required to achieve their business goals.  

There are several aspects of this holistic approach – the ability to collect data from any data source, the ability to process and analyze that data locally and globally for AI-enabled actionable insights anywhere, and the ability to securely share those insights with the people that need them to do their jobs, as well as key ecosystem partners. 

Microsoft’s adaptive cloud approach provides capabilities across each of these domains:             

Azure IoT Operations

Today we’re announcing the General Availability of Azure IoT Operations, enabling IoT edge data collection, processing, and delivery of edge data into global systems. Azure IoT Operations provides the ability to connect and communicate with OPC UA servers, edge data normalization and contextualization, scalable, bidirectional edge to cloud communication through native integration with Microsoft Fabric, tooling to power event driven applications at the edge, and an intuitive user interface to manage assets and configure data pipelines.

Microsoft Fabric

Microsoft Fabric is a comprehensive, end-to-end analytics and data platform built to meet the needs of organizations seeking a unified solution. It brings together all aspects of data management- spanning data integration, movement, processing, ingestion, transformation, real-time event routing, and reporting - into a single platform.

Since the launch of Fabric Real-Time Intelligence at Microsoft Build, there's been continued customer excitement about the ability to build real-time solutions quickly and easily to solve key business problems. And now, Fabric Real-Time Intelligence is Generally Available.  Microsoft Fabric offers a complete set of capabilities for event and streaming data with Real-Time Intelligence, including ingestion, transformation, storage, analytics, visualization, tracking, AI, and real-time actions. These capabilities are critical to help derive insights and meaning from the vast amount of data available through Azure IoT Operations.

Real-Time Intelligence complements Fabric’s extensive suite of capabilities, including Data Engineering, Data Factory, Data Science, Data Warehousing, Power BI, and Database management, all integrated as part of a cohesive data experience. Instead of relying on different databases or data warehouses, organizations can centralize data storage with OneLake. Analytical models can be informed by bringing together data from multiple sources in one place.  AI capabilities are seamlessly embedded within Fabric, eliminating the need for manual integration. With Fabric, customers can also easily transition their raw data into actionable insights for business users.

From a data governance standpoint, Microsoft Fabric provides a set of capabilities that help customers manage, protect, monitor, and improve the discoverability of an organization's sensitive information.  Azure IoT Operations is Azure Arc-enabled so when deploying applications to the edge, common standards, access controls, and policies are applied to data across distributed environments as well. This consistent approach to data governance across the data estate ensures the right people and partners can access the data and insights they need to perform their business function.

Scaling AI initiatives on a unified data foundation

Once organizations have a consistent edge to cloud data foundation that includes proper governance and security mechanisms, they are well positioned to scale their AI initiatives across the enterprise.  As organizations invest in solutions build on the Azure AI platform, they want to be able to deploy them to any location where insights are needed and manage them in a consistent way.  The adaptive cloud approach extends Kubernetes to the edge with Arc-enabled Kubernetes, providing the ability to create and deploy container-based AI applications to the edge and manage those over their lifecycle.  As mentioned earlier, Azure-Arc managed applications also inherit policies from the Azure Resource Manager, providing access control for AI generated insights as well.

An example of such an application is Azure AI Video Indexer.  Azure AI Video Indexer enabled by Arc is an Azure Arc extension enabled service that runs video and audio analysis, and generative AI on edge devices. The solution is designed to run on Azure Arc enabled Kubernetes and supports many audio and video formats, including MP4 and other common formats. At Ignite, a new multi-modal video description feature is being launched in preview. This functionality leverages the generative AI Phi3.5v to produce video summaries by examining text, audio, and visual components without the need to view the entire video.                                                     

At Ignite, we are also announcing the private preview of a new Arc-enabled service focused on enabling generative AI applications at the edge.  This service is deployable as a turnkey Arc for Kubernetes extension and allows customers to use industry-leading small and large language models to search their on-premises data using a technique called Retrieval Augmented Generation, or “RAG”. With it, organizations can use Azure-consistent tools and APIs to create Copilot-like experiences, while meeting data locality and compliance requirements. 

As the technology to derive value from data continues to become more intelligent and more accessible, the need for a modern data foundation to support those tools increases as well.  This modern data foundation must support the ability to uncover and act on insights anywhere and do so in a secure manner.  At Ignite this year, we’re looking forward to meeting with you to discuss how Azure’s adaptive cloud approach can help unify your distributed data estate and scale your business outcomes.

Learn more:

You can find a full listing of opportunities to learn more about our Adaptive cloud approach at Ignite here: Adaptive Cloud at Ignite – GitHub. 

Source:

* Gartner, Hype Cycle™ for Edge Computing, 2024, Thomas Bittman, 15 July 2024 (Report accessible to Gartner subscribers only).  GARTNER is a registered trademark and service mark of Gartner, Inc. and/or its affiliates in the U.S. and internationally, and HYPE CYCLE is a registered trademark of Gartner, Inc. and/or its affiliates and are used herein with permission. All rights reserved.