AI/ML with Microsoft Fabric and SAS Viya. A discussion about right approach to drive the Synergies

Microsoft Fabric Will Deliver Scalable Cloud Analytics for Generative AI applications

Microsoft Fabric is an all-in-one analytics solution for enterprises that covers everything from data movement to data science, Real-Time Analytics, and business intelligence. It offers a comprehensive suite of services, including data lake, data engineering, and data integration, all in one place. Microsoft Fabric brings together new and existing components from Power BI, Azure Synapse, and Azure Data Factory into a single integrated environment. These components are then presented in various customized user experiences such as Data Engineering, Data Factory, Data Science, Data Warehouse, Real-Time Analytics, and Power BI onto a shared SaaS foundation.

Ideal First Step: Preparing the Data Landscape for AI/ML Applications 

Arrival of Generative AI is influencing the data analytics for Enterprises. 

Firstly, it’s amplifying the need for solutions that can manage distributed data at a large scale. The potential of enterprise AI can only be realized if data, currently scattered across numerous locations, can be made accessible to Language Learning Models (LLMs) or Other popular Models.

LLMs also demand a substantially larger volume of data (moreover they accelerate data generation itself). The process of collecting the data necessary for training a model is not as straightforward as executing some queries and serializing the results.

The structure of data is also becoming increasingly complex: training datasets; benchmarks and evaluations; preference optimization for fine-tuning based on expert feedback; audits and safeguards for bias, safety, and other risks, and so forth.

Additionally, with the rising popularity of Retrieval Augmented Generation (RAG), there are more immediate peer-to-peer requirements for one department to fine-tune models or create embeddings at scale by utilizing data from other departments.

Which Data Architecture should be Leveraged?

There is lot of literature on using distributed platforms (systems that work across different areas), pipelines across domains (ways of moving and transforming data), federated ownership (shared control), and self-explanatory data (data that is easy to understand) and these are under different names such as Data Mesh.

Microsoft has been thinking about data as a product and using a self-service platform model for data for a long time. This means an attempt to treat data like something that can be packaged and delivered to user groups who can then use it themselves (rather than depending upon specialized DATA & AI teams with limited resources or already overburdened under staffed IT Teams) for creating their own Generative AI applications.

Data Mesh is a type of decentralized data architecture that organizes data based on different business domains such as marketing, sales, human resources, etc. Microsoft Fabric’s data mesh architecture supports this approach by allowing data to be grouped into domains. It also enables decentralized governance, giving each business unit or department some level of ability to set their own rules and restrictions for data management based on their unique needs. Hence creating a Data Management Landing Zone (apart from the multiple AREA Specific Data Landing Zone)

In Microsoft Fabric, a domain is a way of organizing and grouping data that is related to a specific area or field within an organization. This is commonly done by grouping data based on business departments, allowing each department to manage their data according to their own regulations and needs.

In summary, Microsoft Fabric is a comprehensive data analytics platform, while the Data Mesh is an architectural pattern that can be implemented within Microsoft Fabric to organize and manage data in a decentralized manner. The two concepts are not in opposition but rather, Data Mesh is a way to use Microsoft Fabric more effectively in large and complex organizations.

In the context of Microsoft Fabric, these developments underscore the importance of a robust, scalable, and efficient data management system. This system should be capable of handling the complexities and volumes of data required by modern AI models, while also ensuring that data is accessible, usable, and secure.

SAS Viya and Microsoft Fabric - A Match Made in the AI God's Heaven 

SAS Viya Platform: A powerful AI/ML Model management Platform

SAS Viya is a powerful cloud-based analytics platform built by Microsoft's coveted partner SAS Institute Inc. that combines AI (Artificial Intelligence) and traditional analytics capabilities. SAS Viya seamlessly integrates with Microsoft Azure services, enhancing the analytics capabilities and providing a powerful platform for data-driven decision-making.

SAS Viya and Microsoft Fabric can find synergies in several ways, especially when SAS Viya is deployed on Azure. Here’s how they can complement each other:

1. Data Integration and Management: Microsoft Fabric’s capabilities in data management can be leveraged by SAS Viya to access and prepare data for analytics. This integration can streamline the process from data ingestion to preparation, ensuring that the data is ready for advanced analytics and AI modeling

2. AI and Analytics: SAS Viya’s advanced analytics and AI capabilities can enhance the insights generated from data within Microsoft Fabric. The integration of SAS Decision Builder into Fabric, for example, enables users to automate decisions and create composite AI workflows, which can be crucial for businesses looking to operationalize AI and analytics.

3. Model Deployment and Operations: With SAS Viya on Azure, users can benefit from Azure Machine Learning to build and deploy analytic models more efficiently. This includes using SAS Model Manager for governance and performance tracking, and integrating with Azure Machine Learning for deployment in the Microsoft Cloud.

4. Security and Governance: Both platforms emphasize security and governance. Microsoft Fabric provides a secure environment for data analytics, while SAS Viya offers governance capabilities for AI models. This synergy ensures that the entire analytics process is secure and compliant with industry standards.

5. Scalability and Performance: Azure’s cloud infrastructure allows SAS Viya to scale up and out without affecting performance. This means that as the demand for analytics grows, the combined solution can grow with it, providing consistent performance and reliability.

6. Decisioning Capabilities: The integration of SAS decision intelligence into Microsoft Fabric can help customers automate decisions seamlessly. This is particularly useful in industries like financial services for credit scoring or manufacturing for defect detection.

By combining the strengths of SAS Viya’s analytics and AI with Microsoft Fabric’s data management and AI capabilities, organizations can achieve a more seamless, efficient, and powerful analytics experience on Azure.