Data Validation at Scale with Azure Synapse
In the world of Artificial Intelligence and Machine Learning, data quality is paramount in ensuring our models and algorithms perform correctly. By leveraging the power of Spark on Azure Synapse, we can perform detailed data validation at a tremendous scale for your data science workloads.
Codename Project Bose: Calculate Azure Cost of an Enterprise by cost centers, divisions, projects
While working on various customer and partner facing roles, I felt the necessity of a simple and flexible solution to align Azure Cost to the customer’s organizational structure. “Project Bose” is a fully operational prototype derived from the same thought process. This is a side project I am working on during my leisure time. I found various customers derived similar solutions in-house, and there are ISV solutions as well. But there are a few fundamental differences between “Project Bose” and all the other solutions I found. “Project Bose” has a flexible backend and hence any changes in organizational structure can easily be implemented on it without disruption. It is also independent of using Resource Tags, which gives it the opportunity to remain non-vulnerable to erroneous values injected intentionally or non-intentionally by IT-Ops. Project Bose is dedicated to an eminent Indian physicist, who predicted (theoretically) a special behavior of atoms, which later became famous as Bose-Einstein condensates. He also discovered a sub-atomic particle, later named after him – Boson. Why Project Bose? To replace Azure specific hierarchy of Cost Analysis (Tenant>Management Group>Subscription>Resource Group) with organization specific hierarchy such as Business Unit>Department>Cost Center>Project/Application. Provide an alternative option to mark projects/applications under different divisions/departments – moving away from using resource tags. Flexible way to replicate organizational hierarchy through easy-to-use interface and change/edit the tree vertically or horizontally with minimum effort and no disruption. A great way of presenting ongoing and past expenses on different projects and slice and dice cost at different levels. Optional implementation of forecasting. How does it work? Admins – can create the hierarchy according to organizational structure (ex. BU>Division>Cost Center>Project) and add items at each level. This structure is flexible. They can change the hierarchy at any time. For example – can change name of a Division, add or remove a Cost Center etc. Project/Application Owners – can add their project under a specific branch of the organizational tree and then align Azure Resource Group(s) to it. Business Owners/Decision Makers – see current and past cost, trends and forecasting. Base Architecture Azure Deployment Sample Visualization The code base with IaC is available with my GitHub repo. I can share with anyone interested.
Validate data using an Azure Function and Great Expectations
Great Expectations is a great tool for validating the incoming data into your data platform, and what better way to run it then having it triggered by new files by using Azure Function! In this blog post, I will discuss what the main concepts of Great Expectations are, how to get it running in a Azure Function and how to embed that in a larger event-driven architecture. Finally, a link to the code on Github is given so you can get started yourself!Distributed ML Training for Lane Detection, powered by NVIDIA and Azure NetApp Files
Microsoft, NetApp and Run:ai have partnered in the creation of this article to demonstrate the unique capabilities of the Azure NetApp Files together with the Run:ai platform for simplifying orchestration of AI workloads. This article provides a reference architecture for streamlining the process of both data pipelines and workload orchestration for Distributed Machine Learning Training for Lane Detection, by ensuring the use of the full potential of NVIDIA GPUs.High-performance storage for AI Model Training tasks using Azure ML studio with Azure NetApp Files
This article describes how to provide enterprise grade high performance persistent storage with data protection capability for AI Model training tasks using studio compute instances with Azure NetApp Files (ANF).
