Introduction In mature data platforms, scaling compute is rarely the primary challenge. Shared, elastic Spark pools already provide sufficient processing capacity for most workloads. The harder pro...

The Two Pillars That Determine Success Data Quality Cannot Be an Afterthought The most common mistake we see is treating data quality as something to address after the pipeline is running. This a...

Understanding the Problem Space When organizations first approach multi-source data ingestion, they typically start with explicit configuration. Each database connection is defined individually, ea...

Author's: Amudha Palani amudhapalani​, Eric Kwashie ekwashie​, Peter Lo PeterLo​ and Rafia Aqil Rafia_Aqil​    Disaster recovery (DR) is a critical component of any cloud-native data analytic...

Author's: Lavanya Sreedhar LavanyaSreedhar​, Peter Lo PeterLo​, Aryan Anmol aryananmol​, Shreya Harvu shreyaharvu​ and Rafia Aqil Rafia_Aqil​    In this guide, we provide practical guidance f...

Introduction Data fuels analytics, machine learning, and AI  but only if it’s trustworthy. Most organizations struggle with inconsistent schemas, nulls, data drift, or unexpected upstream changes t...

Author's: Peter Lo PeterLo​, ​Amudha Palani amudhapalani​, Geoffrey Rathinapandi geofegeo​ and Rafia Aqil  Rafia_Aqil​      Observability in Azure Databricks is the ability to continuously ...

Authors Sailing Ni*, Joy Yu*, Peng Yang*, Richard Sie*, Yifei Wang* *These authors contributed equally. Affiliation Master of Science in Business Analytics (MSBA), UCLA Anderson School o...

Author's: Chris Walk cwalk​, Dan Johnson danjohn1234​, Eduardo dos Santos eduardomdossantos​, Ted Kim tekim​, Eric Kwashie ekwashie​, Chris Haynes Chris_Haynes​, Tayo Akigbogun takigbogun​ and Rafi...

Co-Authored by: Sanjeev Nair Sanjeev Nair​ and Rafia Aqil Rafia_Aqil​    This guide walks through a proven approach to Databricks cost optimization, structured in three phases: Discovery, Clu...