In this guest blog post, Eric Melcher, Chief Technology Officer at Profisee, discusses how organizations can free their data scientists from the time-consuming tasks of data preparation and cleansing to let them focus on delivering insights, building models, and innovating with Profisee MDM in Azure Marketplace.
A recent report from Anaconda reveals that over 80 percent of data practitioners identify data preparation and cleansing as their most time-consuming tasks. These activities consume nearly 40 percent of their time – a costly investment, especially considering the high salaries and scarce availability of skilled data scientists. It’s a lot of time spent getting data merely ready to deliver value.
Yet the need for clean, trusted data – or “gold medallion data” in the context of a medallion lakehouse architecture – is growing exponentially. So how is this work done today? And more important, can it be done better?
A smarter way to prep and cleanse data
In modern lakehouse platforms such as Microsoft Fabric, Databricks, and Snowflake, data preparation and cleansing typically rely on custom code – often using R, Scala, or Python. Let’s call this the “build” approach.
While technically sound, this method struggles to scale. As the number of data sources, volume of data, and breadth of use cases grow, so does the complexity. Fortunately, there's a better option: plug-and-play, enterprise-grade technologies designed specifically to reduce this burden.
Enter master data management (MDM) – the “buy” approach.
MDM platforms are purpose-built to streamline and automate the cleansing and preparation of data, cutting the time and effort required by as much as 30 to 70 percent.
The diagram above illustrates how this works. Using the medallion architecture concept popularized by Databricks, we can follow the path raw data (bronze medallion data) takes to become a consumable, master data product (gold medallion data). An MDM platform like Profisee integrates raw data from source systems and deduplicates, enriches, and standardizes the records for use in Microsoft Fabric or another lakehouse solution.
With gold medallion data available for use by downstream systems like business intelligence, AI, ERP, and marketing automation, business users are empowered to derive business insights and improve operational efficiency without having to go through IT, helping organizations realize the full value of their data.
What’s involved in data preparation and cleansing?
Delivering consumable data (gold medallion data) for analytics – including AI – involves several key steps:
- Data profiling: Understanding data distribution, patterns, and completeness
- Standardization: Addressing formatting inconsistencies, extracting structure from unstructured sources, and filling in missing values
- Deduplication: Matching records across sources and applying survivorship rules to retain the best attributes
- Validation and remediation: Ensuring adherence to governance rules and involving humans as needed for oversight
With this much complexity, the method used to tackle these steps makes all the difference.
To build or to buy?
The “build vs. buy” debate is a familiar one in data management. Historically, many teams tried to build custom MDM solutions in-house. But experience proved the drawbacks: excessive timelines, resource strain, frequent maintenance, and inconsistent outcomes. Over time, the market moved toward purpose-built MDM platforms that offered speed, scale, and reliability.
Now, with the rise of lakehouse architectures, we’re seeing a similar pattern repeat. Early adopters – often highly technical – may lean into custom code. But will that suffice for enterprise-wide adoption?
Not likely.
Build vs. buy: a comparison
|
Factor |
Build (Custom Code) |
Buy (MDM Platform) |
|
Business Data Stewardship |
❌ Requires custom UI, rules, and workflows (e.g., Microsoft Power Apps) |
✅ Prebuilt UI, configurable rules and workflows, ready for business users |
|
Hierarchies & Rollups |
❌ Manual effort or custom code |
✅ Native support |
|
Data Quality Rules |
❌ Requires development and scripting |
✅ Built-in logic and visual rule builders for business users |
|
Matching & Survivorship |
❌ Custom logic, testing, and ongoing tuning; prone to failure if sources change |
✅ Built-in, rule-driven logic with business user review and drag-and-drop configuration |
|
Flexibility |
❓ Depends on engineering bandwidth; high maintenance as rules and sources evolve |
✅ Highly adaptable, business-friendly rule building |
|
Scalability |
❓ High cost to scale new use cases |
✅ Proven scalability across data volumes and use cases |
|
Total Cost of Ownership |
❌ Low license costs, but high labor and long-term support effort |
✅ Higher initial investment, but significantly lower long-term costs |
|
Time to Value |
⏳ 6–9 months |
⚡ 3–4 months |
Profisee + lakehouse = trusted, scalable data
If you’re building a medallion architecture on Databricks, Microsoft Fabric, or Snowflake, now is the time to reduce the reliance on custom scripts and unlock rapid scale with Profisee MDM. Profisee offers native integration with Microsoft Fabric and is tightly integrated with Databricks and Snowflake via bi-directional connectors.
Once connected, Profisee:
- Aggregates raw data from multiple source systems
- Identifies and removes duplicates while preserving best attributes
- Applies quality rules (including those from Microsoft Purview)
- Involves business users for oversight where needed
- Publishes harmonized, gold medallion data back into your lakehouse or source systems
Customers consistently report 30 percent to 70 percent reductions in data prep and cleansing efforts – some even say their lakehouse initiatives wouldn’t be viable without Profisee.
Let data scientists be scientists
Your data scientists shouldn’t be spending their days fixing errors or duplicating MDM functionality in code. Their focus should be on delivering insights, building models, and innovating.
With Profisee, the heavy lifting of data prep and cleansing is offloaded to a platform built specifically for it – complete with governance, flexibility, and scale. The result? Business users take ownership of their data, data scientists focus on high-value work, and data investments in Microsoft Azure, Databricks, Fabric, or Snowflake start delivering faster.
Ready to reduce the grind?
Profisee makes the promise of a medallion lakehouse architecture real – by making the data trustworthy, usable, and business-ready from the start.
Less prep. More progress.
For Azure Marketplace customers, Profisee SaaS Enterprise Master Data Management is available as a transactable solution. To learn more about how Profisee can help you make the most of your Microsoft Fabric investment, visit our website.
