By: Allen Jones, Principal Software Engineer, Azure Incubations, Microsoft
The need to detect and react to data changes is pervasive across modern software systems. Whether it’s Kubernetes adapting to resource updates, building management systems responding to sensor shifts, or enterprise applications tracking business state, the ability to act precisely when data in databases or stateful systems changes is critical. Event-driven architectures, underpinned by technologies like Apache Kafka, Azure Event Hubs, and serverless functions, are the go-to solution. Examples include processing IoT telemetry to trigger alerts or handling user activity streams in real-time analytics. Yet, while these systems excel at generic event processing, they demand significant additional effort when the goal is to isolate and respond to specific data changes—a specialized subset of this paradigm we call change-driven architecture.
Drasi, the open-source Data Change Processing platform, is an exciting new option for building change-driven solutions easily. This article explores how Drasi’s unique capabilities—centered around clearly defined change semantics, the continuous query pattern, and the reaction pattern—provide a more efficient and effective alternative to traditional event-driven implementations. By codifying consistent patterns and reducing complexity, Drasi empowers solution architects and developers to build responsive systems faster with greater precision and less complexity, which makes them less brittle and easier to update as solution requirements change.
The Limits of Generic Event Processing
Event-driven architectures are powerful options for handling high-volume data streams to support data transfer, aggregation, and analytics use cases, but they fall short when the task is to react to specific, often complex and contextual, data transitions. Developers must bridge the gap between generic events and actionable changes, often by:
- Parsing general purpose event payloads to infer state changes e.g., decoding Kubernetes pod events to detect a "Running" to "Failed" transition. Because the implementor of the event generating system does not know all possible consumer use cases, they often produce very generic events (e.g. xxxAdded, xxxUpdated, xxxDeleted), or large all-encompassing event documents to accommodate as many uses cases as possible,
- Filtering irrelevant events to isolate meaningful updates e.g., sifting through audit logs to identify compliance violations. Most event generating system don’t produce just the events consumers need, they produce a firehose of events covering everything in the system, leaving it to the consumer to filter this down to what they need.
- Maintaining state to track context across events e.g., aggregating inventory transactions to detect threshold breaches. Most events relate to a single thing occurring. In the case of updates, they sometimes provide before and after versions of the changed data, but they lack any ability to relate the change to other elements of the system. The consumer must create and maintain an accurate set of state.
Polling, a common alternative, wastes resources querying unchanged data and risks delaying, or even missing entirely, important changes due to the polling intervals, while custom change log processing requires intricate logic to map low-level updates to high-level conditions. For instance, detecting when a database replication lag exceeds 30 seconds might involve correlating logs from multiple systems. These options may be tolerable occasionally but become costly to build, support, and maintain when repeated organization-wide many times for many scenarios.
Changes vs. Events
Drasi distinguishes changes from events. Events signal that something happened, but their producer-defined structure and ambiguous semantics require consumers to interpret intent—e.g. a "UserUpdated" event might not clarify what changed. In contrast, a change in Drasi has precise semantics: a set of additions, updates, and/or deletions to a continuous query result set occurring because of changes to the source data. The structure of the change is defined by the author of the query (i.e. the consumer), not the designer of the source system. This consumer-driven data model reduces dependency on producer assumptions and simplifies engineering and downstream processing.
Continuous Query Pattern
Central to Drasi is the continuous query pattern, implemented using the openCypher graph query language. A continuous query runs perpetually, fed by change logs from one or more data sources, maintaining the current set of query results and generating notifications when those results change. Unlike producer-defined event streams, this pattern empowers consumers to specify the relevant properties and their relationships using a familiar database-type query.
Consider this example monitoring infrastructure health:
MATCH (s:Server)-[:HOSTS]->(a:Application)-[:DEPENDS_ON]->(db:Database)
WHERE s.status = 'active' AND db.replicationLag > 30
RETURN
s.id AS serverId, s.location AS datacenter,
a.name AS appName, db.replicationLag AS lagSeconds
At any time, this continuous query result contains the answer to the question “Which applications are experiencing a DB replication lag of 30 seconds or more?”. The MATCH clause spans servers, applications, and databases—potentially from distinct systems like an inventory database and monitoring tool. The WHERE clause specifies the condition (active servers with lagging databases), and the RETURN clause shapes the result set. Drasi ensures this result set stays current, notifying subscribers only when it changes. With strict temporal semantics, historical result sets can also be retrieved, aiding audits or debugging.
Reaction Pattern
The reaction pattern leverages the consistency of change notifications. Because a query defines inclusion semantics, state transitions are clear. For example, in a building management system:
MATCH
(r:Room)-[:HAS_SENSOR]->(t:TemperatureSensor),
(r:Room)-[:LOCATED_IN]->(b:Building)
WHERE
b.id = 'HQ1' AND
t.currentReading < 18 AND
r.occupied = true
RETURN
r.id AS roomId, t.currentReading AS temp, b.id AS buildingId
A room enters the result set when its temperature drops below 18°C while occupied, triggering heating. It exits when conditions change, stopping the action. The standard structure of change notifications, either room additions or removals, ensures straightforward reaction logic.
Real-World Applications
Drasi’s change-driven approach shines across many domains. For resource optimization, this continuous query identifies underutilized VMs to trigger cost-saving reactions:
MATCH (vm:VirtualMachine)-[:DEPLOYED_IN]->(s:Subscription)
WHERE
vm.status = 'running' AND
vm.utilizationPercent < 10 AND
duration.between(vm.lastHighUtilization, datetime()) > duration('P7D')
RETURN
vm.id AS vmId, vm.utilizationPercent AS usage, s.owner AS owner
For business exceptions, this continuous query flags delayed orders due to stock shortages:
MATCH (o:Order)-[:HAS_ITEM]->(i:OrderItem)-[:REQUIRES]->(p:Product)
WHERE
o.status = 'processing' AND
p.stockLevel < i.quantity AND
duration.between(o.placementDate, datetime()) > duration('P2D')
RETURN
o.id AS orderId, p.id AS productId, p.stockLevel AS stock
In both cases, Drasi simplifies detection and reaction, avoiding the custom logic required in event-based systems.
Addressing Misconceptions and Scaling Efficiently
Because Drasi is fundamentally a new type of data processing service, it’s easy to underestimate its value, which can come from underestimating the complexity of change detection or assuming event-driven tools suffice. While building a solution with events or polling might work as a one-off solution, repeating the same approach multiple times across an organization is inefficient and expensive. The solutions are also complex and brittle, making them difficult to maintain and update to address changing requirements. Drasi’s standardized approach, rooted in its change-driven architecture and theoretical underpinnings, streamlines change-driven systems, enabling teams to focus on defining queries and reactions rather than rebuilding infrastructure.
Conclusion
For change-driven architectures, Drasi is superior to generic event-based technologies because it targets the specific challenge of reacting to data changes. Its continuous query pattern, precise change semantics, and streamlined reaction pattern deliver cleaner, more efficient solutions than the traditional approaches. Available on GitHub under the Apache 2.0 License, Drasi offers solution architects and developers a robust, scalable foundation to transform complex change detection into declarative, cost-effective implementations—making it the optimal choice for responsive, data-centric systems.
Microsoft