Azure Data Explorer has a mature, open source Kafka Connect sink connector that is in use in production by multiple customers. The connector was *gold* certified in September 2020 by Confluent and is now listed in Confluent Connector Hub. The certification involved stringent testing by Confluent for quality, standards, best practices, feature completeness, delivery guarantees and performance. To meet the certification requirements, we added a number of new features to our connector, renamed some, and removed support for some with version 1.x. This article details the same.
For comprehensive information about our connector, visit our Git repo.
Improved delivery guarantees
Our connector now offers "At least once" delivery guarantees (no data loss).
1. Comprehensive upfront validation
The connector validates for all required and dependent sink properties, access and grants and shuts down upfront with appropriate error messages.
2. Configurable behavior on error
A new sink property with possible values of fail (shut down connector task), ignore (ignore and continue processing), and log (log errors and continue processing).