Event Hub is a big-data streaming platform and event ingestion service part of the messaging service ecosystem in Azure (if you are interested in a comparison between other similar services you can look here: Compare Azure messaging services - Azure Event Grid | Microsoft Docs).
The focus of this article is not a comparison between a full-blown Kafka cluster and the Event Hub Kafka endpoint (you can find a good feature comparison there: Use event hub from Apache Kafka app - Azure Event Hubs - Azure Event Hubs | Microsoft Docs), what I'm going to analyze is the authentication options we have while using the Kafka Interface using a C# client.
First of all, there is good news: Event Hub for Kafka support both SAS (shared access signatures) and OAuth 2.0 (in our case Azure AD) authentication as you can see here: Use event hub from Apache Kafka app - Azure Event Hubs - Azure Event Hubs | Microsoft Docs.
Both authentication methods support a fine grade authorization on the Data Plane to control which operation your client can do: Publishing messages, Receiving messages, or both.
When using SAS, you can define multiple Policies in which select the operations you want to enable.
When using Azure AD, the same level of control (and more since you can configure also the Control Plane) is achieved by the Azure Role-Base Access Control (also referred to as RBAC) system (Authorize access with Azure Active Directory - Azure Event Hubs | Microsoft Docs) thanks to the built-in roles for Data Owner, Data Sender, and Data Receiver and the additional roles for the Schema Registry (https://docs.microsoft.com/en-us/azure/event-hubs/schema-registry-overview#azure-role-based-access-c...)
Ok, sounds promising, then I started to look for sample code around using a C# library authenticating with an OAuth 2.0 token obtained from Azure AD but I didn't found anything, so I decided to create my sample code and the best .NET Kafka library I found out there it was the “Confluent.Kafka”.
Ok, let's try to use this one!
I've double-checked, and this library seems able to leverage OAuth 2.0 token to connect to a Kafka Server as I found in their doc something interesting (Class ClientExtensions | Confluent.Kafka:(
Digging a bit more in the details I found also an Integration test in their git hub repo (confluent-kafka-dotnet/OauthBearerToken_PublishConsume.cs at fd4b95b420418153aa39d7c94df1e1135c09c07...) showing how to use the Token Refresh Handler callback:
Cool! That sample code was using a simple token generator utility, now should be as easy as to inject an Azure AD Token in this callback!
Ok; now try to embed our Azure Identity library!
As mentioned earlier Event Hub support Azure AD Authorization using RBAC (Authorize access with Azure Active Directory - Azure Event Hubs | Microsoft Docs), therefore now we need to grant the permission to the security principal we want to use, retrieve a valid token with the right scope and provide to the Kafka client library in the callback we saw just above.
Ok, what what?? Security Principal?
Right, basically when you want to use Azure AD to get a token you need to own an Identity (aka a Principle) and with our Microsoft Identity platform (Azure AD).
To deep dive a bit more in these details you can see the following doc link:
Authenticate an application to access Azure Event Hubs resources - Azure Event Hubs | Microsoft Docs
Authentication a managed identity with Azure Active Directory - Azure Event Hubs | Microsoft Docs
This means that if you are running your Kafka Client within an Azure compute service (VMs, Functions, App Services, AKS Pods, Container Instances… etc) which supports "Managed Identities" the complexity of handling and rotating the secret is directly managed by the platform.
Additional details around Azure Identity can be found here: Azure Identity client library for .NET | Azure SDK for Net (windows.net) and here: Azure Services that support managed identities - Azure AD | Microsoft Docs
We will see that applied later in the article.
Ok, so as first step I’ve created an App Registration on my Azure AD Tenant:
And generated a Secret for it:
Then I moved to my Event Hub and configured the Role of “Event Hubs Data Owner” to the service principle I’ve previously created under my Azure AD Tenant:
Finally using the Confluence.Kafka client library I’m creating a .NET Core sample app that leverages the DefaultAzureCredential library and asks for a Token.
Considering the library using the v2 endpoint of Azure AD I need to use the “scope” rather than the resource URI only to obtain an Access Token (please look at this: Microsoft identity platform scopes, permissions, & consent | Microsoft Docs) I’ve just appended /.default to the resource URI.
Then, before debugging my code I’ve configured Visual Studio to inject 3 environments variable: the Tenant Id, the Client Id and the Client Secret (from my App Registration):
As mentioned above, the DefaultAzureCredential has a fallback strategy to discover the credentials to be used. You can find more information here: azure-sdk-for-net/README.md at main · Azure/azure-sdk-for-net (github.com)
In a nutshell: it can leverage both Environmental variables (as I did for my debugging session), Managed Identities, or other developers-friendly types of authentication.
The takeaway here is that using that identity library you do not have to change your code from debugging to production, and in production, you can leverage Managed Identity whenever they are available in your execution environment (VMs, AppWeb, Functions, AKS Pods, Container Instances).
Executing the sample
Ok so we have a .NET C# sample code which grabs a token from Azure AD, create a Kafka Producer and simulating some data flow toward Event Hub:
The full source code is here: algorni/dotnet-kafka-event-hub-aad: A .NET Client for the Kafka Event Hub endpoint using AAD Token f...
I've forced myself to leverage the Kafka endpoint given the easy-to-migrate nature of it.
If I had being using the native API of Event Hub to create an instance of Producer leveraging the same DefaultAzureCredential library would be as simple as this single line of code:
EventHubProducerClient producerClient = new EventHubProducerClient(txtNamespace.Text, txtEventHub.Text, new DefaultAzureCredential())
This would set up a working Event Hub Producer client which can be used in a comparable way as the Kafka Producer.