We're excited to announce today the preview of the Kafka Extension for Azure Functions. With this new extension you can now have functions trigger in response to messages in Kafka Topics, or write to a Kafka Topic through the output binding. This extension is supported when hosting functions in the Premium plan enabling it to elastically scale and trigger on Kafka messages. You can also use Kafka with Azure Functions containers in Kubernetes alongside Kubernetes-based Event Driven Autoscaling (KEDA).
How to get started with the Kafka Trigger
To get started with using the Kafka trigger, you need to include the extension in your function project.
.NET Functions
For .NET functions, you can pull in the Kafka NuGet extension.
dotnet add package Microsoft.Azure.WebJobs.Extensions.Kafka --version 2.0.0-beta
JavaScript, TypeScript, Python, Java, and PowerShell Functions
For functions in other languages, you need to install the extension into your project using the Azure Functions core tools
func extensions install --package Microsoft.Azure.WebJobs.Extensions.Kafka --version 2.0.0-beta
You can then create a function that can activate and run whenever a message is dropped in a Kafka Topic.
Example C# Trigger using Confluent Cloud (managed Kafka)
public static class kafka_example
{
[FunctionName("kafkaApp")]
public static void ConfluentCloudStringTrigger(
[KafkaTrigger(
"BootstrapServer",
"users",
ConsumerGroup = "<ConsumerGroup>",
Protocol = BrokerProtocol.SaslSsl,
AuthenticationMode = BrokerAuthenticationMode.Plain,
Username = "<APIKey>",
Password = "<APISecret>",
SslCaLocation = "confluent_cloud_cacert.pem")]
KafkaEventData<string> kafkaEvent,
ILogger logger)
{
logger.LogInformation(kafkaEvent.Value.ToString());
}
}
Example host.json
{
"version": "2.0",
"extensions": {
"kafka": {
"maxBatchSize": 100
}
}
}
You can find more settings documented here
Example functions.json
Note: This is auto-generated for C#
{
"bindings": [
{
"type": "kafkaTrigger",
"consumerGroup": "azfunc",
"protocol": "saslSsl",
"authenticationMode": "plain",
"username": "<KafkaAPIKey>",
"password": "<KakfaSecret>",
"sslCaLocation": "<RootCertificateFile>",
"topic": "<KafkaTopicName>",
"brokerList": "<Server>.eastus.azure.confluent.cloud:9092",
"name": "kafkaEvent"
}
],
}
Please find a complete end to end walkthrough and sample app using the Confluent Cloud in this GitHub repo Kafka extension sample with Confluent Cloud.
This extension is being developed in the open, please contribute, try out and post any issues on the Azure Functions Kafka extension GitHub repo.
Along with the Rabbit MQ Trigger that we announced last year that is now also supported in the Premium Plan, we continue to innovate to bring in more Cloud Native and Open Source event sources. We are looking forward to see what applications you build with this !