Announcing Preview of the Azure Functions Kafka extension
Published Apr 27 2020 09:30 PM 11.4K Views
Microsoft

kafka-functions-love.png

We're excited to announce today the preview of the Kafka Extension for Azure Functions. With this new extension you can now have functions trigger in response to messages in Kafka Topics, or write to a Kafka Topic through the output binding. This extension is supported when hosting functions in the Premium plan enabling it to elastically scale and trigger on Kafka messages. You can also use Kafka with Azure Functions containers in Kubernetes alongside Kubernetes-based Event Driven Autoscaling (KEDA).

 

How to get started with the Kafka Trigger

To get started with using the Kafka trigger, you need to include the extension in your function project.

 

.NET Functions

For .NET functions, you can pull in the Kafka NuGet extension.

dotnet add package Microsoft.Azure.WebJobs.Extensions.Kafka --version 2.0.0-beta
 
JavaScript, TypeScript, Python, Java, and PowerShell Functions

For functions in other languages, you need to install the extension into your project using the Azure Functions core tools

func extensions install --package Microsoft.Azure.WebJobs.Extensions.Kafka --version 2.0.0-beta

You can then create a function that can activate and run whenever a message is dropped in a Kafka Topic.

 

Example C# Trigger using Confluent Cloud (managed Kafka)

 

public static class kafka_example
    {
        [FunctionName("kafkaApp")]
        public static void ConfluentCloudStringTrigger(
             [KafkaTrigger(
                "BootstrapServer",
                "users",
                ConsumerGroup = "<ConsumerGroup>",
                Protocol = BrokerProtocol.SaslSsl,
                AuthenticationMode = BrokerAuthenticationMode.Plain,
                Username = "<APIKey>",
                Password = "<APISecret>",
                SslCaLocation = "confluent_cloud_cacert.pem")]
        KafkaEventData<string> kafkaEvent,
        ILogger logger)
        {        
            logger.LogInformation(kafkaEvent.Value.ToString());
        }
    }

 

 

 

 

 

Example host.json

 

{
  "version": "2.0",
  "extensions": {
    "kafka": {
      "maxBatchSize": 100
    }
  }
}

You can find more settings documented here

 

Example functions.json

Note: This is auto-generated for C#

{

  "bindings": [
    {
      "type": "kafkaTrigger",
      "consumerGroup": "azfunc",
      "protocol": "saslSsl",
      "authenticationMode": "plain",
      "username": "<KafkaAPIKey>",
      "password": "<KakfaSecret>",
      "sslCaLocation": "<RootCertificateFile>",
      "topic": "<KafkaTopicName>",
      "brokerList": "<Server>.eastus.azure.confluent.cloud:9092",
      "name": "kafkaEvent"
    }
  ],

}

Please find a complete end to end walkthrough and sample app using the Confluent Cloud in this GitHub repo Kafka extension sample with Confluent Cloud.

 

This extension is being developed in the open, please contribute, try out and post any issues on the Azure Functions Kafka extension GitHub repo.

 

Along with the Rabbit MQ Trigger that we announced last year that is now also supported in the Premium Plan, we continue to innovate to bring in more Cloud Native and Open Source event sources. We are looking forward to see what applications you build with this !

7 Comments
Version history
Last update:
‎Aug 27 2020 02:49 PM
Updated by: