Announcing Preview of the Azure Functions Kafka extension

[Anirudh Garg]

We're excited to announce today the preview of the Kafka Extension for Azure Functions. With this new extension you can now have functions trigger in response to messages in Kafka Topics, or write to a Kafka Topic through the output binding. This extension is supported when hosting functions in the Premium plan enabling it to elastically scale and trigger on Kafka messages. You can also use Kafka with Azure Functions containers in Kubernetes alongside Kubernetes-based Event Driven Autoscaling (KEDA).

How to get started with the Kafka Trigger

To get started with using the Kafka trigger, you need to include the extension in your function project.

.NET Functions

For .NET functions, you can pull in the Kafka NuGet extension.

dotnet add package Microsoft.Azure.WebJobs.Extensions.Kafka --version 2.0.0-beta

 

JavaScript, TypeScript, Python, Java, and PowerShell Functions

For functions in other languages, you need to install the extension into your project using the Azure Functions core tools

func extensions install --package Microsoft.Azure.WebJobs.Extensions.Kafka --version 2.0.0-beta

You can then create a function that can activate and run whenever a message is dropped in a Kafka Topic.

Example C# Trigger using Confluent Cloud (managed Kafka)

public static class kafka_example
    {
        [FunctionName("kafkaApp")]
        public static void ConfluentCloudStringTrigger(
             [KafkaTrigger(
                "BootstrapServer",
                "users",
                ConsumerGroup = "<ConsumerGroup>",
                Protocol = BrokerProtocol.SaslSsl,
                AuthenticationMode = BrokerAuthenticationMode.Plain,
                Username = "<APIKey>",
                Password = "<APISecret>",
                SslCaLocation = "confluent_cloud_cacert.pem")]
        KafkaEventData<string> kafkaEvent,
        ILogger logger)
        {        
            logger.LogInformation(kafkaEvent.Value.ToString());
        }
    }

Example host.json

{
  "version": "2.0",
  "extensions": {
    "kafka": {
      "maxBatchSize": 100
    }
  }
}

You can find more settings documented here

Example functions.json

Note: This is auto-generated for C#

{

  "bindings": [
    {
      "type": "kafkaTrigger",
      "consumerGroup": "azfunc",
      "protocol": "saslSsl",
      "authenticationMode": "plain",
      "username": "<KafkaAPIKey>",
      "password": "<KakfaSecret>",
      "sslCaLocation": "<RootCertificateFile>",
      "topic": "<KafkaTopicName>",
      "brokerList": "<Server>.eastus.azure.confluent.cloud:9092",
      "name": "kafkaEvent"
    }
  ],

}

Please find a complete end to end walkthrough and sample app using the Confluent Cloud in this GitHub repo Kafka extension sample with Confluent Cloud.

This extension is being developed in the open, please contribute, try out and post any issues on the Azure Functions Kafka extension GitHub repo.

Along with the Rabbit MQ Trigger that we announced last year that is now also supported in the Premium Plan, we continue to innovate to bring in more Cloud Native and Open Source event sources. We are looking forward to see what applications you build with this !

[FatimaMookhtiar]

Hi Anirudh, thanks for this! I have tried out  and used the C# example and it works like a charm! However, i'm relatively new to Java and we have a requirement for implementing this in Java too. Any help on the Java equivalent would be highly appreciated!

https://functionscdn.azureedge.net/public/ExtensionBundles/Microsoft.Azure.Functions.ExtensionBundle... return the version as 1.1.1

I can't seem to figure where the KafkaTrigger is in the extensions for Java? The preview version docs aren't available too for Java.

Many thanks, Fatima.

[Anirudh Garg]

Hi @FatimaMookhtiar , 

Please have a look at the following: https://github.com/Azure/azure-functions-kafka-extension/blob/dev/samples/java/README.MD and https://github.com/Azure/azure-functions-kafka-extension/blob/dev/samples/java/README.MD for examples on how to use Java with this extension. 

Thanks,

Anirudh

[FatimaMookhtiar]

Update- It works! Thanks so much! 

[FatimaMookhtiar]

There's a few things missing in the steps which I'd like to highlight as might help others, they are below:-

1. I needed to have the librdkafka dlls present in the bin folder to get it working, else I kept getting a runtime exception complaining that it couldn't find the dll. To get it I launched the extensions.csproj and installed the nuget package (https://www.nuget.org/packages/librdkafka.redist/) . This then gave me access to the dlls, which I copied into the bin folder of the functionApp. It then worked fine!

2. Also a small tweak to the readme, you need to add --package to it. 

$ func extensions install --package Microsoft.Azure.WebJobs.Extensions.Kafka --version 2.0.0-beta

 3. I also renamed the local in pom.xml to something else to get rid of the warning. 

<repository>
    <id>extensions</id>
    <url>file://${project.basedir}/src/repo</url>
</repository>

 Is there any specific timeline when this might move from Preview to a stable version? Thanks, Fatima.

[Anirudh Garg]

Thanks Fatima I edited the blog post with your feedback. I will also update the sample about your comment on librdkafka dll. Thank you so much for your feedback. This is exactly the kind of feedback we are waiting for before we can make it GA. Currently we are hoping it will be the next 2-3 months if not earlier.

[FatimaMookhtiar]

This sounds great, thanks. We might have a team using this, so will post feedback based on testing. Cheers. 

[Tech7868]

Hi @Anirudh Garg - can you please let us know if Kafka trigger is supported by MS or still in preview?

Also i was testing but facing a problem if my function fails to process a message due to any reason, it still gets committed. How to not let that message gets committed?

Thanks