Blog Post

Apps on Azure Blog
2 MIN READ

Announcing Preview of the Azure Functions Kafka extension

Anirudh Garg's avatar
Anirudh Garg
Icon for Microsoft rankMicrosoft
Apr 28, 2020

We're excited to announce today the preview of the Kafka Extension for Azure Functions. With this new extension you can now have functions trigger in response to messages in Kafka Topics, or write to a Kafka Topic through the output binding. This extension is supported when hosting functions in the Premium plan enabling it to elastically scale and trigger on Kafka messages. You can also use Kafka with Azure Functions containers in Kubernetes alongside Kubernetes-based Event Driven Autoscaling (KEDA).

 

How to get started with the Kafka Trigger

To get started with using the Kafka trigger, you need to include the extension in your function project.

 

.NET Functions

For .NET functions, you can pull in the Kafka NuGet extension.

dotnet add package Microsoft.Azure.WebJobs.Extensions.Kafka --version 2.0.0-beta
 
JavaScript, TypeScript, Python, Java, and PowerShell Functions

For functions in other languages, you need to install the extension into your project using the Azure Functions core tools

func extensions install --package Microsoft.Azure.WebJobs.Extensions.Kafka --version 2.0.0-beta

You can then create a function that can activate and run whenever a message is dropped in a Kafka Topic.

 

Example C# Trigger using Confluent Cloud (managed Kafka)

 

public static class kafka_example
    {
        [FunctionName("kafkaApp")]
        public static void ConfluentCloudStringTrigger(
             [KafkaTrigger(
                "BootstrapServer",
                "users",
                ConsumerGroup = "<ConsumerGroup>",
                Protocol = BrokerProtocol.SaslSsl,
                AuthenticationMode = BrokerAuthenticationMode.Plain,
                Username = "<APIKey>",
                Password = "<APISecret>",
                SslCaLocation = "confluent_cloud_cacert.pem")]
        KafkaEventData<string> kafkaEvent,
        ILogger logger)
        {        
            logger.LogInformation(kafkaEvent.Value.ToString());
        }
    }

 

 

 

 

 

Example host.json

 

{
  "version": "2.0",
  "extensions": {
    "kafka": {
      "maxBatchSize": 100
    }
  }
}

You can find more settings documented here

 

Example functions.json

Note: This is auto-generated for C#

{

  "bindings": [
    {
      "type": "kafkaTrigger",
      "consumerGroup": "azfunc",
      "protocol": "saslSsl",
      "authenticationMode": "plain",
      "username": "<KafkaAPIKey>",
      "password": "<KakfaSecret>",
      "sslCaLocation": "<RootCertificateFile>",
      "topic": "<KafkaTopicName>",
      "brokerList": "<Server>.eastus.azure.confluent.cloud:9092",
      "name": "kafkaEvent"
    }
  ],

}

Please find a complete end to end walkthrough and sample app using the Confluent Cloud in this GitHub repo Kafka extension sample with Confluent Cloud.

 

This extension is being developed in the open, please contribute, try out and post any issues on the Azure Functions Kafka extension GitHub repo.

 

Along with the Rabbit MQ Trigger that we announced last year that is now also supported in the Premium Plan, we continue to innovate to bring in more Cloud Native and Open Source event sources. We are looking forward to see what applications you build with this !

Updated Aug 27, 2020
Version 14.0
  • Thanks Fatima I edited the blog post with your feedback. I will also update the sample about your comment on librdkafka dll. Thank you so much for your feedback. This is exactly the kind of feedback we are waiting for before we can make it GA. Currently we are hoping it will be the next 2-3 months if not earlier.

  • FatimaMookhtiar's avatar
    FatimaMookhtiar
    Copper Contributor

    Hi Anirudh, thanks for this! I have tried out  and used the C# example and it works like a charm! However, i'm relatively new to Java and we have a requirement for implementing this in Java too. Any help on the Java equivalent would be highly appreciated!

     
     

    https://functionscdn.azureedge.net/public/ExtensionBundles/Microsoft.Azure.Functions.ExtensionBundle/index.json return the version as 1.1.1

     
     
     
     

    I can't seem to figure where the KafkaTrigger is in the extensions for Java? The preview version docs aren't available too for Java.

     

     

    Many thanks, Fatima.

  • FatimaMookhtiar's avatar
    FatimaMookhtiar
    Copper Contributor

    There's a few things missing in the steps which I'd like to highlight as might help others, they are below:-

    1. I needed to have the librdkafka dlls present in the bin folder to get it working, else I kept getting a runtime exception complaining that it couldn't find the dll. To get it I launched the extensions.csproj and installed the nuget package (https://www.nuget.org/packages/librdkafka.redist/) . This then gave me access to the dlls, which I copied into the bin folder of the functionApp. It then worked fine!

     

    2. Also a small tweak to the readme, you need to add --package to it. 

    $ func extensions install --package Microsoft.Azure.WebJobs.Extensions.Kafka --version 2.0.0-beta

     3. I also renamed the local in pom.xml to something else to get rid of the warning. 

    <repository>
    <id>extensions</id>
    <url>file://${project.basedir}/src/repo</url>
    </repository>

     Is there any specific timeline when this might move from Preview to a stable version? Thanks, Fatima.

  • FatimaMookhtiar's avatar
    FatimaMookhtiar
    Copper Contributor

    This sounds great, thanks. We might have a team using this, so will post feedback based on testing. Cheers. 

  • Tech7868's avatar
    Tech7868
    Copper Contributor

    Hi Anirudh Garg - can you please let us know if Kafka trigger is supported by MS or still in preview?

    Also i was testing but facing a problem if my function fails to process a message due to any reason, it still gets committed. How to not let that message gets committed?

     

    Thanks