Blog Post

Messaging on Azure Blog
1 MIN READ

Kafka Compression on Azure Event Hubs Public Preview

ChristinaCompy's avatar
ChristinaCompy
Icon for Microsoft rankMicrosoft
Nov 15, 2023

Kafka compression

Microsoft is happy to announce the public preview of Kafka Compression for Apache Kafka on Azure Event Hubs.  Kafka Compression enables message producers to transmit compressed data which is decompressed by the consumer.  By compressing your payload, you are able to save a lot of bandwidth and reduce the memory overhead on the brokers supporting your traffic. 

Kafka producer application developers can enable message compression by setting the compression.type property.  

Compression.type = none | gzip

 

These changes are exposed in the header which then allows the consumer to properly decompress the data. 

 

The feature is currently only supported for Apache Kafka traffic and not AMQP or web service. 

 

There is more information on this feature in documentation at https://aka.ms/kafkacompression 

Updated Nov 15, 2023
Version 1.0
azure event hubs
  • JohanBerg1's avatar
    JohanBerg1
    Copper Contributor
    Dec 27, 2023

    I got the message from the support team that the compression public preview is only enabled for the Premium and Dedicated tier of Event Hubs as of now. We haven't tested this yet. But the message is that if you are using the Premium or Dedicated tier it should just work you do not need to enable anything other than setting the compression to gzip.

  • JohanBerg1's avatar
    JohanBerg1
    Copper Contributor
    Dec 13, 2023
    Here is the debug from the producer. Looks like the broker does not support the message format. Are you sure there isn't anything that we need to enable for this "public preview feature" on our Event Hub instance?
     
    Logger: 1702475535.726 RDKAFKA-7-PRODUCE: rdkafka#producer-2: [thrd:sasl_ssl://ehn-eastus-d1v1-awc-exchange.servicebus.windows.net:]: sasl_ssl://ehn-eastus-d1v1-awc-exchange.servicebus.windows.net:9093/0: arte_1 [0]: Produce MessageSet with 113 message(s) (8169 bytes, ApiVersion 7, MsgVersion 2, MsgId 1, BaseSeq 0, PID{Id:39,Epoch:0}, gzip)
    Logger: 1702475535.726 RDKAFKA-7-SEND: rdkafka#producer-2: [thrd:sasl_ssl://ehn-eastus-d1v1-awc-exchange.servicebus.windows.net:]: sasl_ssl://ehn-eastus-d1v1-awc-exchange.servicebus.windows.net:9093/0: Sent ProduceRequest (v7, 8222 bytes @ 0, CorrId 6)
    Logger: 1702475535.763 RDKAFKA-7-RECV: rdkafka#producer-2: [thrd:sasl_ssl://ehn-eastus-d1v1-awc-exchange.servicebus.windows.net:]: sasl_ssl://ehn-eastus-d1v1-awc-exchange.servicebus.windows.net:9093/0: Received ProduceResponse (v7, 50 bytes, CorrId 6, rtt 37.09ms)
    Logger: 1702475535.763 RDKAFKA-7-REQERR: rdkafka#producer-2: [thrd:sasl_ssl://ehn-eastus-d1v1-awc-exchange.servicebus.windows.net:]: sasl_ssl://ehn-eastus-d1v1-awc-exchange.servicebus.windows.net:9093/0: ProduceRequest failed: Broker: Message format on broker does not support request: actions Permanent,MsgNotPersisted
    Logger: 1702475535.763 RDKAFKA-7-MSGSET: rdkafka#producer-2: [thrd:sasl_ssl://ehn-eastus-d1v1-awc-exchange.servicebus.windows.net:]: sasl_ssl://ehn-eastus-d1v1-awc-exchange.servicebus.windows.net:9093/0: arte_1 [0]: MessageSet with 113 message(s) (MsgId 1, BaseSeq 0) encountered error: Broker: Message format on broker does not support request (actions Permanent,MsgNotPersisted)
    Logger: 1702475535.763 RDKAFKA-7-DRAIN: rdkafka#producer-2: [thrd:sasl_ssl://ehn-eastus-d1v1-awc-exchange.servicebus.windows.net:]: Beginning partition drain in state Assigned for PID{Id:39,Epoch:0} epoch bump for 1 partition(s) with in-flight requests: message sequence gap
  • JohanBerg1's avatar
    JohanBerg1
    Copper Contributor
    Dec 05, 2023

    Do I need to do anything to the Event Hubs resource to enable this preview feature? Couldn't find anything obvious and it doesn't seem to work just as is.

    Tried setting Compression.type = gzip

    Messages just disappears, no error message.

  • ChristinaCompy's avatar
    ChristinaCompy
    Icon for Microsoft rankMicrosoft
    Dec 12, 2023

    Hi Johan,

    You don't have to enable anything on the Event Hubs resource itself.  This is producer side compression and is enabled on the Kafka producer side with compression.type=gzip.  

    The message should not disappear though.  Did you get an error?

  • JohanBerg1's avatar
    JohanBerg1
    Copper Contributor
    Dec 13, 2023

    No, we didn't get any error, messages just disappear.
    Okay, so we will try again and troubleshoot more on our end then if there is nothing else we need to enable on the Event Hubs side.

  • 0livet0's avatar
    0livet0
    Copper Contributor
    Dec 27, 2023

    Hi ChristinaCompy 

    I have same issue here and today. 

    Is there any specific azure kafka python library requirement to make it work?

    BR