Tombstone events in Azure Event Hub for Log Compacted Topics

[deichrenner]

• Package Name: azure-eventhub
• Package Version: 5.12.1
• Operating System: MacOS 14.6
• Python Version: 3.12.4

Describe the bug
We want to use log compacted Event Hubs (see https://learn.microsoft.com/en-us/azure/event-hubs/use-log-compaction) with the python SDK. To use the full potential of the log compacted topics we need to be able to send tombstone events to indicate the end-of-life for a certain event indicated by its partition_key.

We naively followed the documentation and tried to send an empty event with an existing partition_key like

from azure.eventhub import EventData

tombstone_event = EventData(None)

However, the body of an event cannot be None. Therefore, the next best guess was to use EventData(b''). But this just sends and empty string and does not trigger the tombstone mechanic of the queue.

To Reproduce
Steps to reproduce the behavior:

1. Create a log compacted eventhub with a retention for tombstone events of 1 hour.

2. Send a message with a null body (as mentioned in the documentation, linked above)

   Example code:

   import asyncio
   import os
   import uuid
   
   import dotenv
   from azure.eventhub import EventData
   from azure.eventhub.aio import EventHubProducerClient
   from azure.identity.aio import DefaultAzureCredential
   
   dotenv.load_dotenv()
   
   credential = DefaultAzureCredential()
   
   
   async def run():
       producer = EventHubProducerClient.from_connection_string(
           os.getenv("EVENT_HUB_CONNECTION_STRING"),
           eventhub_name=os.getenv("EVENT_HUB_NAME"),
           credential=credential,
       )
       async with producer:
           user_id = str(uuid.uuid4())
           event_data_batch = await producer.create_batch(partition_key=user_id)
           event_data_batch.add(EventData(b''))
           await producer.send_batch(event_data_batch)
   
           await credential.close()
   
   
   if __name__ == "__main__":
       asyncio.run(run())

Expected behavior
After one hour, I would have expected the topic to be empty. However, the message with the empty body still remains in the topic.

Question
How to create an Event with a null body?

[github-actions]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[kashifkhan]

Thank you for the feedback @deichrenner . We will investigate and get back to you asap.

[kashifkhan]

@deichrenner I have reached out to the service team to see exactly what is happening and where this issue resides. Will update as soon as I have something.