Category: Azure Event Hubs

AzureAzure Event Hubs

Boost Data Reliability with Geo-Replication for Azure Event Hubs

Unknown's avatar on June 28, 2024

This week, Microsoft announced the public preview of geo-replication for Azure Event Hubs. Geo-replication enhances Microsoft Azure data availability and geo-disaster recovery capabilities by enabling the replication of Event Hubs data payloads across different Azure regions.

With geo-replication, your client applications continue to interact with the primary namespace. Customers can designate a secondary region, choose replication consistency (synchronous or asynchronous), and set replication lag for the data. The service handles the replication between primary and secondary regions. If a primary change is needed (for maintenance or failover), the secondary can be promoted to primary, seamlessly servicing all client requests without altering any configurations (connection strings, authentication, etc.). The former primary then becomes the secondary, ensuring synchronization between both regions.

In summary, geo-replication is designed to provide you with the following benefits:

  • High availability: You can ensure that your data is always accessible and durable, even in the event of a regional outage or disruption. You can also reduce the impact of planned maintenance events by switching to the secondary region before the primary region undergoes any updates or changes.
  • Disaster recovery: You can recover your data quickly and seamlessly in case of a disaster that affects your primary region. You can initiate a failover to the secondary region and resume your data streaming operations with minimal downtime and data loss.
  • Regional compliance: You can meet the regulatory and compliance requirements of your industry or region by replicating your data to a secondary region that complies with the same or similar standards as your primary region. You can also leverage the geo-redundancy of your data to support your business continuity and resilience plans.

How to get started with Azure Event Hubs Geo-replication?

If you want to try out Azure Event Hubs Geo-replication, please check out the official documentation over at Azure Event Hubs Geo-replication documentation and they also have a demo here.

I look forward to when this becomes GA and is available in more regions.

Enjoy!

References

https://techcommunity.microsoft.com/t5/messaging-on-azure-blog/announcing-public-preview-for-geo-replication-for-azure-event/ba-p/4164522

Azure Event Hubs Geo-replication documentation

AzureAzure Event Hubs

Azure Event Hubs Unveils Large Message Support

Unknown's avatar on May 22, 2024

This week Microsoft announced in public preview, support for large messages (up to 20 MB) in Azure Event Hubs in its self-service scalable dedicated clusters, enhancing its capabilities to handle a wide range of message sizes without additional costs.

This new feature allows for seamless streaming of large messages without requiring any client code changes, maintaining compatibility with existing Event Hubs SDKs and the Kafka API. This enhancement ensures uninterrupted business operations by accommodating instances where messages cannot be divided into smaller segments. The service continues to offer high throughput and low latency, making it a robust solution for data streaming needs.

What are some cases for large message support?

Here are some key use cases for the new large message support in Azure Event Hubs:

  • Multimedia Streaming: Handling large video, audio, or image files that cannot be split into smaller segments.
  • Data Aggregation: Transmitting aggregated data sets or logs that exceed typical message size limits.
  • IoT Applications: Streaming large sensor data or firmware updates from IoT devices.
  • Batch Processing: Sending large batches of data for processing without needing to break them down.

These enhancements ensure seamless and uninterrupted business operations across various scenarios.

How do you enable large message support?

To enable large message support in your existing Azure Event Hubs setup, follow these steps:

  1. Use Self-Serve Scalable Dedicated Clusters: Ensure your Event Hubs are built on the latest infrastructure that supports self-serve scalable dedicated clusters. If you are using Event Hubs, then you will need to create an Event Hub Cluster to take advantage of large message support.
  2. No Client Code Changes Needed: You can continue using your existing Event Hubs SDK or Kafka API. The only change required is in the message or event size itself.

For more detailed instructions, visit the documentation at aka.ms/largemessagesupportforeh.

How do Azure Event Hubs differ from Azure Event Hub Clusters?

Azure Event Hubs and Event Hub Clusters serve different purposes within the Azure ecosystem:

  • Azure Event Hubs: This is a fully managed, real-time data ingestion service that can receive and process millions of events per second. It’s designed for high-throughput data streaming and is commonly used for big data and analytics.
  • Azure Event Hub Clusters: These are dedicated clusters that provide isolated resources for Event Hubs. They offer enhanced performance, scalability, and the ability to handle large messages (up to 20 MB). Clusters are ideal for scenarios requiring high throughput and low latency.

Enjoy!

References

https://techcommunity.microsoft.com/t5/messaging-on-azure-blog/announcing-large-message-support-for-azure-event-hubs-public/ba-p/4146455

https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-quickstart-stream-large-messages

https://learn.microsoft.com/en-us/azure/event-hubs/compare-tiers

Azure Event Hubs Overview