Tag: Messaging

AnalyticsMicrosoft Fabric

Bringing Real‑Time Intelligence to Airport Data Streams (Type-B Messages) – Part 1

If you’ve ever wondered what happens to your bag after you drop it off at the check-in counter, you’re not alone. There’s an entire world of events firing beneath the surface of every airport, and it turns out it makes for a pretty compelling real-time data scenario.

I recently put together a use case walkthrough on Bringing Real-Time Intelligence to Airport Data Streams using Microsoft Fabric. In this post, I want to break down the architecture, explain the data model, and show how you can build a real-time observability pipeline over something as relatable as baggage tracking.

Why Airports?

Airports are a great analogy for event-driven systems because every action generates a traceable event:

  • You book a flight → event
  • You check in → event
  • You drop off your bag → event

And that bag doesn’t just teleport to the carousel. It travels through a complex network of baggage belts, ramps, weigh stations, scanners, and machinery. It gets loaded onto the plane, unloaded at the destination, sent through customs (maybe), and eventually delivered to the baggage belt — or it gets lost.

Every step is a state change. Every state change is an opportunity to capture data and act on it in real time.

The Event Model

For this use case, I modelled three categories of events published to the stream:

Baggage Events

  • Airport.Baggage.CheckedIn
  • Airport.Baggage.Screened
  • Airport.Baggage.Inspected
  • Airport.Baggage.Rejected
  • Airport.Baggage.Loaded
  • Airport.Baggage.Unloaded
  • Airport.Baggage.CustomsCleared
  • Airport.Baggage.Withheld
  • Airport.Baggage.ArrivedAtBelt
  • Airport.Baggage.Delivered
  • Airport.Baggage.Lost

Flight Operational Events

  • Airport.Flight.Closed
  • Airport.Flight.Departed
  • Airport.Flight.Arrived

Passenger Events

  • Airport.Passenger.CheckedIn

This structure follows a clean domain-driven naming convention that maps naturally to a topic-per-domain strategy in Event Hubs or Fabric Eventstream.

Real Airports Use Type-B Messages

In the real world, airports and airlines don’t talk to each other over REST APIs or CloudEvents — they use Aviation Type-B messages, a fixed-format ASCII text messaging standard that’s been in use since the 1960s. These messages are transmitted over dedicated aviation networks operated by SITA (Société Internationale de Télécommunications Aéronautiques) and ARINC (now part of Collins Aerospace), and they remain the backbone of operational messaging across the global aviation industry today.

The key Type-B message types that map directly to this use case are:

MessageNameDescription
BSMBaggage Source MessageGenerated at check-in; carries the bag tag number, passenger details, and routing
BTMBaggage Transfer MessageUsed for interline transfer bags moving between airlines
BPMBaggage Processed MessageConfirmation that a bag has been processed at a handling point
BUMBaggage Unload MessageSignals that bags have been removed from an aircraft
MVTMovement MessageCommunicates flight departure (AD), arrival (AA), and estimated times (ET)
LDMLoad Distribution MessageDescribes how cargo and baggage are distributed across the aircraft
CPMContainer/Pallet Distribution MessageDetails the ULD (Unit Load Device) positioning on the aircraft

IATA Resolution 753

IATA Resolution 753 mandates that airlines track every bag at a minimum of four key touchpoints:

  1. Passenger handover at check-in
  2. Loading onto the aircraft
  3. Delivery to the transfer area (for connecting flights)
  4. Return to the passenger at arrival

Resolution 753 exists because lost and mishandled bags cost the industry hundreds of millions of dollars annually, and real-time tracking directly reduces that. It came into effect in 2018 and drove significant investment in baggage scanning infrastructure and data exchange across airlines and ground handlers.

Bridging Type-B to Modern Streaming

Here’s where it gets interesting from a platform perspective. Type-B messages carry all the right information — they’re just locked inside a legacy fixed-format protocol on a private network. The modernization opportunity is to parse and bridge those messages into a modern event stream.

In practice, that means something like:

  1. A SITA or ARINC Type-B feed gets received by a gateway or middleware layer
  2. Each message is parsed and mapped to a structured event (e.g., a BSM becomes an Airport.Baggage.CheckedIn CloudEvent)
  3. That event is published to Azure Event Hubs or a Fabric Eventstream endpoint
  4. From there, the full Fabric RTI pipeline takes over

The Baggage Handling Simulator in this demo effectively plays the role of that gateway — it generates CloudEvents that mirror what you’d produce by parsing a real Type-B feed. If you were building this for production, the simulator would be replaced by a Type-B parser wired up to a live SITA or ARINC connection.

Architecture Overview

Here is an overview of the architecture in Microsoft Fabric Real-Time Intelligence:

The pipeline follows a standard Ingest → Analyze → Act pattern inside Microsoft Fabric Real-Time Intelligence:

Ingest & Process

Events are published to Azure Event Hubs or directly to a Microsoft Fabric Eventstream endpoint as CloudEvents. The Real-Time Hub acts as the central discovery and governance point for all streaming sources in the workspace.

Eventstream picks up those events and routes them into the Eventhouse — Fabric’s purpose-built KQL database engine optimized for high-throughput, time-series, and log-style workloads.

Inside the Eventhouse, I use a classic Bronze / Silver / Gold medallion layering approach:

  • Bronze — raw ingested events, exactly as received
  • Silver — cleaned and enriched data via Update Policies (KQL-based transformation rules that fire automatically on ingest)
  • Gold — aggregated and pre-computed views via Materialized Views for fast querying

Analyze & Transform

KQL Querysets sit on top of the Eventhouse and let you write ad-hoc and saved queries in Kusto Query Language. KQL is incredibly expressive for time-series data — you can window events, calculate SLAs, detect anomalies, and join across streams with just a few lines.

Visualize & Act

From there, you have a few options:

  • Real-Time Dashboard — a native Fabric dashboard that auto-refreshes on a schedule or on data change, built directly from KQL queries
  • Power BI — for richer semantic model-based reporting or executive dashboards
  • Activator — Fabric’s alerting and automation engine; you can define rules like “if a bag hasn’t moved in 30 minutes, fire an alert”
  • Data Agents — AI-powered agents that can answer natural language questions over your KQL data

Data can also land in OneLake, so it’s available for downstream batch analytics and data science workloads. This is not enabled by default, so it’s something you would need to turn on.

The Baggage Handling Simulator

To drive the demo, I used the Baggage Handling Simulator — a Python CLI built by Clemens Vasters that simulates realistic airport baggage operations. I forked the repository and then adjusted it for Type-B messaging. You can find my fork on GitHub: calloncampbell/BaggageHandling-TypeB-Simulator at feature/type-b-messages

The simulator generates:

  • Baggage tracking events (check-in, screening, loading, unloading, delivery)
  • Passenger events (check-in, boarding)
  • Flight lifecycle events (scheduled, closed, departed, arrived)

Events are published as CloudEvents to either Azure Event Hubs or a Fabric Eventstream endpoint. Flight schedules are also persisted to SQL Server, which gives you a relational anchor to join against your streaming data if needed.

This is a great reference simulator if you want to explore real-time analytics without having to stand up your own IoT or event infrastructure.

Let’s look at the simulator…

Now let’s look at a basic Real-Time Dashboard:

What I Took Away

What I find compelling about this use case is how approachable it is. Most people have been through an airport. Most people have waited anxiously at a baggage belt. That shared experience makes the data model immediately intuitive — and that makes it a great teaching scenario for real-time streaming concepts.

From a Fabric RTI perspective, this use case demonstrates a few things I think are really powerful:

  1. The medallion pattern works in streaming too. Update Policies and Materialized Views give you that Bronze/Silver/Gold structure without a separate transformation job or orchestration layer.
  2. KQL is a first-class citizen. It’s not just a query language — it’s the transformation layer, the alerting layer, and the visualization layer.
  3. Activator closes the loop. Moving from insight to action inside the same platform — without building custom workflows — is genuinely useful.

If you’re interested in exploring Microsoft Fabric Real-Time Intelligence, this airport scenario is a solid and fun way to get started. In Part 2 of this post, I’ll dig into the Fabric Real-Time Intelligence setup.

Enjoy!

References

AzureAzure Event Hubs

Boost Data Reliability with Geo-Replication for Azure Event Hubs

This week, Microsoft announced the public preview of geo-replication for Azure Event Hubs. Geo-replication enhances Microsoft Azure data availability and geo-disaster recovery capabilities by enabling the replication of Event Hubs data payloads across different Azure regions.

With geo-replication, your client applications continue to interact with the primary namespace. Customers can designate a secondary region, choose replication consistency (synchronous or asynchronous), and set replication lag for the data. The service handles the replication between primary and secondary regions. If a primary change is needed (for maintenance or failover), the secondary can be promoted to primary, seamlessly servicing all client requests without altering any configurations (connection strings, authentication, etc.). The former primary then becomes the secondary, ensuring synchronization between both regions.

In summary, geo-replication is designed to provide you with the following benefits:

  • High availability: You can ensure that your data is always accessible and durable, even in the event of a regional outage or disruption. You can also reduce the impact of planned maintenance events by switching to the secondary region before the primary region undergoes any updates or changes.
  • Disaster recovery: You can recover your data quickly and seamlessly in case of a disaster that affects your primary region. You can initiate a failover to the secondary region and resume your data streaming operations with minimal downtime and data loss.
  • Regional compliance: You can meet the regulatory and compliance requirements of your industry or region by replicating your data to a secondary region that complies with the same or similar standards as your primary region. You can also leverage the geo-redundancy of your data to support your business continuity and resilience plans.

How to get started with Azure Event Hubs Geo-replication?

If you want to try out Azure Event Hubs Geo-replication, please check out the official documentation over at Azure Event Hubs Geo-replication documentation and they also have a demo here.

I look forward to when this becomes GA and is available in more regions.

Enjoy!

References

https://techcommunity.microsoft.com/t5/messaging-on-azure-blog/announcing-public-preview-for-geo-replication-for-azure-event/ba-p/4164522

Azure Event Hubs Geo-replication documentation

AzureAzure Event Hubs

Azure Event Hubs Unveils Large Message Support

This week Microsoft announced in public preview, support for large messages (up to 20 MB) in Azure Event Hubs in its self-service scalable dedicated clusters, enhancing its capabilities to handle a wide range of message sizes without additional costs.

This new feature allows for seamless streaming of large messages without requiring any client code changes, maintaining compatibility with existing Event Hubs SDKs and the Kafka API. This enhancement ensures uninterrupted business operations by accommodating instances where messages cannot be divided into smaller segments. The service continues to offer high throughput and low latency, making it a robust solution for data streaming needs.

What are some cases for large message support?

Here are some key use cases for the new large message support in Azure Event Hubs:

  • Multimedia Streaming: Handling large video, audio, or image files that cannot be split into smaller segments.
  • Data Aggregation: Transmitting aggregated data sets or logs that exceed typical message size limits.
  • IoT Applications: Streaming large sensor data or firmware updates from IoT devices.
  • Batch Processing: Sending large batches of data for processing without needing to break them down.

These enhancements ensure seamless and uninterrupted business operations across various scenarios.

How do you enable large message support?

To enable large message support in your existing Azure Event Hubs setup, follow these steps:

  1. Use Self-Serve Scalable Dedicated Clusters: Ensure your Event Hubs are built on the latest infrastructure that supports self-serve scalable dedicated clusters. If you are using Event Hubs, then you will need to create an Event Hub Cluster to take advantage of large message support.
  2. No Client Code Changes Needed: You can continue using your existing Event Hubs SDK or Kafka API. The only change required is in the message or event size itself.

For more detailed instructions, visit the documentation at aka.ms/largemessagesupportforeh.

How do Azure Event Hubs differ from Azure Event Hub Clusters?

Azure Event Hubs and Event Hub Clusters serve different purposes within the Azure ecosystem:

  • Azure Event Hubs: This is a fully managed, real-time data ingestion service that can receive and process millions of events per second. It’s designed for high-throughput data streaming and is commonly used for big data and analytics.
  • Azure Event Hub Clusters: These are dedicated clusters that provide isolated resources for Event Hubs. They offer enhanced performance, scalability, and the ability to handle large messages (up to 20 MB). Clusters are ideal for scenarios requiring high throughput and low latency.

Enjoy!

References

https://techcommunity.microsoft.com/t5/messaging-on-azure-blog/announcing-large-message-support-for-azure-event-hubs-public/ba-p/4146455

https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-quickstart-stream-large-messages

https://learn.microsoft.com/en-us/azure/event-hubs/compare-tiers

Azure Event Hubs Overview

AzureCloud

An overview of Azure Web PubSub | Azure Friday

In this episode of Azure Friday, David Fowler and Liangying Wei join Scott Hanselman to show how to build real-time applications with WebSockets and Azure Web PubSub, a fully managed service that supports native and serverless WebSockets. 

[0:00:23]– Introduction
[0:05:55]– Live chat demo
[0:12:52]– Collaborative whiteboard demo
[0:13:41]– Using with ngrok and CloudEvents
[0:17:30]– Using Azure CLI to broadcast messages
[0:18:45]– Wrap-up

Source: Channel 9

Resources

Development

An Introduction to Windows Azure Service Bus Brokered Messaging

Here is a great post by Mike Wood over on simple talk about Windows Azure Service Bus Brokered Messaging.

Enjoy!