How RabbitMQ, Kafka, and Azure Service Bus Fit into Enterprise Integration

The decision usually arrives late. Your ERP already talks to the CRM through a nightly file drop. The HCM system pushes worker changes through a vendor webhook that nobody monitors. Then someone asks for inventory visibility across three warehouses within a minute of a scan, and the existing pattern will not stretch that far.

At that point the question becomes which broker to standardize on, and it tends to get answered in a hurry. RabbitMQ, Apache Kafka, and Azure Service Bus all move messages between systems, but they are not interchangeable. Each makes a different promise about ordering, retention, routing, and who carries the operational load. Choosing badly is rarely fatal. It is expensive, because teams end up writing application code to compensate for the shape of the broker they picked.

This article covers what each product does well, where each one fails, and the operational details that only surface after go live. It closes with a decision framework organized by requirement rather than by vendor, plus guidance for the common case where you end up running two of them.

Why Enterprise Integration Outgrew Point to Point Connections

A finance team asks for one integration. Engineering wires the ERP directly to the tax engine over REST, and it works fine. The problem is not the first connection. It is the eleventh.

The math is unforgiving. Twelve systems that each need to talk to each other produce sixty six possible connections, and each one carries its own authentication, retry logic, error handling, and schema assumptions. Even a realistic estate that only wires up a third of those pairs ends up with more integration code than the applications it connects. No single team owns that code, so nobody upgrades it.

Direct calls also couple availability. When the order service calls the tax engine synchronously and the tax engine slows to eight seconds per request, the order service exhausts its thread pool and stops accepting checkouts. The failure moves upstream, away from its cause. Add naive retries and the slow system now receives three times the load it was already failing to handle, which is how a degraded dependency becomes a full outage.

A broker breaks that chain. The producer writes to the broker and moves on; the consumer reads when it is healthy. This changes the failure mode from cascading outage to growing backlog, which is a far better problem to have at 2 a.m. If you are still deciding whether a given interface belongs in this pattern at all, the practical distinction between how synchronous and asynchronous integration differ is the right place to start.

The broker is not free. You are adding a tier that has to be sized, patched, monitored, and understood by people on call. That cost is only worth paying when the coupling problem is real.

The Three Messaging Models You Are Actually Choosing Between

The first decision is not RabbitMQ versus Kafka. It is which delivery model matches the work in front of you. A broker moves and stores messages; it does not decide business sequence, compensate for a failed step, or map fields between two systems. Keeping that boundary clear matters, and data orchestration differs from simple transfer in ways that change which tool you need.

Queue Based Message Delivery

A queue holds work items until a consumer takes one. The broker tracks per message state, delivers each message to exactly one consumer in a competing consumer pool, and removes it once acknowledged. Routing decisions happen in the broker, so a message can be filtered or fanned out before any consumer sees it. This model suits commands and tasks: post this invoice, provision this account, send this notification.

Distributed Commit Log and Event Streaming

A log is an append only sequence, split into partitions, that consumers read by position. Reading does not delete anything. Messages age out on a retention policy, not on acknowledgment, so five different consumers can read the same events independently and a sixth can replay them next quarter. The broker stays simple and the consumer tracks its own position. This is the model behind most streaming work, and the trade offs are the same ones covered in event driven architecture versus request response.

Managed Cloud Messaging as a Platform Service

The third model is not a different delivery semantic. It is a different ownership model. You get queue and topic semantics with a service level agreement, cloud native identity, private networking, and no cluster to patch. You give up version control, tuning depth, and portability. For a team of four engineers supporting thirty integrations, that trade is often the correct one.

RabbitMQ in Enterprise Integration

Where RabbitMQ Fits Best

RabbitMQ implements AMQP 0-9-1, where publishers send to an exchange rather than a queue. The exchange applies a routing key against its bindings to decide which queues receive a copy. A topic exchange with a key like invoice.us.overdue lets you bind one consumer to invoice.us.# and another to invoice.#.overdue without the publisher knowing either exists. That per message routing is the single most useful thing RabbitMQ does, and neither of the other two products matches it.

It fits task distribution, request and reply patterns, and heterogeneous estates where a mix of AMQP, MQTT, and STOMP clients need one broker. Latency is low for small messages, and dead letter exchanges give you a clean place to route messages that expired, were rejected, or exceeded a delivery limit. Publisher confirms close the other half of the loop: the broker asynchronously tells the publisher a message was accepted and persisted, which the RabbitMQ documentation covers in its guide to publisher confirms and consumer acknowledgements.

Operational Realities and Limits

For anything that must survive a node loss, use quorum queues. The RabbitMQ documentation on replicated quorum queues states that classic queue mirroring was removed entirely in RabbitMQ 4.0, so quorum queues are now the replicated queue type. They use Raft consensus and need a majority of replicas available, which means an odd cluster size and real disk performance, since every write goes to disk before it is confirmed.

Two defaults catch teams out. RabbitMQ 4.0 sets a delivery limit of 20 on quorum queues, where previously redelivery was unbounded; if you have not configured a dead letter exchange, a message that fails 20 times is dropped rather than parked. Separately, classic queue lazy mode is gone: since 3.12 the version 2 classic queue behaves lazily by default, so tuning advice written against older lazy queue settings no longer applies.

The failure you will actually see is a consumer that stops acknowledging. It stays connected, so nothing alerts, but its unacknowledged count climbs to its prefetch limit and then it takes no more messages. Meanwhile the queue depth grows and, if you left prefetch unlimited, that one stuck consumer already holds every message while healthy consumers sit idle. Eventually the node hits its memory alarm and blocks publishers, which surfaces as write timeouts in an application nobody connected to the queue. The fix is a modest per consumer prefetch, a delivery limit paired with a dead letter exchange, and an alert on unacknowledged count rather than queue depth alone.

RabbitMQ is a poor fit for replay and audit. Once a message is acknowledged it is gone, so there is no going back to reprocess last Tuesday. It is also weak for very high sustained throughput on a single queue, for large payloads, and for reliable cross region replication, where federation and shovel are asynchronous and eventually consistent by design.

Apache Kafka in Enterprise Integration

Where Kafka Fits Best

Kafka writes to a partitioned, append only log. Each topic is split into partitions, each partition is an ordered sequence, and consumers in a consumer group each own some partitions and commit offsets marking their position. Retention is a policy, not a side effect of consumption, so a new consumer can start at the beginning of retained history without coordinating with anyone.

That makes it the right choice for change data capture off ERP databases, for audit trails you must be able to reconstruct, and for cases where five teams want the same event stream for different reasons. Log compaction adds a second mode: instead of discarding old records by age, the broker retains the most recent record per key, which gives you a rebuildable snapshot of current state per employee, per SKU, or per account. Broker level durability comes from replication and the in sync replica set, and the Apache Software Foundation covers the acknowledgment settings in the Apache Kafka project documentation. Producing with acks set to all against a replication factor of three with a minimum in sync replica count of two is the usual enterprise baseline.

Operational Realities and Limits

Ordering is guaranteed within a partition, never across a topic. Since the message key determines the partition, ordering per customer or per order is achievable and global ordering is not. Partition count is a one way door in practice: you can add partitions, but doing so changes the key to partition mapping, so records for an existing key start landing somewhere new and ordering for that key breaks at the boundary.

Consumer group rebalancing is where duplicates come from. A consumer that takes longer than max.poll.interval.ms to process a batch stops sending heartbeats to the group coordinator, gets evicted, and its partitions are reassigned. The new owner resumes from the last committed offset, so every record processed since that commit is processed again. The symptom is duplicate rows appearing in bursts under load, not steadily. The fix is smaller poll batches, moving long work off the poll thread, idempotent consumers, and the newer rebalance protocol that became generally available in Kafka 4.0 and reassigns partitions incrementally instead of stopping the whole group.

Operating a cluster is a real job. Kafka 4.0 removed ZooKeeper and runs on KRaft only, which removes one system to babysit but does not remove capacity planning, rack aware replica placement, retention sizing, Connect workers, or schema governance. Consumer lag is the metric that matters most and the one teams forget to alarm on, which is exactly the discipline described in setting alerting thresholds before integrations fail.

Kafka is a poor fit for low volume workloads, where a three broker cluster to move four thousand messages a day is pure overhead. It is also poor at per message routing: consumers read whole partitions and filter in application code, so there is no server side selector. It has no native per message scheduling, no priority, and no way to parallelize a consumer group beyond the partition count.

Azure Service Bus in Enterprise Integration

Where Azure Service Bus Fits Best

Service Bus gives you queues for point to point work and topics with subscriptions for publish and subscribe, where each subscription can carry SQL style filters that select only matching messages. That server side filtering means a subscription for high value orders never sees the rest. The default receive mode is peek lock, which hides the message from other receivers while you process it and requires an explicit completion; receive and delete removes it immediately and is only appropriate when losing a message is acceptable.

Sessions give ordered, single threaded processing for a related group of messages. Set a session identifier, and one receiver holds a lock over the whole session and processes it in order. Duplicate detection is built in, and Microsoft’s duplicate detection documentation states the history window defaults to ten minutes, with a minimum of twenty seconds and a maximum of seven days. Scheduled delivery, deferral for messages you want to set aside without losing them, and a dead letter queue attached to every entity round out a feature set that suits business process integration.

Operational Realities and Limits

Tier differences are not cosmetic. Microsoft’s published Service Bus quotas and limits put the standard tier message size at 256 KB and cap it at 1,000 operations per second, while premium removes the fixed operations limit and scales with messaging units. Payloads above 1 MB require the premium tier over AMQP, and the premium messaging documentation describes support for single messages up to 100 MB even though the default per entity maximum is 1 MB.

The lock expiry problem is the one you will meet first. Lock duration cannot exceed five minutes, and if your handler runs past it without renewing, the lock is lost, the message returns to the queue, and the delivery count increments. The work already committed downstream is not rolled back, so you get a duplicate. The default maximum delivery count is 10, after which the message moves to the dead letter queue. The symptom is a matched pair: a record written twice, and a copy sitting in the dead letter queue an hour later. Renew the lock explicitly for long handlers, or shorten the unit of work so it finishes well inside the window.

Sessions have a sharper edge. The lock covers the session, not the individual message, so a poison message at the head of a session blocks every later message carrying that session identifier until it dead letters or the lock expires. One malformed order can stall a single customer’s queue for an hour while every other customer flows normally, which makes the incident very hard to see on aggregate dashboards. Watch the age of the oldest message per session enabled entity, not just total depth.

The honest limitation is ecosystem gravity. Identity, networking, monitoring, infrastructure as code, and function bindings all assume Azure. That is a genuine advantage if you are already committed, and a genuine liability if a business unit runs on another cloud. Service Bus also does not replay: once a message is completed it is gone, so it is the wrong tool for event sourcing or analytics backfill.

Choosing Between Them: A Practical Decision Framework

Work the requirements first and let the product fall out of them. In order of how often they prove decisive:

• Ordering: do you need order per entity, per partition, or not at all
• Replay: must a consumer reprocess history after it was already consumed
• Throughput profile: steady high volume, or bursty and modest
• Routing: does the broker need to decide who gets what
• Team capacity: who is on call for the broker itself at 3 a.m.
• Constraints: existing cloud commitment, data residency, and audit requirements

If you need replay or audit reconstruction, the log model is the only one that provides it, and that decision is usually made before any other. If you need server side routing or per message scheduling, the log model is the wrong shape and you will rebuild those features badly in application code. If your throughput is modest and your team is small, a managed service removes work you were never going to do well anyway.

Data residency and compliance narrow the field faster than most teams expect. A self managed broker can run in any region or datacenter you control, while a managed service runs where the provider offers it and encrypts under the provider’s key hierarchy unless you configure customer managed keys.

Total cost of ownership is where the comparison usually goes wrong. Kafka has no license cost and is frequently the most expensive option, because a production cluster needs someone who understands partition rebalancing, retention sizing, and broker replacement at three in the morning. That is a specialized skill set, it commands a premium in the US market, and one person holding it is a single point of failure. A managed service converts that headcount into a monthly bill that is easier to forecast and harder to hide. Compare fully loaded cost including on call rotation, upgrade cycles, and the integration work each option avoids, which is the same discipline used when quantifying the return on integration investment. Consumption pricing on any cloud service is subject to change, so model a range rather than a point estimate.

Running More Than One Broker Without Creating a Mess

Most enterprises above a certain size end up with two. A Kafka cluster arrives with an analytics or CDC project, Service Bus arrives with an Azure application team, and RabbitMQ was already there under a product nobody wants to touch. This is not automatically wrong. It becomes wrong when no boundary is drawn.

Draw the boundary by purpose. Streaming, replay, and analytics belong on the log. Transactional business process work with routing and dead letter handling belongs on the queue. Write that down and enforce it at design review, because the alternative is two systems that both half own order events.

Bridge in one direction only. A Kafka Connect sink writing selected topics into a Service Bus queue is understandable; a bidirectional bridge creates loops that are very hard to debug once a message re-enters its source. Give the bridge an owner, a schema contract, and its own dead letter path, and make the mapping between a Kafka key and a Service Bus session identifier explicit rather than implicit.

The observability problem is the real cost. Consumer lag, queue depth, and dead letter counts live in different tools with different retention and different alerting, so nobody can answer where a specific order went. Propagate a correlation identifier through every hop, log it at every consumer, and build one dashboard that shows depth, lag, and dead letter age for all brokers side by side. Before any of this, find out what you already run, because auditing an existing integration estate routinely turns up a broker or two that no current team remembers deploying.

Getting the Implementation Right

Assume every message will be delivered more than once. All three products give you at least once delivery in practice, so consumers need an idempotency key and a record of what they have already applied. This single decision removes most of the pain from rebalances, lock expiries, and redeliveries.

Dead letter handling needs a named owner and a runbook, not just a configured destination. Decide who reads it, how often, and what the replay path looks like before go live, because an unmonitored dead letter queue is just a slower way to lose data. Retries should use exponential backoff with jitter so a recovering downstream system does not get hit by every client at once.

Set thresholds on the metrics that predict failure rather than confirm it: age of the oldest message, consumer lag trend, dead letter arrival rate, and unacknowledged count. Document message contracts before anyone writes a consumer, including required fields, versioning rules, and what happens to unknown fields, in the same way a production grade integration requirements document captures them. If your team needs help designing the broker topology or building the consumers, our custom integration development support covers that work.

Frequently Asked Questions

Should we start with Kafka or RabbitMQ for our first broker?

Start with the model your workload needs, and if that is unclear, start with queues. RabbitMQ has a shorter path to production, a smaller operational footprint, and enough routing flexibility to cover most first integrations. Kafka pays off when you need retained history or several independent consumers of the same stream, and its operational demands are hard to justify before that. Moving from queues to a log later is normal; running an underused cluster for two years is harder to defend.

Can Azure Service Bus replace Kafka?

Not for streaming workloads. Service Bus does not retain messages after completion, so there is no replay, no offset rewind, and no way for a new consumer to read history. It also has throughput ceilings that streaming workloads pass quickly. If your requirement is durable business messaging with routing, filtering, and dead letter handling, it replaces Kafka comfortably. If your requirement is event sourcing, analytics ingestion, or change data capture, look at Azure Event Hubs or a Kafka cluster instead.

Is exactly once delivery achievable in practice?

Not end to end, in the general case. Kafka offers exactly once semantics within its own transactional boundaries, and Service Bus deduplicates by message identifier inside a configured window. Neither covers what happens when your consumer writes to an external database and then fails before committing an offset. The workable pattern is at least once delivery plus idempotent consumers keyed on a stable business identifier. Teams that chase exactly once at the transport layer usually spend more than teams that build idempotency once.

When is a broker overkill?

When you have fewer than about six integrations, latency requirements are relaxed, and volumes are low. Two systems exchanging a few thousand records a day through a well monitored batch process do not need a message broker, and adding one introduces a component to patch and monitor for no reduction in coupling. Brokers earn their keep when connection count grows, when producers and consumers scale independently, or when a slow downstream system is taking a fast upstream one down with it.

What actually drives messaging costs?

For managed services, message operations, namespace tier, and retained data drive the bill, and premium tiers price on reserved capacity rather than per message. For self managed brokers, the drivers are storage, network egress between availability zones, and engineering time. Replication factor multiplies storage directly, and long retention on Kafka is often the largest single line item. Confirm current figures against the official pricing pages, since rates and tier definitions change.

Can we run Kafka on Azure?

Yes, through several routes with different trade offs. You can self manage brokers on virtual machines or Kubernetes, use a vendor managed Kafka service available through the Azure marketplace, or use Azure Event Hubs, which exposes a Kafka compatible endpoint that most producer and consumer clients can use without code changes. The Event Hubs route removes cluster operations but does not support every Kafka feature, so validate the specific APIs your applications rely on before committing.

How hard is it to migrate between brokers?

Harder than the client library swap suggests. Ordering semantics, redelivery behavior, and dead letter mechanics differ enough that consumer logic usually needs reworking, and any queue depth you are carrying has to be drained or replayed during cutover. The practical approach is dual publishing to both brokers, migrating consumers one at a time, and keeping the old path live until traffic is fully shifted. Budget for a parallel run measured in weeks, not a weekend cutover.

What should we expect around security and encryption?

All three support transport encryption and encryption at rest, with the differences appearing in identity and key management. Service Bus integrates with Microsoft Entra ID for role based access and supports customer managed keys and private endpoints on premium. Kafka and RabbitMQ support TLS and SASL or certificate based authentication, but you configure and rotate it yourself. Whichever you choose, apply least privilege at the topic or queue level rather than granting namespace wide credentials to every application.

What staffing does each option realistically require?

A self managed Kafka cluster needs at least two engineers with real cluster experience, so that upgrades and incidents do not depend on one person. RabbitMQ is lighter but still needs someone who understands quorum queue behavior, memory alarms, and cluster partitions. A managed service shifts most of that to the provider, leaving your team responsible for entity design, consumer reliability, and monitoring. The support model matters as much as the technology, and it should be settled before the platform decision is final.

We keep returning to the same conclusion on client engagements: the broker matters less than the discipline around it. Teams with idempotent consumers, documented contracts, owned dead letter queues, and alerting on the right metrics succeed on all three of these products. Teams without those things struggle on all three. Pick the model that matches your ordering, replay, and routing requirements, be realistic about who will operate it, and revisit the choice when your volume profile changes. If you want a second opinion on a topology before you build it, we are happy to look.