Thursday, February 6, 2014

RabbitMQ Federated Queues


RabbitMQ added support for federated queues in v3.2.0. This feature gives a simple way to move messages from one Rabbit cluster to another. I'll show you one way to set this up. The sample code can be found on GitHub. I'm using EasyNetQ to handle the publishing and subscription. It's a very nice RabbitMQ client library. Check it out.

The Clusters

Note: The virtual host names are not the same on the two clusters. Broker A is using the virtual host FederationDemo. Broker B is using the virtual host FederatedStuff.

The RabbitMQ documentation does cover the federation plug-in.  In our scenario, there are two clusters. Each is an upstream for the other. Message hops are at 1 to prevent the messages from circling back to the publisher. Below are pictures of the upstreams defined on each of the clusters.

Shows the upstream definition on Broker A.
Broker A Upstream

Shows the upstream definition on Broker B.
Broker B Upstream

Each broker will need to have a policy defined. This policy is used by the broker to figure out what things come from the upstream bluster. The policies are very similar for both clusters. Below are pictures of the policies:

Federation Policy on Broker A

Federation Policy on Broker B

The Clients

This example will use two console applications. Each will subscribe to and publish two messages. The messages will ping-pong between the two clients. The first client, FooConsole, will publish two messages: start, and stop. The second client, BarConsole, will publish the following: started and stopped.

This example uses two clients. A message from one client will be transported to another. Then a response message will be transported back to the original client.The message sequence is Start, Started, Stop, and Stopped.

I've put the publishing and subscriptions into one class, so we could see everything going on. Both classes, Foo and Bar, are very similar. Here's a look at the Foo class:

Foo subscribes to two messages: Started and Stopped. It then publishes a Start message to get the ball rolling. On the other end, Bar subscribes to Start and Stop messages. It responds to each message with one of its own messages. A Start from Foo causes Bar to send Started. A Stop from Foo causes Bar to send a Stopped message.

Wiring Two Joints

When an app uses EasyNetQ to subscribe, EasyNetQ creates the queues and exchanges for us. This doesn't happen when working with federated queues. The queues will be federated, but there will be no bindings made on the upstream cluster. The pictures below show the downstream and upstream clusters after a client subscribes to a message.

Upstream (publisher) Federated Queues

Downstream (subscriber) Federated Queues

The last little bit is to bind the exchange to the queue on the upstream cluster. This allows the messages to flow from the upstream publisher to the downstream subscriber.

Federated Queue Without Binding
Publishing Exchange Bound To Queue

With everything put together, it's now possible to show the two console apps passing messages across the broker. Note: I've disabled EasyNetQ's default debug logging by using a null logger.

The Console Display

Wrapping It Up

The addition of federated queues to RabbitMQ really simplifies transporting messages between clusters. Hopefully this helps show how you can get two clients to communicate across a federation.