Thursday, August 22, 2013

RabbitMQ connection_closed_abruptly error

Preface

One of the hiccups we encountered when starting with RabbitMQ occurred when we put a proxy in front of our cluster. The proxy began probing the broker's nodes to ensure they were alive. This resulted in a lot of spam in the logs:

=INFO REPORT==== 21-Aug-2013::11:14:32 ===
accepting AMQP connection <0.12744.35> (<proxy ip>:55613 -> <rabbit ip>:5672)

=WARNING REPORT==== 21-Aug-2013::11:14:32 ===
closing AMQP connection <0.12744.35> (<proxy ip>:55613 -> <rabbit ip>:5672):
connection_closed_abruptly

A Typical(?) Cluster

I'm not certain there is a typical setup for a RabbitMQ cluster. In our case, we went with what we've been calling a binary star. We took this phrase from the 0mq docs. The idea is pretty simple: it's an active/active or active/passive pair with mirrored queues.

A proxy is placed between the clients and the binary star. This separates the clients from the cluster implementation. The biggest benefit for is is the consistent IP for clients. This simplifies the client configurations. It also allows us to perform maintenance on the cluster w/o interrupting service.

RabbitMQ.config

The default Windows location for the config file, rabbitmq.config, is in the %appdata%/RabbitMQ directory. It's possible to change this location, but we'll assume the default location for now. A clean install will often not have this config file.

The first step is to create the rabbitmq.config file. You can create this in any text editor. Just remember that it must be named rabbitmq.config. It may not have a .txt or any other extension. Add the following to the file, and save it:

[
	{rabbit, [
		{log_levels,[{connection, error}]},
	]}
].

This will configure the connection logging to only log errors. The other types of logging, like start up messages, will be unaffected by this.

Making It Take Affect

The RabbitMQ documents indicate that the service should be restarted for the new config to take affect. We have found it necessary to re-install the broker. Fortunately, re-installation of the broker did not wipe the setup. The steps are as follows:
  1. Stop the service in the services management screen.
  2. Uninstall RabbitMQ.
  3. Install RabbiMQ.
  4. Verify the service is running.
More RabbitMQ.config

It's also possible to enable TCP keepalive support, by adding the appropriate line (#11):

[
	{rabbit, [
		{log_levels,[{connection, error}]},
		{tcp_listen_options,
		   [binary,
			 {packet,raw},
			 {reuseaddr,true},
			 {backlog,128},
			 {nodelay,true},
			 {exit_on_close,false},
			 {keepalive,true}]}
	]}
].

Wrap Up

The big gotcha was having to re-install the RabbitMQ service to get the .config file changes to stick. Otherwise, this was a basic config change to ease up on the logging.  Hopefully, this will help someone else...

Wednesday, August 14, 2013

Getting Started With RabbitMQ

Preface

I was tasked with giving a quick RabbitMQ intro to another development group. This post is the result of that task. Most of this stuff will be regurgitated from other sources on the web. It's just meant to help people get up and running.

First we'll look at installing RabbitMQ. Next we'll do some basic configuration, and take a quick peek at the management plug-in. Finally, we'll build a couple simple console apps to use the broker.

RabbitMQ is based on Erlang. There is a Wikipedia article which summarizes what RabbitMQ is. There's also a Google Tech Talk video on YouTube.

This article will assume that you know how to use Visual Studio: create solutions, add projects to solutions, etc. It will also assume that you are familiar with using NuGet.

Getting Ready

Before doing the exercises in this tutorial, you'll need to grab the Erlang and RabbitMQ installers.

Installers
  1. Erlang - R16B01 was used.
  2. RabbitMQ - 3.1.4 was used.
Tools Used
  1. Visual Studio 2012
  2. NuGet
  3. EasyNetQ - v0.11.1.104 was used.
  4. Windows 7 - No, we haven't switched, and we might not. ;P
Installing Erlang and RabbitMQ

The RabbitMQ installation page describes how to install the components. It's pretty short and simple. It amounts to first installing Erlang, then installing RabbitMQ. The default settings should suffice.

The installers should take care of any firewall settings. If they don't, you may see a dialog similar to the one below. Go ahead and add the rules for Erlang.

Windows Firewall dialog for Erlang.


The RabbitMQ management plug-in uses a different port: 15672. If you want to access the management page remotely, you'll need to add the appropriate firewall rules for it.

Copy The Cookie

Once installation is complete, you'll want to copy the .erlang.cookie file to your RabbitMQ profile directory. Doing this now will avoid some headaches later in life, especially if you decide to cluster nodes. It should be located in your %systemroot% directory. Copy it to your %appdata%/RabbitMQ directory. It is important that these files be identical.

Batch Files and Initial Status

The installer should have added items to your Start menu. They'll be located in Start->All Programs->RabbitMQ Server. Open the RabbitMQ Command Prompt (sbin dir) shortcut.

Start menu with RabbitMQ stuff.


This will open a command prompt. A number of operations can be performed with this console window: starting/stopping the broker, enabling plug-ins, etc. Checking the broker's status can be done by entering 'rabbitmqctl status' at the prompt.

Sample 'rabbitmqctl status' output.


Enabling the Management Plug-in

The next step to perform is to enable the management plug-in. This plug-in provides a web interface for controlling the broker. It can be used to manage users, exchanges, etc. The management plug-in can be issuing the commands illustrated in the following image. Also note that in most cases, I've found it necessary to restart the RabbitMQ service from the services control panel.

Enabling the management plug-in.

The commands are (in order):

  1. rabbitmqctl stop_app
  2. rabbitmq-plugins enable rabbitmq_management
  3. rabbitmqctl start_app


Using the Management Web Page

The plug-in docs describe many of the features available. We're going to use the UI to create a test user and a virtual host. Later we'll use it to see the broker at work, create a queue, and alter some bindings.

You can access the management UI by opening a browser and navigating to http://localhost:15672. You'll be prompted with the login screen. The default username and password are guest/guest. There are a number of tabs available after logging in. The first is the Overview:

RabbitMQ management Overview.


First we'll create a virtual host for testing. Virtual hosts are managed in the Virtual Host tab of the Admin screen. We'll create one called Sandbox:

RabbitMQ management UI Virtual Hosts screen.


Users are managed in the Users tab of the Admin screen. Expand the 'Add a user' accordion and enter a name along with a password. For this example, I used SandboxUser as the username, and sandbox as the password:



Users must have permission to access virtual hosts. There are a few ways to do this, but we'll use the Virtual Hosts tab of the Admin screen. Click on the Sandbox virtual host and expand the 'Permissions' accordion. Select the user name in the 'User' drop down and click the 'Set Permission' button. Do this for both the SandboxUser and guest users:



The Sample Apps

The source code in this article can be found on GitHub. The solution is divided into three projects: Publisher, Subscriber, and Messages. The name of each project implies what it does. I'm bypassing use of an IoC or test projects for simplicity. Hopefully, I'll get some time in a future post to illustrate how one can mix an IoC container and do some TDD...

Here's what it looks like:

Sample solution contents.


The publisher sends messages to the broker. The subscriber listens to the broker for a message. The message contract is defined in the Messages project. It is referenced in both the Publisher and Subscriber.

The publisher and subscriber projects both reference a library called EasyNetQ. EasyNetQ provides a fairly simple API for interacting with a RabbitMQ broker. At the heart of this is the IBus interface. IBus is used in publishing and subscribing to messages. Our apps use a BusFactory class to create an instance of this interface:

    public static class BusFactory
    {
        public static IBus Create()
        {
            var settings = ConfigurationManager.ConnectionStrings["rabbit"];

            if (settings == null || string.IsNullOrEmpty(settings.ConnectionString))
                throw new InvalidOperationException("Missing connection string.");

            return RabbitHutch.CreateBus(settings.ConnectionString);
        }
    }


The BusFactory class uses a connection string defined in the app.config:

    <connectionStrings>
        <add name="rabbit" connectionString="host=localhost;virtualHost=Sandbox;username=SandboxUser;password=sandbox;prefetchcount=1;" />
    </connectionStrings>


The heart of our demo is in the DemoPublisher. This class illustrates the heart of the publishing operation. A message is created, a publishing channel is opened, and the message is published.

    public class DemoPublisher
    {
        private readonly IBus bus;

        public DemoPublisher(IBus bus)
        {
            this.bus = bus;
        }

        public void Publish()
        {
            var message = new ExampleMessage { Greeting = "Hello, world!" };

            using (var channel = bus.OpenPublishChannel())
                channel.Publish(message);
        }
    }


At the other end of the demo is the subscriber. The DemoSubscriber.ListenForAMessage() method uses an instance of IBus to notify the broker that it is listening for messages of type 'ExampleMessage'. When a message of that type is received, it writes the greeting to the console, and marks the done flag as true.

    public class DemoSubscriber
    {
        private readonly IBus bus;

        public DemoSubscriber(IBus bus)
        {
            this.bus = bus;
        }

        public void ListenForAMessage()
        {
            var done = false;

            bus.Subscribe<ExampleMessage>("subscriber", message =>
            {
                Console.WriteLine(message.Greeting);
                done = true;
            });

            SpinWait.SpinUntil(() => done);
        } 
    }


Running the Publisher

When the publisher is run by itself for the first time, EasyNetQ will create an exchange. However the message sent to the broker seems to disappear. This is because there is no queue defined to receive it. We can create a test queue to catch the message.

First we create a queue:

Creating a queue in the management UI.


Then we bind the exchange to the queue. This can be done either in the queue detail view, or in the exchange detail view. This example shows it being done in the queue detail view. This view is reached by clicking on the queue name, after it's been created. It's important to note that the name must match exactly. It's okay to leave the 'Routing key' and 'Arguments' sections blank.

Binding an exchange to a queue in the management UI.


When the publisher sends a message to the broker, that message is directed to the queue, TestQueue.

Showing the message count in a queue.

Running the Subscriber

When EasyNetQ makes a subscription, it creates the queue and automatically binds it to the exchange. We can see the results both in the console output, and in the queues on the management page:

Showing the sample output from two console apps; one publisher and one subscriber.

Showing the queues with both apps active.

Run the Subscriber First

In creating the subscriptions and publishing the message, EasyNetQ handles the creation of the exchanges and queues for you. Be default, RabbitMQ will discard messages for which there is no destination queue. Thus, you will want to ensure that the subscribers are initialized, before you publish a message.

Wrapping It Up

That's about it for creating a simple demo for RabbitMQ. There are a lot of topics which we haven't covered: clustering, federations, high availability to name a few. Although a little dated, Manning's RabbitMQ in Action: Distributed Messenging for Everyone is worth a read. The documentation for EasyNetQ and MassTransit are other good references.