Hydra is an attempt at a new type of messaging system. In traditional push-based systems, clients register with some sort of server infrastructure to say that are interested in particular sorts of messages, and the infrastructure pushes messages out to clients as they are posted. This makes for complex servers that have to know who is subscribed for what sort of messages, how to route messages through the infrastructure, and how to deal with clients that unexpectedly crash without unsubscribing. In a good system the servers will also have some redundancy, so will have to know how to recover from failures, reroute messages to backup servers, and have those servers take over delivery from the failed ones. In contrast Hydra has virtually no infrastructure – it consists simply of a set of CouchDb databases with full-mesh replication. When a message is posted, it percolates by replication to every database; clients that are interested in a particular sort of message simply poll their nearest database for messages of that form – the infrastructure knows nothing about clients at all. If a database goes down, clients simply poll a different one – the messages will arrive there because of full-mesh replication; when the database comes back, replication will ensure that it automatically catches up on any missed messages. No message routing, no client subscription management, no failover rerouting logic.

Components

Hydra consists of two parts. The first is the set of CouchDb databases. These contain a single CouchDb view (not unlike a SQL view), which has a few lines of JavaScript to help clients when they come to poll for messages. They also have full-mesh replication set up (i.e. every database replicates with every other one), which takes a couple of minutes to configure, but then needs no further intervention, ever.

The second part is a C# client-side library to help with sending and receiving messages. This allows clients to send serialised .NET objects as messages and poll for new incoming messages, which are then raised as events. In principle any serialisable object can be sent as a message but Hydra provides a HydraMessage class containing standard fields like Topic, Source, Destination, Handle (for two-way conversations), Data etc. It also provides some standard CouchDb views allowing clients to poll for messages by Topic and Destination so that they can select just the messages of interest to them. All polling and event raising occurs on thread pool threads so does not affect client responsiveness.

Benefits

The outcome of all this is a highly robust messaging platform. By setting up a few replicating CouchDb instances in different locations, the system can tolerate any number of failures and network problems as long as at least one of the instances is reachable. When any failures are fixed, everything just catches up on missed messages and starts working again. So this gives disaster recovery for free – there is no notion of backup system or failing over: every instance is live and you just use whichever ones are most convenient.

A potential side-benefit is that the interface to CouchDb is HTTP, and messages are sent as JSON. This means that clients do not have to be .NET: anything that talks HTTP and JSON can be a client, so you could have a JavaScript client in a web browser if you wanted.

It’s also pretty simple. The client-side library is a couple of hundred lines of code and the server side is a few lines of JavaScript (though new types of message would probably require a few more lines). Fault-tolerant systems usually involve more code and more complexity, but Hydra delegates all of that to CouchDb so remains lightweight.

Problems

I don’t know of any yet, but CouchDb is an unknown quantity with very few tools for management and monitoring. Hydra might just swap the known problems with MSMQ-based systems for as-yet-unknown ones with CouchDb.

Last edited Apr 15, 2012 at 6:21 PM by NickNorth, version 4

Comments

No comments yet.