Concepts

This chapter contains a brief overview over the concepts and technologies used in Clacks 1.0.

Messaging with AMQP

What is AMQP?

AMQP is the short form for Advanced Message Queueing Protocol and provides a middleware for message oriented applications on the OSI application layer. It provides a secure and reliable way to deliver, forward and store messages.

AMQP is a binary protocol which assures the interoperability between different platforms. Size and form of messages do not matter.

The AMQP infrastructure consists of at least one broker - that’s the instance where messages get processed - and a number of clients. These clients can publish or consume messages. Clients wich consume messages are named consumer and clients publishing messages are named producer.

Note

For more detailed introduction, please refer to the exellent Wikipedia article.

Queues

The base for communication with messages are so called queues. Messages in queues get stored in memory or on disk and will be transferred (in sent order if applicable) to the consumers. Queues are message store and distribution unit in one. Message queues do not depend on each other and can have different properties: private or public, permanent or temporary, permanent or volatile. These properties make a determination whether the queue implements a store-and-forward, a pub-sub or what ever mechanism.

If you compare queues to SMTP, the queues would be like post boxes. The mail sender is a producer and a mail filter the consumer.

Queues can be configured to realize one-to-one, one-to-any, point-to-point and publish-subscribe mechanisms. For members of the AMQP network, it may be interresting if a disk of a connected system is running out of space. In this case you can create some kind of status queue which is taking your systems disk status messages. Interrested consumers can inspect incoming messages and trigger an automatic cleanup or send a short message to the administrator.

Queues can be mirrored on different brokers in a cluster - while this kind of clustering is called federation.

Exchange

An exchange is an instance of the AMQP concept, which can connect queues by several criteria - it provides some kind of routing mechanism for messages. The process of connecting queues is called binding.

AMQP provides a couple of exchanges. One way would be to assign messages and queues be the routing key, or by using an XQuery for XML based messages.

Clacks itself uses two kinds of exchanges: the XML exchange for events, to allow dedicated filtering and the routing key based exchange to provide round robin command queues.

Messages

Sent information consists of AMQP header information and the message body. The contents of the message body is not specified - if you want to you can transport DVD images via AMQP. In case of Clacks the message body consists either of XML messages or JSON encoded RPC calls.

The ‘configuration’ on how to transport messages is done by the header specification.

Broker

Currently, one of the biggest drawbacks of AMQP is that there are different specifications in different states of development available. While there’s a version 1.0 of AMQP available now, there’s no broker which is implementing it right now. There are brokers to support 0.6, 0.7, 0.8, 0.9 etc. and all of them tend to be not 100% compatible to each other.

Here’s an (incomplete) excerpt of available OSS and non OSS brokers:

  • OpenAMQ

    This free broker has been developed by the former creators of AMQP. It’s done in C and supports interresting features like REST based messaging. Sadly it looks like they’ve abandoned OpenAMQ and the company behind it (iMatix) seems to focus on 0mq - another messaging system.

    OpenAMQ only supports older versions of the AMQP standard.

    OpenAMQ does not support any authentication or authorization mechanisms.

  • RabbitMQ

    RabbitMQ is a free broker done in Erlang. It’s maintained by Rabbit Technologies Ltd.

    At the time of writing, they only supports older versions of the AMQP standard.

  • QPID

    QPID is a free broker under the hood of the apache project. They provide a C and a Java based broker. It supports SASL based authentication and access control to queues.

    They support content based filtering via XQuery and it seems to be the most up-to-date broker in concerns of AMQP specifications.

  • Red Hat Enterprise MRG

    This is the commercial version of QPID, maintained by RedHat.

Clacks is using the C broker of QPID, because it makes life easier in several places.

Authentication

Using QPID, we can rely on a SASL based authentication wich can be simply connected to the LDAP directory infrastructure. The authorization itself must be done by Clacks, because the QPID acl concept is not dynamic - mostly for reasons of performance.

The AMQP broker can automatically store the authentication information in the AMQP headers, so that Clacks is able to notice who’s sending messages.


Clacks overview

This overview should provide a big picture of the Clacks components and describes the tasks the components are supposed to do. More detailed information can be found in the developer documentation.

Domains

Systems connected with Clacks components shape a so called domain. If you like the analogy, you can compare a Clacks domain to a Windows domain: it keeps some kind of information about a delimited area of an organization (or in your opinion the world).

A domain is basically constructed of a reverse DNS name - i.e. the default domain is org.clacks, but it could be de.gonicus or whatever you want it to be. AMQP queues are based on the domain, so if you use the default, all created queues start with org.clacks. and shape the namespace to use in AMQP.

clacks.common

The clacks.common component is the base library which is providing common functionality to agents and clients.

clacks.agent

For every domain, you need at least on broker (or a broker federation) and at least one clacks.agent instance. The agent defines the domain queues and puts life to queues by answering to command requests and processing basic events.

If you have more than one agent, they share the command queues using a round robin method of the AMQP broker - so you’ve automatic load balancing. Agents notify each other on a regular base in order to know each others status (load, number of workers, etc.). They also notify each other if they’re joining or leaving the domain - maybe due to a service restart. Because clacks.agent instances can have different plugins loaded, they also inform each others about their capabilities - so that in case it’s needed - commands can be forwarded to a capable agent.

All commands are registered in the CommandRegistry and can be simply accessed from there.