Microsoft Embraces AMQP Open Middleware Standard 122
AlexGr writes to tell us that Microsoft apparently has plans to embrace a little known messaging standard called AMQP (Advanced Message Queuing Protocol). Red Hat, a founding member of the AMQP working group, was very excited about the news and wrote to welcome Microsoft to the party. "Suffice it is to say that AMQP is to high-value, reliable business messaging what SMTP is to e-mail. The proprietary message oriented middleware (MOM) products on the market today like IBM's MQ or Tibco's Rendezvous fulfill the same function as AMQP. But they operate exclusively in single-vendor fashion and utterly fail to interoperate with each other. They are also — perhaps not by coincidence — burdensomely expensive. As a result their use is mostly limited to wealthy organizations such as Wall Street banks (at least the ones who are still in business) that need to exchange huge volumes of business messages very reliably and very quickly. But AMQP's supporters feel the market for such reliable messaging could be much larger if a less expensive and truly open solution became available."
What happened to their old product? (Score:4, Informative)
I can't even remember it's name, although it's installed here somewhere.
The IBM MQ product is actually OK to use (very simple, lots of platforms supported) and especially double plus good if you have an IBM mainframe somewhere on your network.
TIBCO? Shudder. About three quarters of the money they charge goes towards getting your manager drunk enough to sign the purchase order. The product itself isn't worth a damn.
Re:What happened to their old product? (Score:3, Informative)
You mean MSMQ [wikipedia.org]?
Re:What is this anyway? (Score:3, Informative)
Because most competent middlewares supports transaction handling, synchronous and asynchronous, triggering, queueing and prioritizing of messages.
SOAP and XMLRPC doesn't really cut it when you need proper transaction based messaging. I've seen some bastard solutions for this but they perform poorly.
Re:What is this anyway? (Score:3, Informative)
Architectures using messaging are more loosely coupled than those using SOAP or XMLRPC. Essentially, every component is a messaging client, asynchronously consuming and/or producing messages that get posted to specified channels. None of the clients need to know anything about any of the other clients, only what types of messages are acceptable to be put on what channels, and what channels to listen on for types of messages they are interested in.
This makes it easier to add intermediate processing components, to scale out the number of "servers" (though they are actually a kind of clients) handling particular types of messages, etc., since multiple clients can listen on the same channels (though only one will receive each message, for a normal, queue-like channel).
Re:What is this anyway? (Score:5, Informative)
Middleware solutions also come with tools for monitoring, configuring, etc., that simpler solutions often do not. This being said, I've had my fair share of problems with middleware in the past. Like any complex piece of software, you get a lot of functionality, but you also get a lot of baggage and hidden pitfalls that can come back to bite you when your usage of that functionality becomes demanding.
Re:What is this anyway? (Score:3, Informative)
Well, middleware tends to have a bus publish/subscribe architecture. So you throw a whole lot of data onto the bus, and you don't really care where it goes. Clients are clients of the bus, and generally register for data that they're interested in.
The standard example for this kind of stuff is stock trading. Your stock app basically dumps all the transaction information onto the bus in realtime, and clients register for notification of trades on specific ticker symbols or some kind of criteria (>1000 shares, price), or any combination of things.
As long as the message contents are standardized, anyone can listen (or post)...assuming they have the correct permissions, etc.
This doesn't sound hard, until you realize that comes to around 10379 trades/sec given today's DJIA trading volume.
The bus architecture seems to have scaled up pretty well. I remember when trading volumes were a lot less. Note that the bus (solution) generally:
* requires transactions
* requires guaranteed delivery
* requires a guaranteed latency time
* requires absolute uptime during its operational period
* requires that data never be lost, ever
Lots of features fall out from those, such as distributed architecture, redundancy, failover, hot standby, replication, fault tolerance, etc.
Luckily, the realtime platform only needs to run during market hours (and some period before/after). So technically you only need about 6.5 hours of uptime per day (make it 10 hours for settlements, etc).
Why would you use a bus? Because it means you don't have to care how clients connect to you. Throw your message onto the bus, and that's all you need to worry about. You don't have to worry about transaction support, security, all that other crud. Your app's job is to connect to the bus and feed it data.
Likewise, the client just needs to hop onto the bus and register for the messages that it cares about (and presumably understands). You'd think you'd lose performance because it's so decoupled, but you don't.
Re:XMPP (Score:5, Informative)
AMQP seems more "enterprisey" than XMPP, focussed on traditional enterprise applications of messaging, XMPP came up from the IM world. This gives them different areas of focus, though they have considerable overlap in functionality. (There are also a number of other protocols in the messaging space: STOMP & Apache ActiveMQ's OpenWire, and most of the big commercial messaging systems use their own proprietary protocols.)
JMS is an API, XMPP is a protocol. There are JMS implementations that can use XMPP as a transport protocol (Apache ActiveMQ, for instance, can use its native OpenWire protocol, STOMP, or XMPP).
"Reliable" messaging usually means that once the messaging system has acknowledged receipt of the message, it has assumed responsibility for guaranteeing delivery to a suitable endpoint (possibly with some caveats). Mostly, it means in practice that the protocol or implementation provides (or, in the case of a protocol, either mandates or supports in a way which permits clients to discover whether the server is using) channels whose contents are persisted to permanent storage before they are acknowledged.
Re:What is this anyway? (Score:5, Informative)
Let's imagine that your organization uses a few different applications. For example a core banking, a loan management, credit card management, call center, billing etc..
Now let's say that you would like to be able to do "fun" stuff when something "cool" happens in one of those applications. For example, when someone default on a loan payment you'd like the core banking to flag the account as "dangerous", block his credit card, cancel his vip status in the CRM, the call center will generate an automated call to connect you to a customer service rep from the debt collection department that will menace you with the word foreclosure.
The first way to do that is to create many one-to-one relationship between all those application. "onpaymentdefaultduedate" in the loan management you could call successively each API of each application to "make it" do whatever it is that needs to be done. So the loan management will need to know about the crm, the call center, the billing etc...
Problem is that the next time you add a fancy new application (SMS harassment gateway for example) or upgrade an existing one (API change yeah!) you'll need to upgrade all the applications that have a one-one relationship. Which mean bringing on board the vendors of each and every applications for top dollars. Plus after a few years, everyone will have forgotten how the things even works. The vendor does not provide version 2 customization and you'll have to upgrade everything to version 7.3.56 and retrain your staff because the interface has changed and you can't expect the tellers to figure it out by themselves.
The second option is to add a middleware that will "sit" in the middle of all your other application (hence "middleware") and connect all the applications together using a publisher/subscriber model.
In that architecture, the loan application does not know about any other application than the MQ. Whenever you start defaulting it will simply send a message ("loandefaulted userid=12 amount=345.5") to the message queue and the MQ will in turn dispatch it to whatever other applications has register an interest for it by saying something like "subscribe to event loandefaulted from loanapplication". Many applications can register to the same event. That way the CRM can flag you, the call center can call you, legal can sue you and the SMS reminder volley can begin. All of that without any applications having to care about the brand or specific implementation (J2EE or
Of course this is a bit simplistic but cover 80% of the purpose of an MQ: loose coupling of application via an event based mechanism. Add to that option such as guaranteed reliability, prioritization of message, security and more complexe workflow of messages with multiple "queries" and "answers" and maybe you'll get a rough idea of why a MQ is a better design that "hard coupling" of n-application (sometime).
Of course since once you've chosen a MQ and adapted all your applications to use it you're basically tied forever and ever to your MQ vendor who hold you by the balls and can continuously rape you over and over with astronomical maintenance fee since you now have a single coordinated point of failure that can and will eventually take everything down at some point.
But hey, you're a big bank you can take it.. or at least you could until a month ago but whatever...
Clearer?
Not so little known (Score:4, Informative)
Odd this came up just at this moment, though it's hardly little known.
Implementations:
Several here: http://jira.amqp.org/confluence/display/AMQP/AMQP+Products [amqp.org]
Re:XMPP (Score:5, Informative)
While XMPP and AMQP do have some similarities, they were developed to solve quite different problems.
XMPP is better for human-to-human or human-to-machine interaction. It is better for federated networks (user@jabber.someserver.com can communicate with user@jabber.someotherserver.com). It has authentication built in. It's more "Internet ready", with modules for BOSH and IRC.
AMQP is geared more towards traditional machine-to-machine message queuing. It offers more control over the type of message delivery ("exactly once", "pubsub", "at least once"). It's aiming to just be a message queue.
RabbitMQ [rabbitmq.com] is an open-source implementation of AMQP built in Erlang. They seem to be quite fond of XMPP, realizing the different natures of the protocols, and even created a module for ejabberd [rabbitmq.com] (also written in Erlang).
Another interesting mashup between RabbitMQ and ejabberd is is a project I stumbled across called Rabbiter [rabbitmq.com], which looks like some sort of implementation of a Twitter-link setup. They're looking to bring CouchDB (also Erlang) in the mix for persistence.
I'm expecting to see quite a bit of interaction between RabbitMQ, ejabberd, and CouchDB over the next few months.
Ruby AMQP Client [github.com]
Python AMQP Client [barryp.org]
Re:XMPP (Score:4, Informative)
This is the first that I've heard of this technology. Can anyone post how this is different than XMPP (http://xmpp.org)?
Here's a post about XMPP vs AMQP by Peter Saint-Andre:
https://stpeter.im/?p=2099 [stpeter.im]
IMNSHO, XMPP (+XEPs) makes AMQP redundant. Although some people might argue that, being a binary protocol, AMQP is faster.
Re:Sooo.... (Score:2, Informative)
And you get C# which is actually a much nicer language than J++ ever was, integrate way better with Microsoft platform
Microsoft took Java, did a shitty job at writing a VM for it, ignored repeated urgings to conform, then threw a tantrum and reinvented the wheel.
... gave a much needed kick in sun's behind who realized it was finally time to update Java with some interesting features. Both are now alive and kicking.
Competition's always better, and my point wasn't to compare Java and C#. It was to demonstrate Microsoft's approach to such issues, which is to dissect, then "reinvent".
What was your point?
Pay better attention next time
Re:XMPP (Score:3, Informative)
Quite a bit, actually. While its not quite at 1-0 (what every other project in the universe would call "1.0") yet (the most recent release is 0-10), you can look at the high-level requirements list for 1-0 [amqp.org], to get an overview.
Lots of applications probably don't need what it provides over STOMP, either because they don't need it at all, or because they get it somewhere other than the basic messaging protocol, but it certainly covers a lot more than STOMP does.
The words ring hollow (Score:3, Informative)
"They are also -- perhaps not by coincidence -- burdensomely expensive."
I've been using Exchange since 5.5 and have dealt with every revision up to the current one. When the organization that I worked for developed a need for in house IM the first place I went to look was Exchange. They had built in IM in Exchange Server 2000. We're running 2003 and surprise, surprise... IM is gone. Now Microsoft expects us to purchase some Office Communications Server or some BS like that to have the functionality that was previously free.
Given that experience with Microsoft and IM, I don't think that they're really in the business of giving free functionality to people using their software.
As long as I'm posting this, can anyone recommend some good, hopefully free IM clients for internal use? We only need to support about 50 users.
Re:XMPP (Score:5, Informative)
Performance wise, XMPP bills itself as high performance messaging, but the developers are focused on the WAN. AMQP comparatively is ultra-high performance messaging with optimisations for the LAN.
This is confusing as for many projects there is limited need for ultra-high performance data rates. Numbers of the range 100,000 messages per second with latency under one millisecond. At this rate special engineering methods are required, XML, SSL, compression are too slow, focus is upon zero-copy processing, i.e. accessing and updating data in place, because the memory-bus is too slow to perform copies.
There is a discussion between one AMQP and one XMPP developer that sums this up:
http://www.mail-archive.com/jdev@jabber.org/msg19403.html [mail-archive.com]
Another major advantage for AMQP is message routing. You can define which messages are routed to different sites by their content. Again this is an unusual requirement for many projects as they do not exist on such a scale for this to normally be an issue. The closest equivalent is SMTP routing by domain, you can find more discussion on this InfoQ article:
http://www.infoq.com/news/2008/08/amqp-progress [infoq.com]
The main focus on AMQP is to appear a qualified messaging protocol for certified or guaranteed messaging with the necessary tools and support from vendors to promote its usage. XMPP can do a lot of the AMQP functionality already, but most of it is optional functionality rather than a primary design goal. If AMQP support appears in the Visual Studio Development System together with MMC modules for monitoring and administration, for example, its adoption could rapidly grow.
Re:XMPP (Score:3, Informative)
Well, you compare two quite different things (throughput & latency), but I'll bite: You should be able to do 10-100,000 messages per second with Red Hat MRG. If you can't, then there's something wrong with your set up.
Remember that AMQP was initially designed and written by JP Morgan to replace their existing proprietary infrastructure (IBM MQSeries-based IIRC). JP Morgan understand the performance concerns.
Rich.
Why AMQP is important and how to use it (Score:3, Informative)
I work on RabbitMQ which is a messaging implementation that provides AMQP and other patterns. I hope my comment here can clear up some misunderstandings in the community.
Here is an introduction to AMQP from the RabbitMQ team - there are two presentations and a video - made at Google a few weeks ago: http://google-ukdev.blogspot.com/2008/09/rabbitmq-tech-talk-at-google-london.html [blogspot.com]
People who are wondering why AMQP is important compared to MQ, JMS, or whatever, should check out the first presentation. The second presentation includes information about XMPP vs AMQP. The two protocols are strongly complementary and are not alternatives. You can use these today: RabbitMQ as someone mentioned above has clients in lots of languages like Java, Ruby, Python,
Integration with Visual Studio has been done: http://www.rabbitmq.com/dotnet.html [rabbitmq.com]
As someone else pointed out there are protocol adaptors including STOMP and XMPP: http://www.lshift.net/blog/2008/07/01/rabbitmq-xmpp-gateway-released [lshift.net]
It is open source - we welcome questions on the mailing list which you can find linked to from the home page: http://www.rabbitmq.com/ [rabbitmq.com]
Cheers,
alexis