What is message queuing for?

January 9, 2009 3:00 PM

What do I do with a message queue? (programmer geek)

What is message queuing -- in general, what sort of app would I be making if I wanted to utilize the Windows Message Queuing engine?

If its just a FIFO messaging queue why not just do it directly between two systems?

If I had 5 voting machines, would I use message queuing to communicate with the server? Why wouldn't I just connect to a database directly?

When do they make sense to use?

posted by SirStan to computers & internet (14 comments total)

Message queues are excellent in situations where you do not have constant connectivity.
posted by Calloused_Foot at 3:14 PM on January 9

Succinct description of the Messaging pattern here. Amazon's "look inside" excerpt actually has the relevant section available.
posted by kanuck at 3:21 PM on January 9

Yeah, it's a reliability hack.
posted by rhizome at 3:22 PM on January 9

Reliability and scalability. If 10,000 subscribers want to know when you post a message it's easier to have a central mechanism for subscribing to notifications versus having to code that into every possible publisher.
posted by GuyZero at 3:30 PM on January 9

Yeah, this is for asynchronous message handling. Maybe the process that handles the messages is on another machine, maybe you can always reach that machine, maybe messages come in faster than you can process them. The basic idea is to let the framework handle all that crap, so you can concentrate on sending messages and doing what needs to be done to process them.
posted by orthogonality at 3:32 PM on January 9

We use message queues extensively in our apps to allow for multithreadedness. One thread slurps data off the network, throws it onto the message queue, then fifty-billion handling threads pull one off the queue and do something with it.
posted by yellowbkpk at 3:36 PM on January 9

So when do I decide to use a message queue, versus not? Clearly "alot of things in asynchronously". Is it a design mechanism like a 3-tier system? Why not just dump data into a database?
posted by SirStan at 3:43 PM on January 9

Message queues are generally lightweight and fast; they scale up to larger #s of simultaneous connections than a db. From a commercial aspect, they're typically cheaper to purchase, since they do "less" than a database. if you simply tried to build a MQ system on top of a db you'd need more hardware and you'd spend more on software licenses. But ultimately, yes, it's a design mechanism - you could certainly build a mq-like system with pretty much any db.
posted by GuyZero at 3:46 PM on January 9

Here's an actual example that I'm working on. At work we have a status page that is constantly (ajax-y) refreshed once a minute or so. The problem is that often once-a-minute isn't fast enough so people manually refresh.

I wrote a little script that does that refresh and parses it, compares it to the last refresh and generates add-this and delete-that messages. It puts these message into a message queue (MorbidQ/Orbited). There's a little javascript page that handles building the page again and responding to those events by modifying a DOM table with jquery.

Now every modification to the static page only generates a small amount of data, no modification generates no data and everyone gets updates as they happen, not when their next refresh is. And because the data is now JSON not an HTML table it's trivial to make javascript rules to parse, alert, etc based on changes.

There's maybe a few hundred lines of code here all told but strip the MorbidQ toy queue out and replace it with something serious like ActiveMQ or RabbitMQ (and some load balancnig obv) and my little python script will scale to likely hundreds of thousands of simultaneous connections. Sure you can do this kind of thing with a database, but in that case you're making a database act like a queue, and in that case why not just use an actual queue?
posted by Skorgu at 3:52 PM on January 9

A message queue is an abstraction that lets you separate concerns.

Take your voting example. If your voting machines are storing to the database directly that software is now dependent on what happens when a vote is stored, and if one machine is out of date/doing something wrong it can mess up the entire system.

For example, in the first version of your software votes are just stored in a big database table. One vote per user. When the voting is done you run an sql query and determine the vote total. It's a great success and you have a bunch of requests for version two. They want the votes to be signed with the time and a secure hash so that everyone knows that the votes are valid, they want a little bell to ding every time 100 votes are counted, and finally they have a voting machine that will only be connected to the voting network at the end of the day.

None of those changes would require any code changes to the client voting machines in order to implement if you had used a message pattern. However if you wrote directly to the voting database it would require significant changes and probably be loaded down with easily broken, poorly abstracted code.
posted by aspo at 4:15 PM on January 9

Think of a diner's order window; the waiters can drop off orders, the cooks can work on them and just interface through the window. Sure, your waiters could just "step back into the kitchen and cook the order" but what about when you have twenty waiters and they're bumping into each other in the kitchen? What about when you have one waiter for fifty customers and he's taking too long in the kitchen? When you change something in the kitchen why do you have to re-train the waiters?

It's just an abstraction layer that fits naturally over a network gap, or at the front end of a process/thread, that allows for resource "sharing" and design abstraction. Also you can get async input into serialized atomic operations which is often what you want anyway.
posted by fleacircus at 4:58 PM on January 9 [2 favorites]

Think of a message queues like email. You can send whenever you want, even if the receiver isn't online. The receiver can read the messages when ever they want, at whatever rate they want.

Basically anything that isn't urgent, or needs guaranteed deliverability, with or without a connected endpoint can use message queues.
posted by blue_beetle at 5:36 PM on January 9

Normally, you'd use a message queue when your application needs to take some action if another application sends it a message. The message queue handler will set a flag, or set a status indicator, when a message is received. When your application detects that the flag is set (via a polling software interrupt - probably a standard library function), it takes some action in response. Message-handlers allow you to synchronize actions between applications on the same, or a different machine. A user-interface app. might for example, send a message to a media manager app. that indicates the type of media to be played and provides summary media data. The media manager app. would start a media player of the correct type when it sees this message, then send a message back to the first app., providing the media player application's handle. The first app might then start streaming media data directly to the media player app.
The point is that a message-received causes the receiving application to change state. The state-change causes some action to be taken and (if required) an acknowledgement message to be sent back to the originating application. This is how "asynchronous synchronization" is achieved.
posted by Susurration at 6:17 PM on January 9

Queues are essentially decoupling mechanisms. Whenever you don't want latency to propagate through a system, it's best to have work item queues in between the various tiers.

MSMQ is typically used for cross machine (or at least cross process) queues, but there are probably more perpetually churning queues than you expect inside of other bits of server software. IIS, for example, has a couple (or more, depending on what you do with ISAPI and yadda) layers of work item queues going on.
posted by flaterik at 8:23 PM on January 10