What Is a Message Queue? Background Jobs with RabbitMQ

A message queue is a middle layer where an application leaves a job as a message instead of doing it immediately, and another process picks those messages up in order and processes them. The short answer: it separates work that should not keep the user waiting, or that depends on an external system (email, invoicing, notifications, report generation), from the web request; when the external system slows down or fails, the job is not lost, it waits in the queue. RabbitMQ is one of the most widely used open source tools for this.

What is a message queue and how does it work?

The model has three parts. The producer sends a small message describing the work, for example "create the invoice for order 42". The broker, here RabbitMQ, stores the message durably. The consumer is a separate worker process; it takes the message, does the work and acknowledges (ack) that it is done. If the worker crashes before acknowledging, the broker hands the message to another worker. This lets the web server save the order and answer the user immediately while the invoice is created a few seconds later in the background.

What is a message queue used for?

• Sending email and SMS: sign up and checkout screens do not wait even when the provider is slow.
• Invoicing and official integrations: for external services that may answer late, such as e-invoice integration, retries are handled by the queue.
• Incoming webhook events: the event is accepted at once and queued, then processed separately; details in what is a webhook.
• Heavy work: PDF and report generation, image resizing, bulk imports.
• Communication between services: when several services need to react to an event in one service.

A queue makes sure work is not lost, but it does not make sure work is done only once. The same message can arrive more than once, so every consumer must be idempotent: receiving the same job a second time must not change the result.

Acknowledgements, retries and the dead letter queue

A solid queue setup makes three decisions up front. First, acknowledgement: the consumer acknowledges after the work is really finished, not on receipt. Second, retries: on temporary errors the message is retried a limited number of times with a delay, never in an endless loop. Third, the dead letter queue: a message that runs out of retries goes to a separate queue, where it is monitored and resent manually once the cause is fixed. The number of messages piling up in the dead letter queue must be monitored and should raise an alert above a threshold.

The Node.js example below shows a producer that queues the invoice after an order and a consumer that processes it. The message id is the order number; the createInvoiceOnce function in the consumer never creates a second invoice for the same order.

import amqp from 'amqplib';

const conn = await amqp.connect(process.env.AMQP_URL ?? 'amqp://localhost');
const ch = await conn.createChannel();
await ch.assertQueue('invoices', { durable: true });

// Producer: the web request only drops a message and returns.
export function enqueueInvoice(orderId: string) {
  ch.sendToQueue('invoices', Buffer.from(JSON.stringify({ orderId })), {
    persistent: true,
    messageId: orderId, // used for idempotency on the consumer side
  });
}

// Consumer: a separate worker process.
await ch.prefetch(5);
await ch.consume('invoices', async (msg) => {
  if (!msg) return;
  const { orderId } = JSON.parse(msg.content.toString());
  try {
    await createInvoiceOnce(orderId); // safe to run twice for the same order
    ch.ack(msg);
  } catch {
    ch.nack(msg, false, false); // goes to the dead letter queue
  }
});

RabbitMQ, Redis or Kafka?

RabbitMQ is designed for classic job queues: routing, acknowledgements, priorities and dead letter queues come built in. If your application already runs Redis, Redis based queue libraries may be enough for low volume background jobs; for Redis's other uses see our what is Redis article. Kafka is an event log that keeps messages for a long time; it makes sense for high volume event streams and when the same data must be replayed later, and it is overkill for a simple job queue. The choice should follow how long a message must be kept and how many consumers read it, not tool popularity.

When is a message queue unnecessary?

Not every background job needs a queue. For reports or cleanup that run at fixed times, a scheduled job (cron job) is simpler. In a low traffic, single server application, a "pending jobs" table in the database and a small worker reading it is often enough. A queue is one more component to monitor, back up and upgrade. For the same reason it is worth answering microservices or monolith before splitting the application into services.

Conclusion

A message queue safely moves work that should not keep users waiting, or that depends on external systems, into the background. It shows its value when acknowledgements, limited retries, a dead letter queue and idempotent consumers are set up together. If you would like to plan background processing for your project together, see our custom software page or write to us through the get a quote form.