Distributed Queues Proficient¶

What you'll learn¶

• Why services communicate through queues
• RabbitMQ (queue) vs Kafka (log) models
• The three delivery guarantees
• Build an idempotent consumer (handle duplicates)
• Message ordering and consumer groups

Why queues¶

Decouple producers from consumers so work is buffered, retried, and processed independently.

When services talk directly (service A calls service B over HTTP), A is blocked while B works, and if B is down, A fails. A message queue sits between them: A publishes a message and moves on; B consumes it when ready. This decouples producers from consumers in time — B can be slow, restart, or scale out, and A never notices.

   producer ──▶ [ queue / log ]  ──▶ consumer(s)
              (buffers, persists,      (process when ready,
               absorbs bursts)          scale independently)

Queues give you buffering (absorb traffic spikes), resilience (messages survive a consumer crash), and independent scaling. They're the backbone of event-driven systems.

RabbitMQ vs Kafka: two models¶

Broker-with-queues vs distributed-log — pick based on routing needs vs replay/throughput.

The two dominant systems embody different models:

RabbitMQ is a traditional broker — great for task distribution and complex routing. Kafka is a distributed commit log — great for high-volume event streams you may want to replay or feed to multiple independent consumers.

Producer/consumer sketch with RabbitMQ (pika):

import pika, json   # pip install pika

conn = pika.BlockingConnection(pika.ConnectionParameters("localhost"))
ch = conn.channel()
ch.queue_declare(queue="tasks", durable=True)        # survive broker restart
ch.basic_publish(exchange="", routing_key="tasks",
                 body=json.dumps({"id": "m1", "job": "resize"}),
                 properties=pika.BasicProperties(delivery_mode=2))  # persistent

Delivery guarantees¶

At-most-once, at-least-once, exactly-once — what each costs and when you need it.

No broker can give you everything; you choose a point on this spectrum:

• At-most-once — fire and forget. Fast, but messages can be lost (consumer crashes before processing). OK for disposable data (metrics samples).
• At-least-once — the broker redelivers until acknowledged. No loss, but duplicates happen. The common default.
• Exactly-once — no loss, no duplicates. The ideal, but genuinely hard and expensive; often approximated with at-least-once delivery + idempotent consumers.

The practical reality: most systems use at-least-once and make consumers idempotent. That combination gives you no-loss delivery while neutralizing the duplicates.

The idempotent consumer¶

Make processing safe to retry so at-least-once delivery doesn't double-apply effects.

Since at-least-once means the same message can arrive twice (redelivery after a timeout, a retry, a broker hiccup), your consumer must produce the same effect whether it processes a message once or many times. The standard technique: track processed message IDs and skip duplicates. Fully runnable:

class IdempotentConsumer:
    def __init__(self) -> None:
        self.processed: set[str] = set()
        self.effects: list[str] = []

    def handle(self, msg_id: str, payload: str) -> bool:
        if msg_id in self.processed:
            return False                     # duplicate — skip, no double effect
        self.processed.add(msg_id)
        self.effects.append(payload)         # the real side effect — happens once
        return True

c = IdempotentConsumer()
print(c.handle("m1", "charge $10"))    # True  — processed
print(c.handle("m1", "charge $10"))    # False — same id redelivered, ignored
print(c.handle("m2", "charge $20"))    # True  — new message
print("effects:", c.effects)

Output:

True
False
charge $20    ← (from the print of the third handle returning True)
effects: ['charge $10', 'charge $20']

The redelivered m1 is skipped, so charge $10 happens exactly once even though the message arrived twice. This is how you turn at-least-once delivery into effectively-once processing — the pragmatic path to "exactly once" behavior.

Ordering and consumer groups¶

How partitions/queues preserve order and scale consumers.

• Ordering is only guaranteed within a single queue (RabbitMQ) or partition (Kafka) — not globally across partitions. If you need related messages ordered, route them to the same partition (e.g. partition by user_id so one user's events stay ordered).
• Consumer groups (Kafka) let many consumers share the load: each partition is read by exactly one consumer in the group, so you scale out by adding partitions and consumers. This is how you process a high-volume stream in parallel while preserving per-partition order.

Dead-letter queues catch messages that repeatedly fail processing, so one poison message doesn't block the queue or retry forever — it's set aside for inspection.

Practice exercises¶

1. Make IdempotentConsumer persist processed ids to a file (or sqlite) so dedup survives a restart.
2. Add a max-processed-set eviction (e.g. keep only the last N ids) and discuss the risk it introduces.
3. Model a dead-letter path: after 3 failed attempts, route a message to a dead_letter list instead of retrying forever.
4. Explain why partitioning by user_id preserves per-user ordering but not global ordering.
5. Give an example operation that is naturally idempotent (needs no dedup) and one that isn't.