Distributed Schedulers Proficient¶

What you'll learn¶

• Why offload work to background jobs
• Celery and RQ, and how they differ
• Retry with exponential backoff
• Handle permanent failures (dead-letter)
• Schedule periodic and delayed jobs

Why background jobs¶

Move slow or deferred work off the request path so users aren't kept waiting.

Some work shouldn't happen inside a web request: sending email, generating a report, processing an upload, calling a slow third-party API. Doing it inline makes the user wait and ties up a web worker. Instead, you enqueue a job and return immediately; a separate pool of worker processes picks it up and runs it.

   web request ──enqueue──▶ [ task queue ]  ──▶ worker pool (runs the job)
       │                    (Redis / broker)         │
       └─ returns fast ◀─                            └─ retries, logs, results

This is the distributed queue pattern applied to task execution. The job systems below (Celery, RQ) build on a broker like Redis or RabbitMQ.

Celery vs RQ¶

Two Python task-queue libraries — RQ is simple, Celery is feature-rich.

The two most common Python task queues:

Defining and calling a Celery task:

from celery import Celery   # pip install celery

app = Celery("tasks", broker="redis://localhost:6379/0")

@app.task
def send_email(to: str, subject: str) -> str:
    # ... actually send ...
    return f"sent to {to}"

# enqueue (returns immediately; a worker runs it later)
send_email.delay("user@example.com", "Welcome!")

Retry with exponential backoff¶

Retry failed jobs with growing delays to avoid hammering a struggling dependency.

Jobs fail — a network blip, a rate limit, a briefly-down dependency. The right response to a transient failure is to retry, but not instantly and not forever. Exponential backoff waits progressively longer between attempts (and real systems add random jitter to avoid a thundering herd of synchronized retries). Fully runnable:

def backoff_delays(base: float, attempts: int, cap: float) -> list[float]:
    """Delay before each attempt: base * 2^i, capped."""
    return [min(base * (2 ** i), cap) for i in range(attempts)]

print(backoff_delays(base=0.5, attempts=6, cap=10.0))

Output:

[0.5, 1.0, 2.0, 4.0, 8.0, 10.0]

Each retry waits twice as long as the last (0.5s → 1s → 2s → 4s → 8s), then caps at 10s so it never grows unboundedly. This spreads out retries so a struggling downstream service gets breathing room instead of being hammered.

Handling permanent failures: dead-letter¶

Route repeatedly-failing jobs to a dead-letter queue for inspection.

Not every failure is transient. A malformed job or a permanently-broken dependency will fail every retry. After a bounded number of attempts, you must give up gracefully and set the job aside (a dead-letter) rather than retry forever. Runnable:

def run_task(task, max_retries: int) -> dict:
    last = None
    for attempt in range(max_retries + 1):
        try:
            result = task(attempt)
            return {"status": "done", "attempts": attempt + 1, "result": result}
        except Exception as e:
            last = str(e)
    return {"status": "dead_letter", "attempts": max_retries + 1, "error": last}

def flaky(attempt):
    if attempt < 2:
        raise RuntimeError("temporary")   # fails twice, then succeeds
    return "success"

def always_fails(attempt):
    raise RuntimeError("permanent")

print(run_task(flaky, max_retries=3))
print(run_task(always_fails, max_retries=2))

Output:

{'status': 'done', 'attempts': 3, 'result': 'success'}
{'status': 'dead_letter', 'attempts': 3, 'error': 'permanent'}

The flaky job recovers on its third attempt and reports done. The always_fails job exhausts its retries and is dead-lettered with the error recorded — so it can be inspected later instead of blocking the queue or retrying endlessly. Celery and RQ provide this (max_retries, retry_backoff, dead-letter routing) so you configure rather than hand-code it, but the logic is exactly this.

Scheduling periodic and delayed work¶

Run jobs on a schedule or after a delay.

Beyond "run this now in the background," schedulers also handle:

• Delayed jobs — "run this in 10 minutes" (send a reminder, expire a session).
• Periodic jobs — "every night at 2am, generate the report" (Celery Beat, rq-scheduler, or cron feeding the queue).

For simple, single-machine timing, OS cron (see Automation & Scripting) may be enough. For distributed, retried, monitored jobs across many workers, a task queue is the right tool.

Practice exercises¶

1. Add jitter to backoff_delays: return min(base * 2**i, cap) * random_factor and explain why jitter prevents synchronized retry storms.
2. Extend run_task to only retry on certain exception types (transient) and immediately dead-letter others (permanent) — don't waste retries on a bug.
3. Track and return the list of per-attempt delays alongside the result.
4. Make flaky idempotent by having it record a side effect only once even across retries.
5. Decide, for five example jobs (email, nightly report, thumbnail, payment, cleanup), whether you'd use cron or a task queue, and why.