Loading...
Loading...
Background job processing, Celery, Sidekiq, and worker patterns
Quarantine holds the failures with evidence headers, and replay runs rate-limited at 50 to 100 messages per second. Both need a human pushing the button. Then reality arrives beyond failures: the email service is down so a message must retry in ten minutes by itself, the digest must go out at 8am sharp rather than whenever a worker happens to be free, and someone asks whether the export finished while you have no answer. A bare pile of notes handles none of this, because it only holds paper. Think of a small office with one diligent manager as the single analogy here: the pile is the inbox tray, and the manager is everything around it that retries the failed phone call, schedules the morning mail, tracks what finished, and lets urgent letters jump the line.
The naive fix is to build each chore by hand around a raw queue, adding a retry loop here and a cron entry there. The most common handmade version is a database table polled with SELECT FOR UPDATE SKIP LOCKED, which is locking one ready row while skipping rows other workers already locked. It works to roughly a thousand jobs a minute, then lock contention climbs past five workers and the one-second poller fires 86,400 empty queries a day per worker asking about nothing. That fails because the chores interact: retries without rate limits become a storm, schedules without visibility become mystery jobs, and priorities without isolation let one long encode block every password reset. A task queue is that office manager as software: the same pile underneath, with retries, schedules, priorities, rate limits, and status boards bolted on. Message queues move data, while task queues get jobs done.
One sentence version: if the note says run this function with these arguments, retry if it fails, and tell me when it lands, that is a task queue.
You write a normal function, mark it as a task, which is a function registered with the queue framework so workers can find and run it, and invoke it with a delayed call instead of calling it directly. That one word is the whole trick: instead of running now in your request, the call serializes, which is converting the function name and arguments into bytes the queue can store, into a note. A worker picks it up seconds later while your request already returned happily.
// Define the job once
@task
def send_email(user_id, template):
user = db.get(user_id)
email.send(user.email, template)
// Enqueue it — returns instantly
send_email.delay(user_id=123, template="welcome")
// A worker somewhere runs it
$ celery worker --app=myapp
The edge case hiding here is what you pass: pass small ids like user 123, never live objects like database connections, because the queue serializes arguments and a connection cannot survive the trip, so every worker would crash trying to unpack it.
Failed calls wait, back off exponentially, which is waiting progressively longer between attempts, and retry a bounded number of times. Giving up loudly after N attempts beats retrying silently forever.
Cron patterns, which are schedule expressions like every morning at 8, run without a separate scheduler to babysit. The queue itself wakes the job.
Pending, running, done, and failed states feed a dashboard, so did the export finish becomes a lookup instead of a guess.
Priorities, which are separate lanes such as urgent versus batch, let password resets jump ahead of weekly digests instead of waiting behind them in one first-in-first-out line.
Rate limits, which cap calls to a downstream provider at actions per second, hold email sending at ten per second and never eleven, so a burst of jobs cannot get your sender account throttled.
A result backend, which is a small store where finished jobs stash outputs for later lookup, holds answers for whoever asks, with a time-to-live so old results expire instead of accumulating forever.
Celery, which is a Python task framework running on RabbitMQ or Redis as the underlying pile, does everything from retries to schedules with deep documentation. The ecosystem means someone already solved your exact problem, at the cost of real broker operations.
BullMQ, which is a Node task library backed by Redis with a genuinely usable dashboard, is fast and natural when you live in JavaScript and TypeScript. It inherits Redis operations, so durability planning means Redis persistence planning.
Sidekiq, which is a Ruby job system using threads inside few processes instead of many processes, is startlingly efficient per machine. Rails shops rarely need anything else until workloads outgrow a single Redis.
SQS, which is a hosted queue billed per call, plus Lambda, which runs your function per batch and scales to zero, means messages trigger functions with nothing to operate. The price is vendor-shaped edges around timeouts, retries, and ordering that you must learn once.
Welcomes, resets, and receipts never need the user waiting on an SMTP round trip, where SMTP is the protocol servers use to hand email to each other. Enqueue with the user id and template, return instantly.
Resizes, transcodes, and PDF exports are minutes of CPU disguised as requests. Enqueue the file id, poll or push the result link later.
Heavy analytics and CSV dumps run for minutes and fail midway. Enqueue, track status on the board, and email a download link when done.
Digests, cleanups, and billing runs are cron with retries and a dashboard. The queue wakes them, tracks them, and pages when they fail.
Your first deploy retries every failure instantly, forever, at full concurrency, so a flaky email provider becomes a self-inflicted flood. Production task queues survive because four dials are set deliberately: retry policy with backoff, acknowledgment timing, concurrency caps, and result handling. Late acknowledgment, which means confirming the job only after success so a crashed worker requeues instead of losing it, is the one beginners skip first.
# Celery: bounded retries with exponential backoff + jitter
@app.task(bind=True, max_retries=4, acks_late=True,
autoretry_for=(TimeoutError,), retry_backoff=30,
retry_jitter=True, time_limit=300)
def send_email(self, user_id, template):
try:
email.send(user_id, template)
except TransientError as e:
raise self.retry(exc=e, countdown=2 ** self.request.retries * 10)
# BullMQ (Node): same ideas, Redis-backed
# new Queue('email', { defaultJobOptions: {
# attempts: 4, backoff: { type: 'exponential', delay: 10000 },
# removeOnComplete: 1000, removeOnFail: 5000 }})
# new Worker('email', handler, { concurrency: 20, lockDuration: 60000 })Acknowledge only after success so a crashed worker requeues instead of losing the job. The lock duration in BullMQ and the visibility timeout in SQS, which is how long a delivered message stays hidden from other workers, play the same role, and both must sit above your slowest realistic runtime, or slow jobs run twice while still running once.
Keep result backends tiny with expirations such as removing completed jobs after 1,000 entries or expiring results after an hour. Teams that store every thumbnail result forever discover their Redis became a database: an expensive, un-backed-up one that pages on memory at 3am.
Two classics close the loop. The serialization bug, which is passing a live object the queue cannot convert to bytes, crashes every worker on unpack, because a task pickled a database connection and connections cannot survive the trip. The long-task starvation, which is one slow job holding a worker slot while urgent jobs wait, parks a 20-minute video encode on a shared pool so password resets queue behind it. The cures are boring and total: pass ids, never objects, and run separate queues with separate workers per latency class, so urgent, default, and batch never share a pool.
| Failure | Smell | Fix |
|---|---|---|
| Retry storm | Provider recovers then instantly collapses under pent-up retries | Exponential backoff with jitter, concurrency cap, and a breaker, which is a guard that fails fast instead of calling a sick dependency, that opens when failures pass 50 percent |
| Head-of-line by long tasks | Resets slow while encodes run | Separate queues for urgent versus batch with separate workers |
| Result backend bloat | Redis memory climbs forever | Expire every result, paginate status reads, archive large outputs to object storage |
Rule of thumb from here on: if the user does not need the result before the response returns, it does not belong in the request. Enqueue it, with retries, schedules, and idempotency keys, which are stable identifiers letting a handler recognize a replay as already done, since workers crash after side effects and the queue alone promises only at-least-once. But the manager assumes the pile drains: twenty workers chew 40 videos a minute while fifty thousand arrive in the hour, and the last uploader waits 1,250 minutes. What decides who gives way when the faucet permanently outruns the drain?