Queues and logs are not the same tool
A queue deletes a message when it is done; a log keeps it and lets readers move a bookmark.
What you will be able to do
- Choose between a work queue and a partitioned log for a given workload
- Explain consumer groups, partitions and offsets
- Estimate storage and throughput for a log-based pipeline
- Recognise when a queue silently loses ordering guarantees
The mental model
Picture a queue as a stack of job tickets on a shared desk. Any worker can take the top ticket, and once the work succeeds the ticket is gone. This is what you want for sending an email, resizing a thumbnail, or charging a card: the work must happen once, and it does not matter which worker does it.
Now picture a log as a ledger that only grows. Writers append entries to the end. Each reader keeps its own bookmark, called an offset, and can move forward, pause, or rewind to the beginning and replay history. Deleting entries is a retention decision — seven days, thirty days — never a side effect of somebody processing them.
| Question | Work queue | Partitioned log |
|---|---|---|
| Who sees a message? | One consumer in the group | Every consumer group sees all of it |
| Is data removed after processing? | Yes, on acknowledgement | No, only when retention expires |
| Can I reprocess yesterday's data? | Only if you kept a copy | Yes, reset the offset |
| Ordering | Best effort, or per-group with limits | Strict within a partition |
| Typical job | Send email, transcode video | Event stream, change data capture, metrics |
| Scaling reads | Add workers to split work | Add partitions and parallel readers |
Partitions are the unit of everything
A log is split into partitions. Each partition is an ordered sequence stored on one broker, and a consumer group assigns each partition to exactly one consumer in that group. That gives you both guarantees at once: order is preserved inside a partition, and parallelism scales with the number of partitions — not the number of consumers.
That second row is the one interviews forget. Retention is a cost commitment. At 10 MB/s sustained, a thirty-day window is roughly 26 TB before replication; with three replicas it is closer to 78 TB. Say the number out loud before you promise replay.
Where queues still win
- Long or variable jobs. A transcode that takes twenty minutes needs a visibility timeout, retries and dead-letter handling — queue semantics, not offsets.
- Per-message delay and scheduling. "Run this in two hours" is natural in a queue and awkward in a log.
- One-off work with a clear owner. If exactly one team consumes it and history has no value, a log is storage you will never read.
- Priority lanes. A queue can serve high-priority work first; a log is read in order by design.
The combination that actually ships
Production systems usually run both. A log carries the facts — order placed, payment captured, item shipped — and small queue-backed workers do the slow side effects. The log is the source of truth for what happened; the queue is a scheduler for what should happen next.
Worth remembering
- A queue is about distributing work; a log is about distributing facts.
- Logs make replay, fan-out and reprocessing cheap because retention is a policy, not an accident.
- A partition is the unit of ordering and the unit of parallelism in a log.
- Adding consumers to a queue is easy; adding a second independent reader to a queue forces you to copy data.