The path of one entry#
A stream is a Redis key holding an append-only sequence of entries, each a small set of field-value pairs with an ID. A consumer group is state stored inside that key: a position in the stream (last-delivered-id) and a pending entries list (PEL) of what was handed out but not yet acknowledged. Consumers are just names within a group.[1]
XADD orders * order 10appends an entry. Redis generates the ID (<milliseconds>-<sequence>) and replies with it.[2]- The XADD wakes consumers blocked in
XREADGROUP … BLOCK. Each group hands out entries after its ownlast-delivered-id, moves it forward, and records each entry in its PEL with the consumer's name, the delivery time and a delivery count.[3] - The consumer processes the entry. Meanwhile the entry's idle time grows in the PEL.
XACK orders mailers <id>removes the entry from the group's PEL. Until then, Redis treats it as possibly unprocessed.[4]
The entry itself stays in the stream after every group has acknowledged it. Acknowledging only changes the group's bookkeeping; deleting entries is a separate concern (trimming, below).[1]
See it happen
Groups, consumers and lag#
Consumers in the same group share the work: each entry goes to one of them. Different groups are independent: each gets every entry, at its own pace. Here mailers has two consumers that take turns, and analytics has one that sees all orders.[1][3] Each group is created once, with XGROUP CREATE orders mailers $ MKSTREAM: $ means it starts after the stream's current last entry, and MKSTREAM creates the stream if it doesn't exist yet.[5]
Delivery is pull: a consumer receives at most COUNT entries per call, and with BLOCK it waits inside Redis until there is something to return.[3] A slow consumer doesn't grow a buffer inside Redis the way a Pub/Sub subscriber does; the entries wait in the stream and the group's lag grows. XINFO GROUPS reports it as the number of entries in the stream that are still waiting to be delivered to the group.[6]
A consumer that goes away for a while loses nothing. When it comes back, > continues from the group's last-delivered-id, and it works through the backlog.[1]
The PEL and XACK: at-least-once#
The PEL is what turns a log into a work queue. Each entry in it has an owner, the time it was last delivered (from which XPENDING computes idle time) and how many times it was delivered. A consumer that crashes before XACK leaves its entries there, owned by it.[1]
On restart, the usual pattern is to first read with ID 0 instead of >: XREADGROUP … STREAMS orders 0 returns this consumer's own pending entries, and adds 1 to their delivery count. When that comes back empty, switch to >.[3]
Reading with NOACK skips the PEL entirely: an entry counts as acknowledged the moment it is delivered. It is meant for cases where occasional message loss is acceptable: a crash drops whatever the consumer was holding.[3]
See it happen
Taking over stuck entries#
Redis never redelivers on its own. If a consumer is gone for good (scaled down, its host lost), its pending entries wait forever. Another consumer has to take them over:[1]
It scans the group's PEL for entries idle at least min-idle-time, makes the caller their owner, resets their idle time and increments their delivery count. It replies with a cursor for the next call, the claimed entries, and (since 7.0) the IDs of pending entries whose data no longer exists, which it removes from the PEL. Because claiming resets idle time, two consumers can't both claim the same entry at the same moment. Redis 8.4 adds a CLAIM min-idle-time option to XREADGROUP that does the same as part of a normal read.[7][3]
Poison entries and the delivery count#
Keeping an entry pending after a crash is exactly right for transient failures, and exactly wrong for an entry that will always crash its consumer. The consumer restarts, re-reads its pending entries, gets the same one, and crashes again.[1] With XAUTOCLAIM in play, the entry moves to another consumer, which then crashes on it too.
The delivery count is the way out. Redis has no dead-letter queue built in, so consumers check the count (from XPENDING orders mailers - + 10, or in the reply of XREADGROUP … CLAIM on 8.4+) and, past a limit, copy the entry to another stream (XADD orders:dead …) and XACK the original.[1][3]
See it happen
Trimming ignores consumer groups#
Streams don't expire entries on their own. You cap them on write: XADD orders MAXLEN ~ 10000 * … (or MINID to drop entries older than an ID). With ~, Redis only removes whole internal nodes (up to stream-node-max-entries, default 100), so the stream can be slightly longer than the threshold, which is much cheaper.[1][8]
Trimming doesn't look at consumer groups. An entry a lagging group never read is deleted like any other, and nothing reports an error. Afterwards, XINFO GROUPS shows that group's lag as nil until it catches up to the end of the stream. A trimmed entry that was still pending keeps its ID in the PEL: reading it returns the ID with nil fields.[2][6][3]
See it happen
Restarts and persistence#
A stream is a key, so it is saved in RDB snapshots and the AOF like any other, and so are its consumer groups: Redis writes the group changes caused by XREADGROUP and XACK to the AOF and to replicas. After a restart with appendonly yes, each group resumes at its last-delivered-id with its PEL intact.[1] Compare Pub/Sub, which has no relation to the key space.[9]
What doesn't survive is anything in flight. An XADD sent while Redis is down fails with ECONNREFUSED; the producer has to retry. An XACK that didn't reach Redis leaves the entry pending, so the consumer processes it again after reconnecting.[10][1]
See it happen
When to use something else#
Streams give you a history, ownership of in-flight work and retries, inside the Redis you may already run. The trade-off is memory: Redis keeps the whole dataset in RAM.[11]
| Redis Pub/Sub | Redis Streams | Kafka | |
|---|---|---|---|
| Stores messages | No | Yes, as a key in RAM (RDB/AOF on disk) | Yes, a replicated log on disk |
| Delivery | Push | Pull: XREADGROUP, can BLOCK | Pull: consumers fetch |
| Guarantee | At-most-once | At-least-once with XACK (at-most-once with NOACK) | At-least-once by default |
| Work sharing | None: every subscriber gets everything | Per entry, within a group | Per partition, within a group |
| Consumer offline | Misses everything | Resumes from the group's last-delivered-id | Resumes from its committed offset |
| Dead consumer's work | Lost | Pending until claimed (XAUTOCLAIM) | Partition reassigned on rebalance |
| Slow consumer | Grows its output buffer in Redis until closed | Lag grows (until trimmed) | Lag grows (until retention deletes data) |