syswork
Redis/

Streams & consumer groups

SimulatorHow it works

Redis · Streams & consumer groups · how it works

How Redis Streams consumer groups actually work

An append-only log stored as a key, read through consumer groups that track what each consumer was given and hasn't acknowledged. Nothing is lost when a consumer dies, but nothing is redelivered unless you claim it.

Delivery
at-least-once
Storage
a key (RDB/AOF)
Model
pull, can BLOCK
Recovery
PEL + XAUTOCLAIM

Modelled on Redis Open Source 8.10 · sources

▶ Open the simulator8 scenarios ↓

On this page

  • The path of one entry
  • Groups, consumers and lag
  • The PEL and XACK
  • Taking over stuck entries
  • Poison entries
  • Trimming ignores groups
  • Restarts and persistence
  • When to use something else
  • What this simulation simplifies
  • Every scenario, step by step
  • References

The path of one entry#

A stream is a Redis key holding an append-only sequence of entries, each a small set of field-value pairs with an ID. A consumer group is state stored inside that key: a position in the stream (last-delivered-id) and a pending entries list (PEL) of what was handed out but not yet acknowledged. Consumers are just names within a group.[1]

checkout-apiXADD orders *← "10-0"redis-server · the key orders (a stream)12345678910group mailerslast-delivered-id 9-0 · lag 1PEL (Pending Entries List): Redis Streams' per-consumer-group list of entries delivered to a consumer but not yet acknowledged with XACK.PEL #8 → mailer-2 · idle 0.3s · 1PEL (Pending Entries List): Redis Streams' per-consumer-group list of entries delivered to a consumer but not yet acknowledged with XACK. #9 → mailer-1 · idle 0.1s · 1group analyticslast-delivered-id 6-0 · lag 4PEL (Pending Entries List): Redis Streams' per-consumer-group list of entries delivered to a consumer but not yet acknowledged with XACK.PEL #6 → analytics-1 · idle 0.2s · 1mailer-1processing #9mailer-2processing #8analytics-1processing #6XACK removes thePEL (Pending Entries List): Redis Streams' per-consumer-group list of entries delivered to a consumer but not yet acknowledged with XACK.entry from the PEL1234
  1. XADD orders * order 10 appends an entry. Redis generates the ID (<milliseconds>-<sequence>) and replies with it.[2]
  2. The XADD wakes consumers blocked in XREADGROUP … BLOCK. Each group hands out entries after its own last-delivered-id, moves it forward, and records each entry in its PEL with the consumer's name, the delivery time and a delivery count.[3]
  3. The consumer processes the entry. Meanwhile the entry's idle time grows in the PEL.
  4. XACK orders mailers <id> removes the entry from the group's PEL. Until then, Redis treats it as possibly unprocessed.[4]

The entry itself stays in the stream after every group has acknowledged it. Acknowledging only changes the group's bookkeeping; deleting entries is a separate concern (trimming, below).[1]

See it happen

  • 01Consumer groups: the happy path→

Groups, consumers and lag#

Consumers in the same group share the work: each entry goes to one of them. Different groups are independent: each gets every entry, at its own pace. Here mailers has two consumers that take turns, and analytics has one that sees all orders.[1][3] Each group is created once, with XGROUP CREATE orders mailers $ MKSTREAM: $ means it starts after the stream's current last entry, and MKSTREAM creates the stream if it doesn't exist yet.[5]

XREADGROUPcommand
GROUP mailers mailer-1group, consumer
COUNT 4max per reply
BLOCK 5000wait ≤ 5s
STREAMS orderskey
>new only

Delivery is pull: a consumer receives at most COUNT entries per call, and with BLOCK it waits inside Redis until there is something to return.[3] A slow consumer doesn't grow a buffer inside Redis the way a Pub/Sub subscriber does; the entries wait in the stream and the group's lag grows. XINFO GROUPS reports it as the number of entries in the stream that are still waiting to be delivered to the group.[6]

A consumer that goes away for a while loses nothing. When it comes back, > continues from the group's last-delivered-id, and it works through the backlog.[1]

See it happen

  • 01Consumer groups: the happy path→
  • 02A consumer is down, then catches up→

The PEL and XACK: at-least-once#

The PEL is what turns a log into a work queue. Each entry in it has an owner, the time it was last delivered (from which XPENDING computes idle time) and how many times it was delivered. A consumer that crashes before XACK leaves its entries there, owned by it.[1]

On restart, the usual pattern is to first read with ID 0 instead of >: XREADGROUP … STREAMS orders 0 returns this consumer's own pending entries, and adds 1 to their delivery count. When that comes back empty, switch to >.[3]

Reading with NOACK skips the PEL entirely: an entry counts as acknowledged the moment it is delivered. It is meant for cases where occasional message loss is acceptable: a crash drops whatever the consumer was holding.[3]

See it happen

  • 07Without XACK: NOACK and a crash→

Taking over stuck entries#

Redis never redelivers on its own. If a consumer is gone for good (scaled down, its host lost), its pending entries wait forever. Another consumer has to take them over:[1]

XAUTOCLAIMcommand (6.2+)
orders mailerskey + group
mailer-2new owner
6000min-idle-time (ms)
0-0scan cursor
COUNT 10max to claim

It scans the group's PEL for entries idle at least min-idle-time, makes the caller their owner, resets their idle time and increments their delivery count. It replies with a cursor for the next call, the claimed entries, and (since 7.0) the IDs of pending entries whose data no longer exists, which it removes from the PEL. Because claiming resets idle time, two consumers can't both claim the same entry at the same moment. Redis 8.4 adds a CLAIM min-idle-time option to XREADGROUP that does the same as part of a normal read.[7][3]

Idle is not dead

Redis only sees time passing. A consumer stuck in a long call or a GC pause looks exactly like a dead one. Its entries get claimed and processed by someone else, and when it wakes up it processes them again; its XACK then returns 0.[4] Consumer groups are at-least-once: make handlers idempotent, and set min-idle-time well above your slowest normal processing time.[1]

See it happen

  • 03A consumer dies: XAUTOCLAIM rescues its entries→
  • 04A hung consumer wakes up: duplicate processing→

Poison entries and the delivery count#

Keeping an entry pending after a crash is exactly right for transient failures, and exactly wrong for an entry that will always crash its consumer. The consumer restarts, re-reads its pending entries, gets the same one, and crashes again.[1] With XAUTOCLAIM in play, the entry moves to another consumer, which then crashes on it too.

The delivery count is the way out. Redis has no dead-letter queue built in, so consumers check the count (from XPENDING orders mailers - + 10, or in the reply of XREADGROUP … CLAIM on 8.4+) and, past a limit, copy the entry to another stream (XADD orders:dead …) and XACK the original.[1][3]

See it happen

  • 05A poison entry and the dead-letter stream→

Trimming ignores consumer groups#

Streams don't expire entries on their own. You cap them on write: XADD orders MAXLEN ~ 10000 * … (or MINID to drop entries older than an ID). With ~, Redis only removes whole internal nodes (up to stream-node-max-entries, default 100), so the stream can be slightly longer than the threshold, which is much cheaper.[1][8]

Trimming doesn't look at consumer groups. An entry a lagging group never read is deleted like any other, and nothing reports an error. Afterwards, XINFO GROUPS shows that group's lag as nil until it catches up to the end of the stream. A trimmed entry that was still pending keeps its ID in the PEL: reading it returns the ID with nil fields.[2][6][3]

A slow group can silently lose data

Size MAXLEN for the longest outage any group must survive, alert on lag, or on Redis 8.2+ trim with XADD orders ACKED MAXLEN …: ACKED only removes entries that every group has read and acknowledged (KEEPREF, the default, removes them regardless; DELREF also drops them from every PEL).[2] Without any cap, the stream grows without bound.[1]

See it happen

  • 06MAXLEN trims entries a group never read→

Restarts and persistence#

A stream is a key, so it is saved in RDB snapshots and the AOF like any other, and so are its consumer groups: Redis writes the group changes caused by XREADGROUP and XACK to the AOF and to replicas. After a restart with appendonly yes, each group resumes at its last-delivered-id with its PEL intact.[1] Compare Pub/Sub, which has no relation to the key space.[9]

What doesn't survive is anything in flight. An XADD sent while Redis is down fails with ECONNREFUSED; the producer has to retry. An XACK that didn't reach Redis leaves the entry pending, so the consumer processes it again after reconnecting.[10][1]

See it happen

  • 08Redis restarts: the stream survives→

When to use something else#

Streams give you a history, ownership of in-flight work and retries, inside the Redis you may already run. The trade-off is memory: Redis keeps the whole dataset in RAM.[11]

Redis Pub/SubRedis StreamsKafka
Stores messagesNoYes, as a key in RAM (RDB/AOF on disk)Yes, a replicated log on disk
DeliveryPushPull: XREADGROUP, can BLOCKPull: consumers fetch
GuaranteeAt-most-onceAt-least-once with XACK (at-most-once with NOACK)At-least-once by default
Work sharingNone: every subscriber gets everythingPer entry, within a groupPer partition, within a group
Consumer offlineMisses everythingResumes from the group's last-delivered-idResumes from its committed offset
Dead consumer's workLostPending until claimed (XAUTOCLAIM)Partition reassigned on rebalance
Slow consumerGrows its output buffer in Redis until closedLag grows (until trimmed)Lag grows (until retention deletes data)

Sources: Pub/Sub[9][12], Streams[1], Kafka[13][14][15].

  • Losing messages is fine? Pub/Sub is simpler and keeps no history.[9]
  • Need long retention or a history larger than RAM? A log on disk like Kafka. It hands out whole partitions: each is consumed by exactly one consumer in a group, in order.[13][15] Streams hand out individual entries.

What this simulation simplifies#

Model vs reality

  • Time is slowed down about a thousand times, and stream sizes are scaled down to match (MAXLEN 40 instead of thousands). Entry IDs use the simulation clock in milliseconds, not Unix time, so they are short.
  • Trimming is exact (=), not node-at-a-time (~).
  • Each consumer processes one entry at a time and sends one XACK per entry. XACK accepts several IDs in one call.[4] The XAUTOCLAIM loop runs every 3 s on its own connection, with COUNT 10.
  • The dead-letter check reads the delivery count Redis tracks; real code gets it from XPENDING or XREADGROUP … CLAIM.
  • The restart reloads everything. With appendfsync everysec (the default), up to about a second of writes before a crash can be lost.[8]
  • A consumer that goes offline closes its connection at once. A lost host would leave a half-open connection until TCP keepalive notices[16][12]; the PEL behaves the same either way.

Every scenario, step by step

What happens in each scenario and when. Times are simulated seconds, as on the simulator's timeline. Click a time to open the simulator paused at that moment, then take control.

01

Consumer groups: the happy path

checkout-api appends orders to the stream. Two mailers share the work in one group; analytics reads every entry in its own group. Each entry is pending until its consumer XACKs it.

▶ Run it

SetupDefaults. checkout-api runs XADD orders MAXLEN 40 * … about every 900 ms. Group mailers has two consumers (300 ms per entry), group analytics has one (300 ms). Every consumer loops on XREADGROUP … COUNT 4 BLOCK 5000 STREAMS orders >.

  1. 0.0sBoth groups are created with $: they start after the stream's current last entry, so neither will see anything older.
  2. 0.5sThe stream is empty, so each consumer's first XREADGROUP parks inside Redis. Consumers don't poll; they wait.
  3. 0.8sThe first XADD arrives. Redis gives it the ID 813-0 (milliseconds, then a sequence number) and replies with it.
  4. 0.8sThe same XADD wakes the blocked clients. mailer-1 blocked first, so it gets order 1; mailer-2 stays blocked. analytics-1 also gets order 1, because its group has its own last-delivered-id.
  5. 1.9smailer-1 finishes and runs XACK. The entry leaves the mailers PEL; the bar under it on the tape turns from pending to acked.
  6. ongoingFrom here on mailer-1 and mailer-2 take turns, and every PEL holds only the one or two entries currently being processed.

Within a group, consumers split the entries; across groups, each group gets all of them. An entry is pending from delivery until its XACK, and that list is what makes recovery possible.

Try next

  • Select mailer-2 and press Take offline: mailer-1 does all the work and nothing is skipped.
  • Lower COUNT to 1 in Blocks & settings and watch the analytics lag climb: one round trip per entry is too slow in slow motion.
02

A consumer is down, then catches up

analytics-1 is redeployed. Unlike Pub/Sub, nothing is lost: the entries wait in the stream, and the group's last-delivered-id remembers where it stopped.

▶ Run it

Setupanalytics-1 goes offline at 3 s (think: a deploy) and comes back at 11 s.

  1. 3.0sanalytics-1's process stops. Nothing changes inside Redis: the group keeps its last-delivered-id, and new entries keep landing in the stream.
  2. 10.5sThe analytics group's lag is 10: ten entries added since its last-delivered-id. The mailers carry on as before.
  3. 11.0sanalytics-1 starts again, reads its own pending entries (none), then > from where the group stopped.
  4. 20.5sReading 4 entries per round trip, it has worked the lag back down to 1. Every order it missed while down was processed.

A stream is storage, so a consumer that goes away loses nothing: its group remembers where it was. Compare Pub/Sub, where the same restart is a permanent gap.

Try next

  • Take analytics-1 offline for longer, past MAXLEN entries, and see the MAXLEN trims entries a group never read scenario happen by hand.
03

A consumer dies: XAUTOCLAIM rescues its entries

mailer-1 is slow, so it holds a batch of entries. Then its host disappears. Its entries stay pending in the PEL, owned by a consumer that will never come back, until mailer-2 claims them.

▶ Run it

Setupmailer-1 is slow (1500 ms per entry). A burst of 10 orders at 4 s gives it a full batch of 4. At 7 s its host disappears and it never comes back.

  1. 4.0sMore entries arrive than the blocked consumers asked for, so the next reads return full batches. mailer-1 takes orders 10 to 13.
  2. 7.0smailer-1 is gone with 4 entries pending. Redis doesn't notice anything about the entries: they stay in the PEL, owned by mailer-1, and their idle time grows.
  3. 12.5sTheir idle time passes min-idle-time (6 s). The rows turn red: they are now claimable.
  4. 13.4smailer-2's periodic XAUTOCLAIM orders mailers mailer-2 6000 0-0 COUNT 10 takes all four. Their owner becomes mailer-2, idle resets, and the delivery count goes to 2.

Redis never redelivers on its own. Pending entries of a dead consumer stay put until another consumer claims them, with XAUTOCLAIM or XPENDING + XCLAIM.

Try next

  • Pull out the XAUTOCLAIM loop block: the four entries stay pending for good, until you Bring online mailer-1.
  • Lower min-idle-time and see how much sooner they're rescued.
04

A hung consumer wakes up: duplicate processing

mailer-1 hangs while holding entries. They are claimed and processed by mailer-2. When mailer-1 recovers it finishes the same entries again. At-least-once means duplicates.

▶ Run it

Setupmailer-1 takes 1200 ms per entry. A burst at 3.5 s gives it a batch of 4. At 6.5 s it hangs (keeps its connection, processes nothing); at 18 s it recovers.

  1. 6.5smailer-1 stops in the middle of order 10, with 11 to 13 waiting in its local batch. Its connection is fine, so Redis sees nothing unusual.
  2. 13.4sAfter 6 s idle, mailer-2 claims all four, processes them and XACKs them. From Redis's point of view they're done.
  3. 18.0smailer-1 wakes up. It has no idea anything happened and carries on with the batch it was holding.
  4. 18.6sOrder 10 is processed twice (two emails). Its XACK returns 0: there's nothing left to remove. Orders 11 to 13 follow.

Claiming can't tell a dead consumer from a slow one. Consumer groups deliver at least once, so handlers must be idempotent (dedupe on the entry ID), and min-idle-time must be well above your slowest normal processing time.

Try next

  • Raise min-idle-time above the hang (say 15 s): no claim, no duplicate, but a longer delay.
05

A poison entry and the dead-letter stream

One malformed order crashes whichever mailer processes it. The entry stays pending, so the restarted mailer gets it again and crashes again. The delivery count breaks the loop.

▶ Run it

SetupAt 2.5 s checkout-api adds one malformed order. The mailers' email template throws on it; analytics only counts it. Max deliveries is 3.

  1. 2.5sOrder 4 goes into the stream like any other entry.
  2. 3.6smailer-2 gets it and its process exits. Order 4 stays pending in the mailers PEL with delivery count 1. A supervisor restarts mailer-2.
  3. 6.9sOn start, mailer-2 reads its own history with XREADGROUP … STREAMS orders 0. That's how it recovers work it held, and here it hands back the poison entry, delivery count 2.
  4. 7.2sCrash again. The same thing repeats at 10.8 s with delivery count 3.
  5. 14.4sThe fourth delivery is over the limit. Instead of processing it, mailer-2 runs XADD orders:dead with the entry and XACKs the original. The loop is broken.

A crash-before-XACK keeps the entry pending, which is exactly right for transient failures and exactly wrong for a bad payload. The delivery count is the signal; dead-lettering is application code, not a Redis feature.

Try next

  • Pull out the Dead-letter stream block: mailer-2 crash-loops for as long as you let it run.
  • From checkout-api, XADD a malformed order while a mailer is down, and see which one it hits.
06

MAXLEN trims entries a group never read

The stream is capped at MAXLEN 10. analytics-1 is down for 16 seconds. Trimming doesn't wait for consumer groups, so the oldest unread entries are cut before analytics gets back.

▶ Run it

SetupThe stream is capped at MAXLEN 10. analytics-1 goes offline at 2 s and comes back at 18 s, while orders keep arriving.

  1. 2.0sanalytics-1 stops, holding order 2 (pending). The analytics group's position stays at order 2.
  2. 10.5sThe 11th entry pushes order 2 off the left edge. Its ID stays in the analytics PEL (the default KEEPREF), but the data is gone.
  3. 11.5sOrder 3, never delivered to analytics, is trimmed. Lag turns to nil: Redis can no longer tell how many entries the group missed.
  4. 18.8sBack online, analytics-1 reads its history and gets order 2's ID with nil fields. It acknowledges it without processing.
  5. 21.0sThen > resumes at the oldest entry still in the stream, order 13. Orders 3 to 12 were trimmed unread: 10 entries the analytics group never saw, and no error anywhere.

Trimming is about memory, not consumers. Size MAXLEN (or MINID) for the longest outage a group must survive, alert on lag, or, on Redis 8.2+, trim with ACKED so entries are kept until every group has acknowledged them.

Try next

  • Pull out MAXLEN trimming and rerun: nothing is lost, and XLEN keeps growing instead.
07

Without XACK: NOACK and a crash

The acknowledgement block is removed: consumers read with NOACK. Nothing is ever pending, which is fine until a consumer crashes while holding entries.

▶ Run it

SetupThe XACK and the PEL block is removed: consumers read with XREADGROUP … NOACK. mailer-1 is slow (1500 ms), a burst at 4 s gives it a batch, and at 7 s it is killed.

  1. ongoingBoth PELs stay empty the whole time. With NOACK, an entry counts as acknowledged the moment it's delivered.
  2. 7.0smailer-1 dies holding orders 10, 11, 12 and 13. Redis has no record that they were in flight, so nothing can claim them: they are lost for the mailers group.
  3. 9.5smailer-1 restarts, finds nothing pending, and carries on with new entries. The four orders are never emailed. (analytics, a separate group, still processed them.)

NOACK turns a consumer group into at-most-once delivery, meant for when occasional loss is acceptable: a crash silently drops whatever the consumer held.

Try next

  • Put the block back and rerun: the same crash leaves four pending entries that mailer-1 re-reads on restart.
08

Redis restarts: the stream survives

redis-server crashes and comes back. Unlike Pub/Sub subscriptions, the stream and the consumer groups are data, reloaded from the AOF. XADDs and XACKs sent during the outage are lost.

▶ Run it

Setupredis-server crashes at 5 s and a supervisor restarts it 3 s later. It runs with appendonly yes.

  1. 5.0sEvery connection drops. Consumers keep processing the entries they already hold, but their XACKs fail.
  2. 5.0sXADDs during the outage fail with ECONNREFUSED. Unless the producer retries, those orders never exist.
  3. 8.0sRedis replays the AOF: the stream, each group's last-delivered-id and its PEL are back, including the entries whose XACK was lost.
  4. 8.7sConsumers reconnect and read their own pending entries first. mailer-1 gets order 5 back, though it already processed it.
  5. 9.0sOrder 5 is processed again; this XACK is the one that finally removes it. analytics-1 does the same with order 4.

Unlike Pub/Sub subscriptions, a stream and its consumer groups are data, so they survive a restart. What doesn't survive is anything only on the wire: XADDs (producer must retry) and XACKs (so entries come back as duplicates).

Try next

  • Select the stream and Crash redis-server yourself while a mailer holds a big batch (send a Burst of 10 first).

References

Modelled on Redis Open Source 8.10. Commands and behaviour follow the redis.io documentation for this release. Features added in recent releases are marked with their version where they appear (XAUTOCLAIM in 6.2, its deleted-ID reply in 7.0, KEEPREF/DELREF/ACKED in 8.2, XREADGROUP CLAIM in 8.4). Sources last checked 2026-11-16. Every claim on this page and in the simulator comes from the sources below; anything the simulation simplifies is listed under “What this simulation simplifies”.

  1. [1]Redis Streams. Redis documentation. Accessed 2026-11-16.
  2. [2]XADD. Redis command reference. Accessed 2026-11-16.
  3. [3]XREADGROUP. Redis command reference. Accessed 2026-11-16.
  4. [4]XACK. Redis command reference. Accessed 2026-11-16.
  5. [5]XGROUP CREATE. Redis command reference. Accessed 2026-11-16.
  6. [6]XINFO GROUPS. Redis command reference. Accessed 2026-11-16.
  7. [7]XAUTOCLAIM. Redis command reference. Accessed 2026-11-16.
  8. [8]redis.conf (self-documented default configuration). Redis source repository. Accessed 2026-11-16.
  9. [9]Redis Pub/Sub. Redis documentation. Accessed 2026-11-16.
  10. [10]connect(2), Linux manual page. Linux man-pages project. Accessed 2026-11-16.
  11. [11]Redis FAQ. Redis documentation. Accessed 2026-11-16.
  12. [12]Redis client handling. Redis documentation. Accessed 2026-11-16.
  13. [13]Apache Kafka design (4.3). Apache Kafka documentation. Accessed 2026-11-16.
  14. [14]KafkaConsumer (Kafka 4.3 API). Apache Kafka documentation. Accessed 2026-11-16.
  15. [15]Introduction (Kafka 4.3). Apache Kafka documentation. Accessed 2026-11-16.
  16. [16]RFC 9293: Transmission Control Protocol (TCP). IETF. Accessed 2026-11-16.