redis-kafka-wal-lab — how it works

What this is

An active-active, geo-distributed key/value system built from open-source pieces: per-region Apache Kafka clusters, per-region Redis OSS instances, and MirrorMaker 2 wiring them together. The dashboard at redis-kafka-wal.pages.dev is a live view of three regions running on a single VM, with a traffic simulator and a writer panel so you can see convergence happen in real time.

The interesting choice is the shape: Kafka is the source of truth for every write, and the three Redis instances are materialized views built by consumer services in each region. Anything you write is durable on Kafka before it's visible anywhere; anything in Redis is rebuildable by replaying the log from offset zero.

Why this shape, vs. just using Redis Enterprise

Redis Enterprise has built-in active-active geo-replication using CRDTs (their CRDB feature). That's the natural “Redis is the whole product” design and it's lower latency than what we do here. Our design pays a few extra hops on the write path (Redis-write → Kafka-produce → MirrorMaker → Kafka-consume → Lua-apply → Redis) in exchange for two properties Enterprise doesn't give you:

• The log is yours. Every write is durable in Kafka and consumable by anything else — analytics, a search index, a fraud-detection job. With Enterprise the replication log is internal.
• Replay-from-zero is a first-class operation. Wipe a region's Redis, rewind the consumer to offset 0, and rebuild deterministic state. With Enterprise you sync from a peer cluster instead.

It's not a cheap-replacement-for: it's a different design point. The right answer depends on whether Redis is your system or one of several views into a stream platform.

Topology

Three regions: us, eu, ap. Each one runs:

• A single-node Kafka cluster in KRaft mode.
• A single-node Redis instance.
• A Go materializer service that consumes events and applies them to local Redis.

Three topics — us.events, eu.events, ap.events — one per origin region. Each topic is only ever produced into by clients in its owning region; MirrorMaker 2 carries it outward to the other two clusters using IdentityReplicationPolicy, so the topic name is the same on every cluster. This is what makes mirror loops structurally impossible: each MM2 directional flow is filtered to only the source region's own topic.

US region              EU region              AP region
┌────────────┐    ┌────────────┐    ┌────────────┐
│ Kafka-US   │⇄  │ Kafka-EU   │⇄  │ Kafka-AP   │
│ us.events  │    │ us.events  │    │ us.events  │
│ eu.events  │    │ eu.events  │    │ eu.events  │
│ ap.events  │    │ ap.events  │    │ ap.events  │
└─────┬──────┘    └─────┬──────┘    └─────┬──────┘
      │                  │                  │
   Materializer      Materializer       Materializer
      │                  │                  │
      ▼                  ▼                  ▼
   Redis-US           Redis-EU            Redis-AP

Six MM2 directional flows wire every pair. Producers point at their own region's Kafka (so writes are local-latency). Consumers in each region subscribe to all three *.events topics on the local cluster — the local one plus the two mirrored copies.

One write, traced end-to-end

Suppose a user clicks buy in the dashboard and the write panel sends INCR sales:total +1 against region eu. Here's what happens:

1. Browser → Cloudflare Pages. The dashboard posts JSON to /api/produce on the bridge service via the Cloudflare Tunnel hostname.
2. Bridge: HLC tick. The bridge has one Hybrid Logical Clock per region; for an EU write it ticks the EU clock and stamps the event with (physical_ms, logical, region=eu). The HLC gives a total order on events even when wall clocks are skewed across regions.
3. Bridge: Avro encode. The event is serialized with Confluent wire format: a magic byte, the schema id from Schema Registry, then the Avro binary payload.
4. Bridge: Kafka produce. The encoded record lands on eu.events at kafka-eu, partition = hash(key) % 12. This is the only step that has to succeed for the write to be considered durable.
5. Local apply. The materializer in EU has subscribed to all three *.events topics on kafka-eu. It picks up the record, decodes it, and runs the per-op Lua script in redis-eu. For INCR, the script checks a per-origin sequence number to dedup, then atomically updates the counter and the seq sidecar.
6. Mirror to peers. MirrorMaker is reading eu.events off kafka-eu and writing to kafka-us and kafka-ap. After ~milliseconds, the same record is on both peer clusters.
7. Remote apply. US's and AP's materializers consume from their local cluster (which now has the mirrored record). Same Lua script runs against redis-us and redis-ap. Same dedup guarantees apply — if the record was already applied (for instance, replay after a crash), the script's seq-greater check rejects it as a no-op.

At this point all three Redis instances have the same value for sales:total. The path the dashboard cares about is the SSE event stream: each of those local apply events in step 5/7 is also seen by a separate set of consumers inside the bridge that fan events out to the browser.

How conflicts are resolved

Active-active means two regions can write to the same key within the MM2 propagation window and arrive at the materializer in either order. Each Redis op family uses a different convergent strategy:

Each apply happens inside a Redis Lua script so the read-decide-write cycle is atomic against concurrent applies on the same key. Here's the LWW script (slightly trimmed):

-- KEYS[1] = key, KEYS[2] = meta
-- ARGV[1..3] = phys, logc, reg     (incoming HLC)
-- ARGV[4]   = op (SET|DEL)
-- ARGV[5]   = value (SET only)
-- ARGV[6]   = ttl_ms (SET only)
local cur = redis.call('HMGET', KEYS[2], 'phys', 'logc', 'reg')
local cphys = tonumber(cur[1]) or -1
local clogc = tonumber(cur[2]) or -1
local creg  = cur[3] or ''
local iphys = tonumber(ARGV[1])
local ilogc = tonumber(ARGV[2])
local ireg  = ARGV[3]

local accept = false
if     iphys > cphys then accept = true
elseif iphys == cphys and ilogc > clogc then accept = true
elseif iphys == cphys and ilogc == clogc and ireg > creg then accept = true
end
if not accept then return 0 end

if ARGV[4] == 'SET' then
  redis.call('SET', KEYS[1], ARGV[5])
  if tonumber(ARGV[6]) > 0 then redis.call('PEXPIRE', KEYS[1], tonumber(ARGV[6])) end
elseif ARGV[4] == 'DEL' then
  redis.call('DEL', KEYS[1])
end
redis.call('HSET', KEYS[2], 'phys', iphys, 'logc', ilogc, 'reg', ireg, 'op', ARGV[4])
return 1

The same pattern repeats for the other op types — the meta sidecar and comparison rule changes, but the read-then-conditionally-apply structure is the same. There's one deliberate simplification: SADD/SREM is LWW-Element-Set, not a true OR-Set. A concurrent SADD m from one region can lose to a slightly-later SREM m from another even if the SREM didn't observe the SADD. For tag sets and feature flags that's fine; for shopping carts where “add wins on conflict” is the right rule, you'd swap in a real OR-Set with per-tag state.

What you're seeing on the dashboard

The page is wired together as follows:

• Three region columns. Each polls GET /api/regions/<region>/state on the bridge every 1.5 seconds. The bridge reads the same set of shopping-themed keys directly from that region's local Redis and serves them as JSON.
• Live event log. The bridge spawns three read-only Kafka consumers (one per region). Every event each one sees is annotated with which region's Kafka observed it and pushed to a fan-out broadcaster. The browser opens a Server-Sent Events stream at GET /api/events and renders one row per unique event_id; the colored blocks on each row fill in as the same event is observed at us-Kafka, then eu-Kafka after MM2 lag, then ap-Kafka. That's the MM2-propagation visualization.
• Live traffic simulator. A goroutine inside the bridge fires a weighted op mix (50% purchase, 20% cart add, 10% drop, 10% VIP join, 5% restock, 5% inventory race) at a configurable rate. POST /api/sim/start starts it; /api/sim/stop stops; /api/sim/rate retunes mid-flight without restart.
• Produce panel. POST /api/produce takes a JSON body describing one op and runs it through the same internal doProduce() function the simulator uses. Bearer token required (set in “settings”).

Failure modes

The repo has scripts to reproduce each scenario:

• Region down (scripts/failover.sh): kill a region's Kafka and materializer; the other two keep accepting writes; on recovery, MM2 drains the backlog and HLC-gated apply means older events lose to anything newer that landed during the outage. Final state on the recovered region matches the rest.
• Network partition (scripts/partition.sh): disconnect a region's Kafka from the docker network. Each side accepts local writes and diverges. After healing, MM2 reconciles in both directions; CRDT/LWW semantics resolve concurrent writes.
• Redis loss / replay (scripts/replay.sh): stop the consumer, FLUSHDB, reset the consumer group offsets to earliest, restart the consumer. The materializer rebuilds Redis state by reading all events from every origin region. Idempotent handlers make this exactly correct (with one documented caveat for streams — see ADR-0008).
• MirrorMaker lag: shows up as growing end-offset gaps between origin and target clusters, visible in scripts/status.sh. The materializers keep consuming local events fine; only cross-region freshness suffers.

Tech stack

Code organization: cmd/bridge, cmd/consumer, cmd/producer are the entry points; pkg/hlc, pkg/event, pkg/crdt are the building-block libraries; web/ is the dashboard you're looking at; infra/k8s and infra/terraform are sketches for taking it beyond the docker-compose lab. See docs/ for the architecture / consistency / conflict-resolution writeups and the ten ADRs covering every load-bearing decision.

Try it

• Open the dashboard in another tab.
• Click start on the live traffic simulator. Three columns will move in near-lockstep with sub-second divergence visible during bursts.
• In settings, paste the bearer token, then use the produce panel to write your own ops. Watch the event log emit one row, observed in three regions with three colored blocks.
• From the repo: ./scripts/scenario.sh --reset for the narrated 40-second flash-sale walkthrough.