You write to the database. You also need to publish an event ("order placed → send confirmation email"). If those two steps aren't atomic, you'll either lose events (DB committed, queue publish failed) or send duplicates (publish succeeded, DB rolled back). The outbox pattern solves this by treating the event as part of the same DB transaction, then publishing from the outbox to your message bus asynchronously. With MongoDB change streams (Post 12), the publisher is one process tailing the outbox collection — no Kafka Connect, no extra infrastructure. This post covers the full pattern, retry + idempotency + ordering, and the deduplication strategies downstream consumers need.

Series map: 18 Mongoose → 19 Beanie+FastAPI → 20 Outbox (this post) → 21 CQRS + ES → 22 RAG → ... → 29 K8s + mongosync + cost.

Learning Outcome

Recognise the dual-write problem.
Build an outbox collection alongside business writes inside one transaction.
Build a change-stream-driven publisher with resume tokens.
Handle at-least-once delivery + downstream idempotency.
Pick between outbox + change streams vs Debezium / Kafka Connect.

Chapter 1 — The dual-write problem

1
2
3
4 # WRONG — not atomic
await db.orders.insert_one(order)
await kafka.publish("orders.created", order)

Two failure modes:

DB insert succeeds, Kafka publish fails → lost event.
Kafka publish succeeds, DB insert fails → phantom event (consumers act on data that doesn't exist).

Neither is acceptable for systems that need consistency between their database state + downstream consumers.

Naive fixes that don't work

Try-catch + retry: same race, just longer odds.
2-phase commit across MongoDB + Kafka: not supported across systems.
Eventual consistency + reconciliation: complex; high latency.

The outbox

Write the event as a row in the same database, same transaction as the business mutation. A separate process publishes from the outbox to Kafka / RabbitMQ / SQS / SNS, marking each event as published (or deleting it).

Because the business write + outbox write are in one transaction, they're atomic. The publisher is async + retryable; downstream consumers handle duplicates idempotently.

The outbox pattern end to end: one transaction, a change-stream publisher to Kafka, and an idempotent consumer

Chapter 2 — Outbox schema

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19 // collection: outbox
{
  _id:           ObjectId(),
  aggregate:     "order",                  // entity type
  aggregate_id:  "o123",                   // entity id (for partitioning)
  event_type:    "order.created",
  payload:       { order_id: "o123", user_id: 1, total: 50 },
  version:       1,                         // schema version
  occurred_at:   ISODate(),
  published_at:  null,                      // set when publisher acks
  attempts:      0,
}
 
// indexes
db.outbox.createIndex({ published_at: 1, occurred_at: 1 })
db.outbox.createIndex({ aggregate_id: 1, occurred_at: 1 })
// optional TTL on published events
db.outbox.createIndex({ published_at: 1 }, { expireAfterSeconds: 60 * 60 * 24 * 7 })

published_at: null means unprocessed. The publisher claims, processes, marks.

Chapter 3 — Producer side — transactional write

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20 const session = client.startSession();
try {
  await session.withTransaction(async () => {
    await db.collection('orders').insertOne(order, { session });
 
    await db.collection('outbox').insertOne({
      aggregate:    'order',
      aggregate_id: order._id,
      event_type:   'order.created',
      payload:      { order_id: order._id, user_id: order.user_id, total: order.total },
      version:      1,
      occurred_at:  new Date(),
      published_at: null,
      attempts:     0,
    }, { session });
  });
} finally {
  await session.endSession();
}

Both writes commit together or neither does. The business state and the to-be-published event are forever consistent.

Chapter 4 — Publisher — change-stream-driven (the cheap option)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22 const cs = db.collection('outbox').watch(
  [{ $match: { operationType: 'insert' } }],
  { fullDocument: 'updateLookup' }
);
 
let lastToken = await loadResumeToken();
 
const stream = lastToken ? db.collection('outbox').watch([], { resumeAfter: lastToken }) : cs;
 
for await (const change of stream) {
  const event = change.fullDocument;
  try {
    await kafka.send({ topic: event.event_type, key: event.aggregate_id, value: JSON.stringify(event) });
    await db.collection('outbox').updateOne({ _id: event._id }, { $set: { published_at: new Date() } });
    await persistResumeToken(change._id);
  } catch (err) {
    // log + back off; do NOT persist the token until success → consumer will re-receive on restart
    await db.collection('outbox').updateOne({ _id: event._id }, { $inc: { attempts: 1 } });
    await sleep(backoff(event.attempts));
  }
}

Properties:

At-least-once delivery — on crash before token persist, we replay.
Resume token persisted after successful publish + mark.
Network blip → retried with backoff, attempts incremented.

This pattern needs no external infrastructure beyond your existing MongoDB. The publisher can run anywhere — a sidecar, a long-running container, a dedicated tiny pod.

Chapter 5 — Publisher — polling fallback

If change streams aren't available (standalone Mongo, very old versions, or a multi-region setup with restricted secondary access):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21 async function publishLoop() {
  while (true) {
    const events = await db.collection('outbox')
      .find({ published_at: null })
      .sort({ occurred_at: 1 })
      .limit(100)
      .toArray();
 
    if (!events.length) { await sleep(1000); continue; }
 
    for (const ev of events) {
      try {
        await kafka.send({ topic: ev.event_type, key: ev.aggregate_id, value: JSON.stringify(ev) });
        await db.collection('outbox').updateOne({ _id: ev._id }, { $set: { published_at: new Date() } });
      } catch {
        await db.collection('outbox').updateOne({ _id: ev._id }, { $inc: { attempts: 1 } });
      }
    }
  }
}

Higher latency (poll interval) + more DB queries; rock-solid. Use change streams when available, polling as fallback.

Chapter 6 — At-least-once → consumer must be idempotent

Once a publisher delivers at-least-once, consumers will eventually see duplicates (publisher crashed mid-ack, network duplicate, replay after rebuild).

Idempotency patterns:

1. Event ID dedup

Every outbox event has a unique _id. Consumers track processed IDs (in a "consumed_events" collection, Redis set, etc.). Skip if already processed.

1
2
3
4
5
6
7
8 const r = await db.collection('consumed_events').updateOne(
  { _id: event._id },
  { $setOnInsert: { processed_at: new Date() } },
  { upsert: true }
);
if (r.upsertedCount === 0) return;          // already processed; skip
await applyEvent(event);

2. Idempotent operations

Make the consumer's side-effect itself idempotent: INSERT ... ON CONFLICT DO NOTHING, upsert by key, "set status to X" rather than increment.

3. Optimistic versioning

Event carries version; consumer rejects updates with version <= current_version.

Combine 1 + 2 for belt-and-braces.

Chapter 7 — Ordering guarantees

Kafka topics + partitions preserve order within a partition. Choose your partition key to match the ordering you need.

1
2
3
4
5
6 await kafka.send({
  topic: event.event_type,
  key:   event.aggregate_id,          // events for the same order land on same partition → ordered
  value: ...
});

For "all events on this order, in order" → key on aggregate_id.
For "every event in occurrence order globally" → single-partition topic (low throughput).

Most outbox usage cares about per-aggregate ordering — keying by aggregate_id is the right default.

What if you don't have Kafka?

SQS doesn't preserve order across the queue (unless FIFO with MessageGroupId). SNS doesn't (FIFO topics do). RabbitMQ exchanges + queues do preserve order per queue.

Match destination to the ordering requirement.

Chapter 8 — Backpressure + DLQ

When the downstream is down for hours:

Outbox grows. Disk usage climbs.
If you have TTL on published_at: null, you'll lose unpublished events — don't TTL the unpublished portion.
The publisher must implement exponential backoff to avoid hammering a downed downstream.

Dead Letter Queue pattern:

1
2
3
4
5
6 if (event.attempts > 10) {
  await db.collection('outbox_dlq').insertOne({ ...event, dlq_at: new Date() });
  await db.collection('outbox').deleteOne({ _id: event._id });
  alert("outbox event moved to DLQ", event);
}

Alert + investigate. Don't silently drop.

Chapter 9 — Outbox + sagas (cross-service workflows)

For workflows spanning services (order → payment → inventory → shipping), combine outbox events with a saga orchestrator. Each step is one service's local transaction + outbox publication. The next service's consumer triggers its step.

If a step fails, the orchestrator publishes a compensating action event → previous services undo.

Outbox makes each step atomic + reliable. The saga adds the cross-step orchestration. Together they're the standard pattern for distributed workflows on document databases.

Chapter 10 — When to use outbox + change streams vs Debezium

	Outbox + change streams	Debezium MongoDB connector
Sync vs async	async publisher process	Kafka Connect cluster
Infrastructure	minimal (just Mongo)	Kafka + Connect
At-least-once	yes	yes
Event format	yours	Debezium envelope (uniform across DBs)
Schema migrations	app-aware	DB-shape only
Best for	one service, simple ops	many services, polyglot DBs into Kafka

If you already run Kafka + Connect, Debezium is a thin add. If you don't, outbox + change-stream publisher is a cheap, self-contained pattern that scales to many millions of events/day.

Post 24 covers Kafka + Debezium specifically.

Chapter 11 — Common pitfalls

Pitfall	Cause	Fix
Lost events	non-transactional dual-write	use outbox
Phantom events on rollback	publish before commit	publish only from outbox post-commit
Duplicates downstream	at-least-once delivery	idempotent consumers
Publisher process crash → unbounded replay	no resume token persistence	save token after each ack
Outbox bloats	publisher down for long	alert on size; DLQ chronic failures; back-pressure
Wrong partition ordering	random key	partition by aggregate_id
Outbox writes blocked by transaction lifetime cap	huge transactions	smaller business transactions
Forgot to mark published	infinite loop re-publishing	always `$set: published_at` post-ack

Practice Exercises

Exercise 1 — Build the outbox

Implement placeOrder from Post 09 that also writes an outbox entry inside the transaction. Confirm both rows appear or neither.

Exercise 2 — Build the publisher

A Node script that change-streams the outbox + prints events to stdout (simulating Kafka). Mark them published.

Exercise 3 — Idempotent consumer

A second script consuming "Kafka" (file or stdin) that dedupes by event _id against a consumed_events collection. Replay the same events twice; confirm the side effect runs once.

Exercise 4 — Crash recovery

Kill the publisher mid-batch. Restart. Confirm no event is lost (you may see duplicates — the consumer dedups them).

Exercise 5 — DLQ

Make the consumer fail for a specific event_type. After 10 attempts, move to outbox_dlq. Confirm.

Exercise 6 — Compare to Debezium

(If Kafka available.) Set up Debezium MongoDB source on the same outbox collection. Compare event shape + ordering + ops surface.

Cheat-sheet

What	How
Problem	DB write + external publish not atomic
Solution	write event to `outbox` collection inside same transaction
Publisher	tail outbox via change stream → publish → mark `published_at`
Persistence	save resume token AFTER successful publish + mark
Delivery	at-least-once
Consumer	idempotent — dedup by event `_id` or use idempotent ops
Ordering	partition by `aggregate_id` in Kafka
Backpressure	back-off; alert on outbox size; DLQ chronic failures
TTL	only on `published_at: { $ne: null }`
Alt: Debezium	when Kafka + Connect already in your infra
Sagas	outbox per service + cross-service orchestrator

What's next

Post 21 covers CQRS + Event Sourcing on MongoDB — separate read + write models, event-stream as source of truth, materialised projections, and the patterns that make them work without overengineering.

References

Outbox Pattern (microservices.io)
Debezium Outbox Event Router
MongoDB Change Streams
Designing Data-Intensive Applications by Martin Kleppmann — Ch 11 (stream processing).

Note: this post is in progress. Planned additions: full TypeScript example service with outbox + publisher + idempotent consumer; Kafka topic partitioning rules; outbox + sagas worked example.