The standard way to flow data between MongoDB and Kafka in 2026 is two connectors: MongoDB Kafka Source Connector (Mongo → Kafka, change-stream-based, official) and MongoDB Kafka Sink Connector (Kafka → Mongo). For cross-database CDC (Postgres → Mongo, etc.), Debezium is the de facto. This post covers all three — what each does, when to use which, configurations that work in production, deduplication + ordering, and the common pitfalls (the failed-on-update-events trap, partition keys, schema registry).

Series map: 22 RAG → 23 SQL→Mongo migration → 24 Kafka + Debezium CDC (this post) → 25 Testing → 26 OpenTelemetry → 27 Geo → 28 Time-series extreme → 29 K8s + mongosync + cost.

Learning Outcome

  • Set up Kafka + Kafka Connect locally.
  • Configure the MongoDB Kafka Source connector with change streams.
  • Configure the MongoDB Kafka Sink connector to write to a collection.
  • Configure Debezium Postgres connector → Kafka → Mongo sink.
  • Handle ordering, deduplication, schema evolution.
  • Recognise when to use this stack vs the simpler outbox + change-stream publisher (Post 20).

Chapter 1 — Why Kafka + Connect

Outbox + change streams (Post 20) is the cheap, self-contained pattern. Kafka + Connect earn their complexity when:

  • You have many consumers for the same data (Mongo change → search index + analytics + cache + audit + downstream service).
  • You have multiple source DBs that should produce events in the same format (Postgres + Mongo + MySQL all into one Kafka topic family).
  • You need replayable storage of events for days/weeks (Kafka retention) independent of source DB oplog.
  • You're already on Kafka for other reasons.

If none apply, stick with Post 20. Connect adds: Kafka cluster, Schema Registry, Connect cluster, connector configs to maintain.

Chapter 2 — Architecture

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
┌──────────────┐                              ┌──────────────┐
│   MongoDB    │                              │   MongoDB    │
│  (source)    │                              │   (sink)     │
└──────┬───────┘                              └──────▲───────┘
       │ change stream                                │
       ▼                                              │
┌──────────────────────┐                  ┌──────────────────────┐
│ Mongo Source         │                  │ Mongo Sink           │
│ Connector            │                  │ Connector            │
└──────┬───────────────┘                  └──────▲───────────────┘
       │                                          │
       ▼                                          │
              ┌────────────────────────────┐
              │       Kafka Cluster        │
              │  topics: orders, users…    │
              └────────────────────────────┘
                            ▲
                            │
┌──────────────────────┐    │
│ Debezium Postgres    │    │
│ Source Connector     │────┘
└──────────────────────┘
       ▲
       │ WAL
┌──────┴───────┐
│  Postgres    │
└──────────────┘

Three connectors, two directions, one Kafka cluster.

Three connectors, two directions, one Kafka cluster: MongoDB to MongoDB and Postgres to MongoDB with Debezium

Chapter 3 — MongoDB Kafka Source Connector

Reads from MongoDB change streams, publishes to Kafka topics.

Install: download the Confluent Hub connector or pull the JAR; place in Connect plugin path.

Config:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
{
  "name": "mongo-source-orders",
  "config": {
    "connector.class":      "com.mongodb.kafka.connect.MongoSourceConnector",
    "connection.uri":       "mongodb+srv://user:pw@host/?retryWrites=true",
    "database":             "shop",
    "collection":           "orders",
    "topic.prefix":         "shop",
    "publish.full.document.only": "true",
    "change.stream.full.document": "updateLookup",
    "copy.existing":        "true",
    "errors.tolerance":     "all",
    "errors.log.enable":    "true"
  }
}

Topic name produced: shop.shop.orders (<prefix>.<database>.<collection>).

Key options:

  • publish.full.document.only: true → publish only the post-image document, not the full change event envelope. Often what consumers actually want.
  • change.stream.full.document: updateLookup → fetch the post-image on update events (Post 12 Ch 4).
  • copy.existing: true → on first start, snapshot the collection into Kafka before starting the change stream.
  • pipeline → an aggregation pipeline filter applied to events.

Apply via Connect REST API:

1
2
curl -X POST http://localhost:8083/connectors -H "Content-Type: application/json" -d @source.json

Chapter 4 — MongoDB Kafka Sink Connector

Reads from Kafka topics, writes to a Mongo collection.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
{
  "name": "mongo-sink-orders",
  "config": {
    "connector.class":   "com.mongodb.kafka.connect.MongoSinkConnector",
    "connection.uri":    "mongodb+srv://user:pw@host/",
    "database":          "shop",
    "collection":        "orders_view",
    "topics":            "shop.shop.orders",
    "document.id.strategy": "com.mongodb.kafka.connect.sink.processor.id.strategy.PartialValueStrategy",
    "document.id.strategy.partial.value.projection.list": "_id",
    "writemodel.strategy": "com.mongodb.kafka.connect.sink.writemodel.strategy.ReplaceOneBusinessKeyStrategy",
    "errors.tolerance":  "all",
    "errors.deadletterqueue.topic.name": "shop.orders.dlq"
  }
}

Key options:

  • document.id.strategy — how to derive _id on the sink doc. Use a strategy that keys on a stable business ID (often _id from the source).
  • writemodel.strategy — how to apply each Kafka message. ReplaceOneBusinessKeyStrategy is upsert-by-key — idempotent, replay-safe.
  • errors.tolerance: all + DLQ topic — bad messages don't stop the connector.

This pattern (Source → Kafka → Sink) is how you build a cross-region Mongo replica via Kafka, populate a read-model collection, or sync to another store.

Chapter 5 — Debezium MongoDB Source Connector

Debezium offers its own MongoDB source connector — equivalent to MongoDB's official Source but with the Debezium event envelope (consistent across all Debezium connectors). Pick one or the other; both work.

Default envelope:

1
2
3
4
5
6
7
8
9
10
11
12
{
  "schema": { ... },
  "payload": {
    "op": "c",                               // c=create, u=update, d=delete, r=read snapshot
    "before": null,
    "after":  { "_id": "o1", ... },
    "source": { "ts_ms": ..., "db": "shop", "collection": "orders", ... },
    "ts_ms":  ...,
    "transaction": null
  }
}

Useful when downstream consumers want before/after + a uniform shape across Postgres + Mongo + MySQL events.

Choose MongoDB's official Source connector if you only need one direction + want simpler messages. Choose Debezium if you're standardising on its envelope.

Chapter 6 — Debezium Postgres → MongoDB Sink

The killer use case: continuously replicate Postgres tables to MongoDB without writing code.

Debezium Postgres source

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
{
  "name": "pg-source",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "pg.example.com",
    "database.dbname":   "shop",
    "database.user":     "debezium",
    "database.password": "...",
    "topic.prefix":      "pg.shop",
    "plugin.name":       "pgoutput",
    "publication.autocreate.mode": "filtered",
    "table.include.list": "public.orders,public.users",
    "decimal.handling.mode": "string"
  }
}

Publishes one topic per table — pg.shop.public.orders, etc.

MongoDB Sink for Debezium events

1
2
3
4
5
6
7
8
9
10
11
12
{
  "name": "debezium-mongo-sink",
  "config": {
    "connector.class": "io.debezium.connector.mongodb.MongoDbSinkConnector",
    "topics": "pg.shop.public.orders",
    "mongodb.connection.string": "mongodb+srv://user:pw@host/",
    "sink.database":   "shop_synced",
    "sink.collection": "orders",
    "delete.tombstone.handling.mode": "delete"
  }
}

Debezium ships a dedicated MongoDB sink connector that understands its own event envelope + handles op: 'd' deletes correctly. Recent versions support the standard Debezium connector pattern (every event type → matching mongo op).

Watch out: the older MongoDB Kafka Sink (Chapter 4) doesn't natively handle Debezium update events from other DBs — you need a CDC handler (ChangeStreamHandler) configured, and there have been update-event bugs (see references). For new Postgres→Mongo work in 2026, prefer the Debezium MongoDB sink — it's purpose-built.

Chapter 7 — Kafka topics + partitioning

For Mongo source events:

  • One topic per collection (default) — easy.
  • Topic key = document _id → events for the same doc land in the same partition → preserved order per doc.

For Debezium Postgres source:

  • One topic per table.
  • Key = primary key → ordered per row.

Partition count = expected throughput. Default 6-12 partitions per topic for medium throughput. Don't go to 100 unless you have parallel consumers to match.

Retention: 7 days default. For replay scenarios, 30 days. For Event Sourcing (Post 21), forever (compact topic or off-cluster archive).

Chapter 8 — Schema Registry + Avro / JSON Schema

Without a registry, every consumer must parse JSON heuristically + handle schema changes manually.

With Confluent Schema Registry (or Apicurio, Karapace):

  • Producers register schemas.
  • Consumers fetch schema by ID embedded in each message.
  • Compatibility rules enforced (backward, forward, full).

Format choice: Avro (binary, compact, strong typing) for high-volume; JSON Schema (human-readable, debuggable) for moderate. Avro is the Kafka native; JSON Schema gaining ground.

Connectors support both. Configure once; downstream consumers get type safety + evolution rules for free.

Chapter 9 — Deduplication + ordering — same rules as outbox

Kafka delivers at-least-once (default). Consumers must be idempotent.

Patterns:

  • Key by business ID → idempotent upserts at the sink.
  • Optimistic versioning — sink rejects events with version <= current.
  • Tombstone records for deletes (key = ID, value = null) — compaction-friendly.

Ordering:

  • Within a partition: guaranteed.
  • Across partitions: not guaranteed.
  • Key your events so all events for one entity land in one partition.

Chapter 10 — Local dev environment

docker-compose.yml:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.6.0
    environment: { ZOOKEEPER_CLIENT_PORT: 2181 }
 
  kafka:
    image: confluentinc/cp-kafka:7.6.0
    depends_on: [zookeeper]
    ports: ["9092:9092"]
    environment:
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
 
  connect:
    image: confluentinc/cp-kafka-connect:7.6.0
    depends_on: [kafka]
    ports: ["8083:8083"]
    environment:
      CONNECT_BOOTSTRAP_SERVERS: kafka:9092
      CONNECT_GROUP_ID: dev
      CONNECT_CONFIG_STORAGE_TOPIC: connect.config
      CONNECT_OFFSET_STORAGE_TOPIC: connect.offset
      CONNECT_STATUS_STORAGE_TOPIC: connect.status
      CONNECT_KEY_CONVERTER: org.apache.kafka.connect.json.JsonConverter
      CONNECT_VALUE_CONVERTER: org.apache.kafka.connect.json.JsonConverter
      CONNECT_PLUGIN_PATH: /usr/share/java,/usr/share/confluent-hub-components
    volumes:
      - ./plugins:/usr/share/confluent-hub-components
 
  mongo:
    image: mongo:8
    command: ["--replSet", "rs0"]
    ports: ["27017:27017"]
 
  postgres:
    image: postgres:16
    environment: { POSTGRES_USER: ..., POSTGRES_PASSWORD: ..., POSTGRES_DB: shop }
    ports: ["5432:5432"]
    command: ["postgres", "-c", "wal_level=logical"]

./plugins holds the downloaded connector JARs (Mongo source/sink + Debezium Postgres). confluent-hub install is the friendly path.

After docker compose up, hit http://localhost:8083 for Connect REST. POST connector configs as in Chapters 3-6.

Chapter 11 — Production operations

  • Connect cluster in distributed mode (multiple Connect workers) for HA + scalability.
  • Monitoring: JMX → Prometheus exporter → Grafana. Watch connector lag + DLQ size.
  • Backups: Connect stores state in Kafka topics — back them up.
  • Schema changes: register new schema versions; downstream consumers handle gracefully.
  • Failover: source connector resumes from stored offset on restart.
  • Atlas: hosted Confluent Cloud + Atlas Source/Sink work seamlessly. Many teams pick this combo for zero ops.

Chapter 12 — Comparison table

Outbox + change-stream (Post 20) Mongo Source/Sink Debezium + Sink
Setup complexity minimal medium (Kafka + Connect) medium-high (Kafka + Connect + Debezium)
Cross-DB CDC no no yes (Postgres / MySQL / etc)
Replay window as long as you keep outbox rows Kafka retention (7-90 days) Kafka retention
Event envelope yours Mongo's Debezium's (consistent across DBs)
Multi-consumer fan-out possible but custom natural via Kafka natural via Kafka
Cost tiny Kafka + Connect Kafka + Connect

Pick the cheapest pattern that fits the requirement.

Chapter 13 — Common pitfalls

Pitfall Cause Fix
Mongo Sink fails on update from Debezium CDC handler config missing use Debezium MongoDB sink instead, or configure ChangeStreamHandler
Update events arrive without fullDocument source change.stream.full.document not set set to updateLookup
Out-of-order events partition key not on business ID re-key
Connector lag grows sink slow scale sink concurrency; partition more
Schema change breaks consumers no Schema Registry adopt; enforce compatibility
DLQ topic ignored no alerts on DLQ size alert when DLQ count > 0
Connect single-node failure not running in distributed mode run ≥ 2 workers
Source replicates everything (huge load) no pipeline filter filter event stream at source
Tombstone confusion deletes received as null configure handler
Debezium PG sequence reset / re-init snapshot mode misconfigured review snapshot.mode

Practice Exercises

Exercise 1 — Local stack

Stand up the docker-compose.yml from Chapter 10. Confirm Kafka, Connect, Mongo (replica set), Postgres all run.

Exercise 2 — Mongo Source

Configure the Mongo Source connector for a shop.orders collection. Insert into Mongo; consume the shop.shop.orders topic with kafka-console-consumer.

Exercise 3 — Mongo Sink

Add a Mongo Sink consuming the topic into shop_view.orders. Insert into source; verify it appears in the sink within seconds.

Exercise 4 — Debezium Postgres → Mongo

Configure Debezium Postgres source + MongoDB sink. Insert into a Postgres table; verify the document appears in Mongo.

Exercise 5 — Filter at source

Add a pipeline filter to the Mongo Source connector — only events with status: "paid". Confirm filter works.

Exercise 6 — DLQ

Misconfigure a sink (wrong field type). Send a bad message; confirm it lands in the DLQ topic instead of stopping the connector.

Cheat-sheet

What How
Mongo → Kafka MongoDB Kafka Source connector (official)
Kafka → Mongo MongoDB Kafka Sink connector
Postgres → Mongo Debezium PG source + Debezium MongoDB sink
Mongo → other DB MongoDB Source + that DB's sink
Full doc on update change.stream.full.document: updateLookup
Publish post-image only publish.full.document.only: true
Snapshot on first run copy.existing: true
Idempotent sink ReplaceOneBusinessKeyStrategy on _id
Error policy errors.tolerance: all + DLQ topic
Schema use Schema Registry (Avro / JSON Schema)
Order per entity key on business ID → same partition
Local dev docker-compose with Kafka + Connect + Mongo + PG
Production distributed Connect cluster + monitoring + alerts
Cheaper alt for single consumer outbox + change-stream publisher (Post 20)

What's next

Post 25 covers testing strategies — Testcontainers + in-memory MongoDB + fixtures + Jest/Vitest/pytest integration patterns.

References

Note: this post is in progress. Planned additions: rolling schema migration with Schema Registry; topic compaction tradeoffs; Kafka Streams for in-flight enrichment.