A single mongod is a single point of failure. A replica set is a group of mongod processes that maintain copies of the same data and automatically elect a new primary when one fails. Three or more members get you durability, automatic failover, optional read scaling on secondaries, and zero-downtime maintenance. This post covers the architecture, the oplog, election protocol, write-concern interaction with replication, secondary read trade-offs, and the canonical 3-member + arbiter topologies.

Series map: 00 Foundations → … → 09 Transactions → 10 Replica Sets (this post) → 11 Sharding → 12 Change Streams → … → 17 Drivers & Case Studies.

Learning Outcome

  • Explain PRIMARY / SECONDARY / ARBITER roles.
  • Initiate a replica set and inspect rs.status().
  • Read the oplog and understand replication lag.
  • Force an election + observe automatic failover.
  • Configure priorities, hidden members, delayed members.
  • Pick the right topology for your durability + cost target.

Chapter 1 — Why replication

Three things at once:

  1. Durability — N copies of the data; loss of one server doesn't lose data.
  2. High availability — automatic failover; clients reconnect to a new primary in seconds.
  3. Read scaling (optional) — secondaries serve eventual-consistent reads.

Replication doesn't scale writes — every write still goes through one primary. For write scaling, you shard (Post 11) on top of replica sets.

Chapter 2 — Member roles

Role Receives writes Serves reads Votes in election Holds data
PRIMARY yes (only one at a time) yes yes yes
SECONDARY replicates from primary's oplog yes (with secondaryPreferred etc.) yes yes
ARBITER no no yes no — voting only
HIDDEN no no (invisible to drivers) yes yes (backups, analytics)
DELAYED no no yes yes (lags primary by N seconds — disaster recovery)

A typical minimal production set: 3 data-bearing members (1 primary + 2 secondaries). Loss of any one keeps a majority alive.

Avoid:

  • Even number of voting members without an arbiter → election can split. Always odd.
  • Arbiter-only failover safety in 2026 → arbiters can't vote on majority writes; prefer a 3-data-member setup unless cost-constrained. (See "election + write concern" pitfall in Chapter 8.)

Chapter 3 — Initiate a replica set (local)

1
2
3
4
5
6
# start three mongod instances on different ports
mkdir -p data/rs0-{0,1,2}
mongod --replSet rs0 --port 27017 --dbpath data/rs0-0 --bind_ip 127.0.0.1 --fork --logpath data/rs0-0/log
mongod --replSet rs0 --port 27018 --dbpath data/rs0-1 --bind_ip 127.0.0.1 --fork --logpath data/rs0-1/log
mongod --replSet rs0 --port 27019 --dbpath data/rs0-2 --bind_ip 127.0.0.1 --fork --logpath data/rs0-2/log

Connect to one + initiate:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
mongosh --port 27017
 
rs.initiate({
  _id: "rs0",
  members: [
    { _id: 0, host: "127.0.0.1:27017" },
    { _id: 1, host: "127.0.0.1:27018" },
    { _id: 2, host: "127.0.0.1:27019" }
  ]
})
 
rs.status()
rs.conf()

Within a second or two, one member is PRIMARY, the other two are SECONDARY.

In Atlas, every cluster is already a replica set — you don't initiate manually.

Chapter 4 — Connection string for a replica set

1
2
3
mongodb://host1:27017,host2:27017,host3:27017/?replicaSet=rs0
mongodb+srv://cluster.abcd.mongodb.net/?retryWrites=true     # SRV form, Atlas

The driver:

  • Reads the seed list (your hosts).
  • Polls each to discover the full topology and the current primary.
  • Re-discovers automatically on failover.

You only need to list a few seed hosts — the driver finds the rest. The SRV form (Atlas) returns the full member list via DNS, so the URI stays short.

Chapter 5 — The oplog

The oplog is a capped collection (local.oplog.rs) on every replica-set member that records every write applied. Secondaries tail it and replay it to stay in sync.

1
2
3
use local
db.oplog.rs.find().sort({ ts: -1 }).limit(5)

Each entry: timestamp, operation type (insert/update/delete), namespace, document or update spec.

Oplog size matters

Default size: 5 % of free disk (with a 50 GB upper cap on most platforms). If a secondary falls behind by more oplog entries than fit, it goes STALE and must do a full resync — slow.

Tune via --oplogSizeMB at start, or replSetResizeOplog admin command. Atlas auto-sizes.

Trick: if you regularly run multi-hour ingest jobs while secondaries catch up, increase oplog size. The "1 day of writes fits" rule of thumb prevents most stale-member surprises.

Chapter 6 — Replication mechanics + lag

A write flowing through the primary's oplog to two secondaries, and an election when the primary stops answering

  1. Client writes to PRIMARY.
  2. PRIMARY records the write to its own oplog and applies it.
  3. Each SECONDARY tails the primary's oplog and applies in order.
  4. Once a majority of voters have applied the write, it's "majority committed".

Replication lag = how far behind the primary a secondary is in oplog time.

Inspect:

1
2
3
rs.printSecondaryReplicationInfo()
// returns each secondary's lag in seconds

Causes of lag:

  • Big write spike (secondaries can't keep up).
  • Slow disk on a secondary.
  • Network latency (especially across regions).
  • Long oplog entries (huge updates).

Persistent lag > a few seconds = warning. Lag growing unbounded = crisis (eventually goes STALE).

Chapter 7 — Elections

If the PRIMARY becomes unreachable, voting members hold an election (under the hood: Raft-like protocol Version 1, "PV1").

Steps:

  1. A secondary that hasn't heard from the primary for electionTimeoutMillis (default 10 s) calls an election.
  2. Other voting members vote yes if the candidate's oplog is up-to-date enough.
  3. Majority of votes → new PRIMARY.
  4. Drivers detect the change via heartbeats; subsequent writes go to the new primary.

Typical failover: 10-20 seconds for clients to reconnect; retryable writes (Post 08) hide most of the disruption.

Force a manual failover (test)

1
2
rs.stepDown(60)            // current primary steps down for 60 s

A useful production drill — practise failover under controlled conditions.

Chapter 8 — Write concern + replica sets

Recap from Post 08 — write concern interacts with replication:

w Meaning
0 fire-and-forget
1 primary acked
"majority" majority of voters acked
N exactly N members acked

"majority" is the production-correct default — guarantees the write survives the primary failing (because at least one surviving voter has it).

Arbiter pitfall

If you have a 2-data + 1-arbiter set (total 3 voters, majority = 2), arbiters can vote in elections but don't store data. A write reaching only the primary + the arbiter (waiting for the second data member) can be lost in failover. Modern guidance: avoid arbiters in 2026 for any system where data loss isn't acceptable. Use 3 data-bearing members.

Chapter 9 — Read preference + secondary reads

Recap from Post 08:

Preference Effect
primary (default) always primary, freshest
primaryPreferred primary if up, else secondary
secondary secondary only
secondaryPreferred secondary first, fall back to primary
nearest lowest-latency member

Use secondary reads for:

  • Reporting / dashboards tolerating seconds of lag.
  • Scaling read-heavy workloads horizontally.
  • Region-local reads via tagSets.

Don't use for:

  • Read-your-own-writes (RYOW) without causal-consistency session.
  • Anything where stale-by-seconds matters (orders, balances, leaderboards).

Chapter 10 — Priorities, hidden, delayed members

Each member has a configuration in rs.conf():

1
2
3
4
5
6
const cfg = rs.conf();
cfg.members[1].priority = 0;     // never become primary
cfg.members[1].hidden = true;     // invisible to drivers (analytics replica)
cfg.members[1].slaveDelay = 3600; // older Mongo; modern uses secondaryDelaySecs
rs.reconfig(cfg);

Common variations:

  • Analytics replica — hidden, priority 0 → drivers never route reads here; you connect directly for heavy analytics queries.
  • Backup target — hidden, priority 0 → take snapshots without affecting production reads.
  • Delayed member — secondaryDelaySecs: 3600 → lags primary by 1 hour. Disaster recovery against fat-finger drops ("oops, restore from the delayed member's state of an hour ago").

Topology recipe — 5-member production

1
2
3
4
5
6
m1: data, primary candidate (priority 1)
m2: data, primary candidate (priority 1) — different AZ
m3: data, primary candidate (priority 1) — third AZ
m4: data, hidden, priority 0  — analytics replica
m5: data, hidden, priority 0, secondaryDelaySecs: 3600 — DR replica

5 voters (majority 3). Tolerates 2 simultaneous failures of voting members.

Chapter 11 — Sizing + topology choices

Topology Voters Majority Tolerated failures When
3 data 3 2 1 minimum production
2 data + 1 arbiter 3 2 1 cheap; avoid in 2026
5 data 5 3 2 larger production, mixed roles
7 data 7 4 3 very high availability; rare
3 data, multi-region 3 2 1 (region) DR; read latency increases; needs tagSets

Voters max is 7 in a replica set (additional members can be non-voting).

Multi-region replica set: one primary serves global writes, secondaries in other regions serve local reads. Trade-off: writes to a "far" primary have high latency. Sharding by region (Post 11) often beats multi-region replica sets for writes.

Chapter 12 — Operational toolkit

1
2
3
4
5
6
7
8
9
rs.status()                              // overall set health, who's primary
rs.conf()                                 // current configuration
rs.reconfig(newCfg, { force: true })      // change configuration
rs.printSecondaryReplicationInfo()        // lag per secondary
rs.printReplicationInfo()                 // oplog stats
rs.stepDown(seconds, electionTimeoutMs)   // force this primary down
rs.freeze(seconds)                        // prevent this member from being elected
rs.syncFrom(host:port)                    // force secondary to sync from a specific source

For Atlas, the UI surfaces the equivalents; you rarely run these by hand.

Atlas dashboards to watch

  • Replication Lag chart.
  • Oplog Window (how many hours of oplog you have).
  • Connections / IOPS / CPU per member.
  • "Failover" event log.

Alert on: lag > 60 s, oplog window < 4 h, any voting member down.

Chapter 13 — Rolling maintenance pattern

To upgrade or restart members without downtime:

  1. Pick a secondary. Drain its connections.
  2. Stop + upgrade + restart it. Wait for SECONDARY state.
  3. Repeat for the next secondary.
  4. Step down the primary (rs.stepDown()); it becomes secondary; another member becomes primary.
  5. Stop + upgrade + restart the old primary. Wait for SECONDARY.

Zero downtime as long as a majority is always up. This is the "rolling upgrade" pattern for any major version bump or OS patch.

Atlas does this automatically when you click "Upgrade".

Chapter 14 — Common pitfalls

Symptom Cause Fix
Failover takes 30+ seconds electionTimeoutMillis too high; or driver serverSelectionTimeoutMS too high lower both modestly; test
Secondaries falling behind undersized oplog or slow disk on secondary enlarge oplog; faster disk
Stale secondary lag exceeded oplog window full resync; or enlarge oplog
Data lost after failover used w: 1 instead of w: "majority" switch to majority
Even number of voters possible split-brain add an arbiter or another data member
Arbiter loses data (2+1 set) known antipattern use 3 data members
Driver doesn't see new primary replicaSet= missing or wrong; DNS slow check connection string; SRV form preferred
Cross-region writes slow primary is far from app route writes to local primary or shard by region
Secondary read returns stale doc replication lag use causal consistency session or primary

Practice Exercises

Exercise 1 — Three-member local set

Initiate the 3-node set from Chapter 3. Insert data. Confirm replication: read on secondary (allow db.getMongo().setReadPref('secondaryPreferred')).

Exercise 2 — Forced failover

Run rs.stepDown(30). Watch a Node script that does inserts every second — note the brief failure + resume (retryable writes save the day).

Exercise 3 — Oplog inspection

use local; db.oplog.rs.find().sort({ ts: -1 }).limit(10). Identify your last few writes.

Exercise 4 — Replication lag

In one window, run a tight insert loop. In another, stop one secondary (mongod SIGSTOP / pause). Restart it after a minute. Watch rs.printSecondaryReplicationInfo() show the catch-up.

Exercise 5 — Hidden analytics replica

Set member 2 to priority: 0, hidden: true. Confirm drivers no longer route reads there. Connect directly to that member for heavy aggregations.

Exercise 6 — Delayed member

Configure member 2 with secondaryDelaySecs: 60. Insert a doc. Immediately read from member 2 (direct connection) — should not see it. Wait 90 s, read again — should see it.

Cheat-sheet

What How
Initiate rs.initiate({ _id, members })
Status rs.status()
Config rs.conf()
Reconfig rs.reconfig(newCfg)
Lag rs.printSecondaryReplicationInfo()
Oplog stats rs.printReplicationInfo()
Step down rs.stepDown(60)
Connection string mongodb://h1,h2,h3/?replicaSet=name (or SRV for Atlas)
Production defaults 3 data members, w: majority, retryable writes/reads
Avoid arbiters in serious workloads (2026)
Reads scale secondaryPreferred for stale-OK; causal consistency for RYOW
Hidden analytics / backup target
Delayed DR against accidental drops (secondaryDelaySecs)
Failover time ~10-20 s typical
Election protocol PV1 (Raft-like)

What's next

Post 11 covers sharding — mongos router, config servers, shard keys (cardinality, frequency, monotonic), ranged vs hashed sharding, chunks + balancer, zone sharding, resharding (5.0+), and query targeting vs scatter-gather.

References

Note: this post is in progress. Planned additions: chained replication trade-offs, multi-region latency tables, ELECTING_PRIMARY state transitions visualised.