Networking is the part of computing that stays true the longest. Frameworks are replaced every few years; the fact that a packet is a bounded chunk of bytes travelling between two addresses, that it may be dropped, duplicated or delayed, and that everything above it is an attempt to hide those facts, has held since the 1970s and will outlive whatever runtime you are using this year. This post is a single self-contained pass over data communication and networking: the switching reality at the bottom, the layered abstractions built over it, the addressing and subnetting arithmetic you have to be able to do by hand, the transport and application protocols that carry the modern web, the delivery machinery that makes it fast at scale, and the diagnostic ladder that turns "the site is slow" into a specific, provable fault. Security is not a separate chapter here — firewalls, TLS and the common attack classes appear at the layer where each actually lives.

Where a cloud provider gives these concepts a branded name — a VPC, a security group, a NAT gateway — the mapping is pointed out, because a VPC is not a new idea: it is subnetting, routing tables and stateful firewalls with a control plane bolted on. The companion post AWS Course — 01 · Network First is the applied version of Chapters 3 and 4.

Learning Outcome

Explain what a network is in switching terms, distinguish bandwidth, throughput, latency and jitter, and use the bandwidth-delay product to explain why it caps a single connection.
Map OSI's seven layers onto TCP/IP's four, walk one real packet through encapsulation header by header, and read an Ethernet frame — including what a switch learns and how ARP, VLANs and MTU interact.
Subnet IPv4 by hand: derive network address, broadcast, usable host range and host count from any prefix, and build a VLSM plan that does not overlap.
Explain NAT and its real costs, read an IPv6 address, describe SLAAC and dual stack, and defend why IPv6 adoption matters.
Describe how routing tables, longest-prefix match, OSPF and BGP move a packet across the internet — and how hijacks and route leaks break that.
Explain TCP's handshake, retransmission, flow control and congestion control, know when UDP is the better answer, and say what QUIC and TLS 1.3 fixed above it.
Trace one DNS name to its answer, compare HTTP/1.1, HTTP/2 and HTTP/3, and diagnose a broken or slow connection from layer 1 upward.

Chapter 1 — Fundamentals

Terminology

Network — nodes plus links, arranged so any node can deliver bytes to any other, directly or via intermediaries.
Node — anything terminating or forwarding traffic: host, switch, router, firewall, load balancer.
Protocol — an agreed message format and set of rules; the contract that lets independently written programs interoperate.
Bandwidth — capacity of a link in bits per second. A property of the link, not of your transfer.
Throughput — the rate your transfer actually achieves. Always ≤ bandwidth, usually well below. Goodput counts only useful application bytes, excluding headers and retransmissions.
Latency — one-way delay. RTT is the there-and-back figure, which is what most tools measure.
Jitter — variation in latency between successive packets. Fatal for real-time media, irrelevant for file transfer.
Packet loss — the fraction of packets that never arrive. Networks are allowed to drop; reliability is built above.

Circuit switching vs packet switching

	Circuit switching	Packet switching
Resource model	dedicated path held for the call's duration	statistical multiplexing, no reservation
Setup	required before data flows	none; the first packet just goes
Idle capacity	wasted — the circuit is held either way	reused by others
Delay	constant, predictable	variable; queues build under load
On failure	call drops, must be re-established	packets reroute around it
Example	analogue telephony, TDM leased lines	the internet, Ethernet, Wi-Fi

The internet chose packet switching because data traffic is bursty: a browser idles for seconds, then wants everything at once, and reserving capacity for an idle conversation is waste. The price is that delay is no longer constant, congestion is possible, and every layer above must cope with loss and reordering. Essentially all of Chapter 5 exists to pay that price.

Bandwidth is not speed

These four terms are routinely conflated, and the conflation causes most bad network decisions.

A 1 Gbit/s link does not make a packet arrive sooner; it makes more packets fit per second. Bandwidth is a truck's capacity, latency is how long the drive takes. And latency has a floor you cannot buy past: light in fibre travels at roughly two-thirds of c, about 200,000 km/s, so London to New York (~5,600 km) has a one-way minimum near 28 ms and an RTT floor near 56 ms. Real paths are not straight, so 70–80 ms is the production figure. No money reduces this; only moving the data closer does (Chapter 7). Throughput is the outcome, determined jointly by bandwidth, latency, loss, window sizes, the far end's capacity and every queue in between.

Watch out: "the network is slow" almost never means bandwidth is exhausted. Far more often it is latency, loss driving retransmission, an undersized window, a DNS stall, or a slow server. Measure before upgrading a link.

Bandwidth-delay product

The BDP is how much data fits "in flight" on a path: bandwidth (bits/s) × RTT (seconds). It matters because a reliable protocol can only have as much unacknowledged data outstanding as its window allows; if the window is smaller than the BDP, the sender stalls waiting for acknowledgements and the link runs empty.

1
2
3 100 Mbit/s, RTT 20 ms → 100,000,000 × 0.020 = 2,000,000 bits =    250,000 bytes ≈ 244 KiB
1 Gbit/s,   RTT 80 ms → 1,000,000,000 × 0.080 = 80,000,000 bits = 10,000,000 bytes ≈ 9.54 MiB

Now the classic consequence. TCP's receive-window field is 16 bits, so without window scaling the maximum window is 65,535 bytes. On that 80 ms transatlantic path:

1
2 65,535 bytes × 8 bits ÷ 0.080 s = 524,280 ÷ 0.080 ≈ 6,553,500 bits/s ≈ 6.55 Mbit/s

A 1 Gbit/s link delivering 6.5 Mbit/s per connection — a 150× shortfall — purely because of a header field sized in 1981. Window scaling (RFC 7323) is the fix and is default everywhere now, but the shape recurs constantly: on long paths the window, not the link, is the limit.

Trick: when one transfer is slow over a long path but many parallel transfers saturate the link, the bottleneck is per-connection windowing or congestion control, not capacity. That is exactly why aria2 -x, download accelerators and multipart S3 uploads exist.

Why layers

Each layer solves one problem using only the service beneath it and offers one service above. The payoff is substitutability: HTTP does not know whether it runs over Ethernet, Wi-Fi or LTE. Each layer adds a header going down and strips it coming up — encapsulation, the subject of Chapter 2.

The cost is worth naming: layering hides information the layer above sometimes needs. TCP cannot tell wireless loss from congestion loss and slows down for both. HTTP/1.1 could not see it was sharing one connection. Most protocol progress of the last decade — QUIC especially — is layer boundaries being renegotiated because the old ones hid too much.

Chapter 2 — OSI and TCP/IP

Terminology

PDU — the unit of data at a layer. Layer 2 calls it a frame, layer 3 a packet, TCP a segment, UDP a datagram.
Encapsulation — wrapping a higher layer's PDU as the payload of a lower one by prepending a header.
Trailer — metadata appended rather than prepended; Ethernet's frame check sequence is the example.
MTU — the largest payload one link carries in a single frame. MSS — the largest TCP payload that fits in one packet without fragmenting.

The two models

OSI defines seven layers; the model RFC 1122 actually describes defines four.

OSI layer	#	TCP/IP layer	Runs here
Application	7	Application	HTTP, DNS, SMTP, SSH, gRPC
Presentation	6	Application	TLS (approximately), encodings, serialization
Session	5	Application	TLS resumption, RPC session state
Transport	4	Transport	TCP, UDP, QUIC (approximately), SCTP
Network	3	Internet	IPv4, IPv6, ICMP, IPsec, routing
Data link	2	Link	Ethernet, Wi-Fi, PPP, ARP, VLAN tags
Physical	1	Link	copper, fibre, radio, line coding

One packet, encapsulated

A browser sending an HTTPS request over IPv4 and Ethernet, using documentation addresses from RFC 5737:

1
2
3
4
5
6
7
8
9
10
11
12 [ GET / HTTP/1.1 ... ]                              application data
   ↓  TLS record layer wraps and encrypts
[ TLS hdr (5 B) | ciphertext + AEAD tag ]           TLS record
   ↓  TCP prepends its header
[ TCP hdr (20 B min) | TLS record ]                 TCP segment       src port 54321 → dst 443
   ↓  IPv4 prepends its header
[ IPv4 hdr (20 B min) | TCP segment ]               IP packet         192.0.2.15 → 203.0.113.80
   ↓  Ethernet frames it                                              protocol 6 = TCP
[ Eth hdr (14 B) | IP packet | FCS (4 B) ]          Ethernet frame    00:1a:2b:3c:4d:5e
   ↓  NIC adds preamble + SFD and transmits                           → 00:1a:2b:00:00:01 (gateway)
                                                                      EtherType 0x0800 = IPv4

Two things cause real bugs. The destination MAC is the gateway's, not the server's — layer 2 addressing is hop-by-hop, layer 3 is end-to-end. IP addresses stay the same across the path (absent NAT); MAC addresses are rewritten at every router. This is the single most useful mental model in networking.

Header sizes determine the MSS. On a 1500-byte MTU: 1500 − 20 (IPv4) − 20 (TCP) = 1460. With IPv6's fixed 40-byte header, 1500 − 40 − 20 = 1440. With the TCP timestamp option (common on Linux, 12 bytes padded), 1448. Over PPPoE at MTU 1492, 1452. Every VPN or overlay subtracts more.

OSI is a teaching tool; TCP/IP is what runs

OSI was a complete protocol suite that lost; what survived is its vocabulary. Seven layers are useful for talking ("that is a layer 7 load balancer") and misleading taken literally, because real protocols refuse to sit in one box. TLS runs over TCP, presents a byte stream, and does key exchange — it has no OSI home. QUIC is transport, session and security at once, in user space over UDP. MPLS is routinely called "layer 2.5" because it fits nowhere. VXLAN (RFC 7348) carries Ethernet inside UDP inside IP inside Ethernet, which the model forbids and every data centre does. ICMP rides inside IP but controls IP itself.

Use OSI numbers as shorthand; reason about the four TCP/IP layers when you need to be right.

Chapter 3 — Link layer

Terminology

Frame — the link layer's PDU: header, payload, frame check sequence.
MAC address — a 48-bit link-layer identifier burned into or configured on a NIC.
Broadcast domain — the set of interfaces a broadcast frame reaches. One VLAN is one broadcast domain.
Trunk port — a switch port carrying multiple VLANs, distinguished by 802.1Q tags; an access port carries one, untagged.

The Ethernet frame

IEEE 802.3, in ubiquitous Ethernet II framing: preamble + SFD (7 + 1 bytes, not part of the frame proper), destination MAC (6), source MAC (6), EtherType (2 — 0x0800 IPv4, 0x86DD IPv6, 0x0806 ARP, 0x8100 802.1Q), payload (46–1500), FCS (4, CRC-32; a failing frame is silently discarded), then a 12-byte interframe gap. The 46-byte payload minimum makes the smallest legal frame 64 bytes. The 1500-byte maximum is the origin of the MTU number everyone quotes.

MAC addresses

Six hex octets. The first 24 bits are the OUI, assigned to a vendor by the IEEE. Two flag bits live in octet 1: the I/G bit (least significant) is 0 for unicast, 1 for multicast, with ff:ff:ff:ff:ff:ff as broadcast; the U/L bit (second least significant) is 1 for locally administered, which every virtual NIC, container veth and randomised phone MAC sets.

Watch out: modern phones and laptops randomise their MAC per network by default as an anti-tracking measure. Any access control, DHCP reservation or licence check keyed on MAC address is now unreliable by design. Use 802.1X or certificates.

What a switch actually does

One table, three rules. A frame arrives on port 3 with source MAC aa:bb:cc:00:00:01, so the switch learns that MAC is reachable via port 3 (entries age out, typically after 300 s). It looks up the destination MAC: found means forward out that one port, not found means flood out every port except the arrival port. Broadcast and multicast are flooded within the VLAN.

Two consequences. Unknown-unicast flooding is normal, briefly — a quiet server whose entry aged out receives VLAN-wide flooded traffic until it replies once. And loops are catastrophic, because Ethernet has no TTL: a frame in a physical loop circulates forever and broadcasts multiply at every switch, saturating the segment in seconds. Spanning Tree (802.1D, and the faster RSTP 802.1w and MSTP 802.1s) exists solely to block redundant links so the active topology is loop-free.

ARP

To send an IP packet on the local link, a host needs the destination's MAC. ARP (RFC 826) supplies it:

1
2
3
4 → broadcast (dest ff:ff:ff:ff:ff:ff):  "Who has 192.0.2.1? Tell 192.0.2.15"
← unicast:                             "192.0.2.1 is at 00:1a:2b:00:00:01"
  cached in the neighbour table for a few minutes

1
2
3
4 ip neigh show                 # Linux neighbour/ARP table
arp -a                        # macOS / BSD / Windows
ip neigh flush dev eth0       # force re-resolution

ARP has no authentication whatsoever: any host on the segment can answer for any IP. That is ARP spoofing, and it turns a shared LAN into an on-path attack surface — the attacker becomes the gateway and sees all victim traffic. Mitigations are switch features, not host features: Dynamic ARP Inspection, DHCP snooping, port security, and 802.1X to authenticate devices before they get a port at all. IPv6 replaces ARP with Neighbor Discovery (RFC 4861) over ICMPv6, which has the same trust problem and the same class of switch-side defences (RA Guard, ND inspection).

VLANs

A VLAN partitions one physical fabric into independent broadcast domains. IEEE 802.1Q inserts a 4-byte tag after the source MAC: TPID (16 bits, 0x8100), PCP (3 bits, 802.1p priority), DEI (1 bit), and VID (12 bits, VLAN 1–4094; 0 and 4095 reserved). Traffic between VLANs must be routed — which is exactly where firewall policy gets applied, and why "put it in its own VLAN" is a security statement, not just a topology one.

Common pitfall: the native VLAN on a trunk is carried untagged. If user traffic uses the native VLAN, a double-tagged frame can hop VLANs. Set the native VLAN to an unused ID and tag everything that matters.

The 12-bit VID caps a fabric at 4094 segments, which is why multi-tenant data centres moved to VXLAN and its 24-bit VNI — roughly 16 million segments, carried as Ethernet-in-UDP so it routes across an IP fabric.

MTU, and why 1500

1500 is the Ethernet payload maximum, and because Ethernet is everywhere it became the default assumption for the whole internet. Jumbo frames (typically 9000) cut per-packet overhead but require every device on the path to agree, so they are a data-centre feature, never an internet one. Tunnels shrink the usable MTU: PPPoE takes 8 bytes, and IPsec, WireGuard, GRE and VXLAN each take their own bite. An endpoint that ignores this produces the classic symptom — pings fine, DNS fine, small pages fine, large transfers hang forever.

Path MTU Discovery (RFC 1191) is meant to solve it: the sender sets the IPv4 Don't Fragment bit and a router that cannot forward returns ICMP "Fragmentation Needed" (type 3, code 4) naming the smaller MTU. A firewall that blindly drops all ICMP turns the path into a PMTUD black hole. IPv6 raises the stakes — routers must never fragment (RFC 8200), so the source depends entirely on ICMPv6 Packet Too Big.

1
2
3
4
5 ping -M do -s 1472 203.0.113.80   # 1472 + 8 (ICMP) + 20 (IPv4) = 1500; bisect downward on failure
ip link show eth0 | grep mtu      # local interface MTU
# last-resort fix on a router: clamp TCP MSS to the real path MTU
iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

Trick: "small requests fine, large uploads hang" is an MTU problem until proven otherwise. It is the highest-yield guess in all of network troubleshooting.

Wi-Fi is not wireless Ethernet

IEEE 802.11 shares Ethernet's framing vocabulary and almost none of its behaviour.

	Wired Ethernet	Wi-Fi
Medium	dedicated per port	shared radio, half duplex
Access method	switched, effectively collision-free	CSMA/CA with per-frame link-layer ACKs
Loss cause	rare; congestion or a bad cable	routine; interference, distance, contention
Rate	negotiated once	adapts per frame to signal quality
Joining	plug it in	authenticate, associate, encrypt (WPA2-CCMP or WPA3-SAE)
Extra problem	none	hidden nodes; RTS/CTS mitigates

Two consequences. Wi-Fi loss is not congestion, but TCP cannot tell and slows down anyway — one reason wireless throughput collapses unhelpfully at the edge of coverage. And airtime is the real resource: a distant client transmitting at a low rate occupies the channel far longer per byte and degrades everyone. When Wi-Fi is "slow", look at signal, channel occupancy and client count before blaming the uplink.

Firewalling starts here too. A stateless filter judges each packet alone; a stateful filter tracks connections so return traffic for an allowed outbound flow is permitted automatically. That is exactly the AWS security group (stateful, per-interface) versus network ACL (stateless, per-subnet) split in AWS Course — 01 · Network First.

Chapter 4 — Network layer

Terminology

Prefix / CIDR block — a range of addresses sharing a leading bit pattern, written address/length. The subnet mask is the same idea as a dotted quad.
Network address — all-zeros host portion; names the subnet. Broadcast address — all-ones host portion (IPv4 only).
Default gateway — the next hop for any destination not on a directly connected subnet.
Autonomous System (AS) — a network under one routing policy, identified by an AS number.

IPv4 addressing and CIDR

Thirty-two bits in four decimal octets; the prefix length says how many leading bits identify the network.

1
2
3 192.168.10.77/26 → mask 255.255.255.192 = 11111111.11111111.11111111.11000000
                 → 26 network bits, 6 host bits

Class A/B/C is dead; CIDR (RFC 4632) replaced it with arbitrary prefix lengths, which is what made aggregation and efficient allocation possible. Usable hosts is 2^(32 − prefix) − 2, subtracting network and broadcast — except /31, which RFC 3021 permits on point-to-point links with both addresses usable, and /32, a single host route.

Prefix	/30	/29	/28	/27	/26	/25	/24	/23	/22	/21	/20	/16
Addresses	4	8	16	32	64	128	256	512	1,024	2,048	4,096	65,536
Usable	2	6	14	30	62	126	254	510	1,022	2,046	4,094	65,534
Last mask octet	252	248	240	224	192	128	0	—	—	—	—	—

Subnetting, step by step

Convert the prefix to a mask and find the interesting octet — the one that is neither 255 nor 0.
Compute the block size: 256 − (mask value in that octet).
Boundaries in that octet are multiples of the block size, starting at 0.
Network address = the boundary at or below the given address; octets to its right become 0.
Broadcast = next boundary − 1; octets to its right become 255.
Usable range = network + 1 through broadcast − 1. Hosts = 2^(32 − prefix) − 2.

Worked example 1 — `192.168.10.77/26`

1
2
3
4
5
6
7 mask 255.255.255.192, interesting octet = 4th, block = 256 − 192 = 64
boundaries: 0, 64, 128, 192   →  77 lies in 64…127
network   192.168.10.64
broadcast 192.168.10.127
usable    192.168.10.65 – 192.168.10.126
hosts     2^(32−26) − 2 = 64 − 2 = 62

Verify by ANDing: 77 is 01001101, mask octet 192 is 11000000, AND gives 01000000 = 64. Correct.

Worked example 2 — `10.13.200.45/21`

Here the interesting octet is the third, which is where most people slip.

1
2
3
4
5
6
7 mask 255.255.248.0, interesting octet = 3rd, block = 256 − 248 = 8
boundaries in octet 3: 0, 8, 16, …, 192, 200, 208   →  200 is itself a multiple of 8
network   10.13.200.0
broadcast 10.13.207.255      (next boundary 208, minus 1 → 207; octets right → 255)
usable    10.13.200.1 – 10.13.207.254
hosts     2^(32−21) − 2 = 2048 − 2 = 2046

The subnet spans exactly eight /24s, 10.13.200.0 through 10.13.207.0. When the interesting octet is the third, remember the fourth octet runs its full 0–255 range in every row.

Worked example 3 — VLSM allocation

Given 172.16.4.0/22 (1,024 addresses, 172.16.4.0–172.16.7.255), serve Engineering (500 hosts), Sales (200), Ops (100), Guest Wi-Fi (50) and two point-to-point router links. Round each up to a power of two, then allocate largest first so every block lands on its own natural boundary:

1
2
3
4
5
6
7
8 172.16.4.0/23    Engineering  510 usable   172.16.4.0   – 172.16.5.255
172.16.6.0/24    Sales        254 usable   172.16.6.0   – 172.16.6.255
172.16.7.0/25    Ops          126 usable   172.16.7.0   – 172.16.7.127
172.16.7.128/26  Guest         62 usable   172.16.7.128 – 172.16.7.191
172.16.7.192/30  Link A         2 usable   172.16.7.192 – 172.16.7.195
172.16.7.196/30  Link B         2 usable   172.16.7.196 – 172.16.7.199
                 free                      172.16.7.200 – 172.16.7.255  (56 spare)

Common pitfall: allocating smallest first produces blocks that are not aligned to their own size, which is illegal. A /23 must start on an even /24 boundary; a /26 must start at .0, .64, .128 or .192. If a block does not begin at a multiple of its size, the plan is wrong.

Summarisation is the reverse: 192.168.8.0/24 through 192.168.11.0/24 are contiguous and 8 is a multiple of 4, so they aggregate to 192.168.8.0/22. This is how the global routing table stays finite — providers announce aggregates, not the thousands of customer prefixes inside them.

Special and private ranges

Range	Purpose	Reference
`10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`	private	RFC 1918
`100.64.0.0/10`	carrier-grade NAT shared space	RFC 6598
`127.0.0.0/8`	loopback	RFC 1122
`169.254.0.0/16`	link-local / APIPA	RFC 3927
`224.0.0.0/4`	multicast	RFC 5771
`192.0.2.0/24`, `198.51.100.0/24`, `203.0.113.0/24`	documentation only	RFC 5737

The full special-purpose registry is RFC 6890. Note 172.16.0.0/12 in particular: it is 172.16.0.0–172.31.255.255, not everything beginning 172. — a classic firewall-rule bug.

NAT and what it costs

NAPT (also called PAT) lets many private hosts share one public IPv4 address; the router keeps a mapping table and rewrites addresses and ports both ways.

1
2
3 192.168.1.10:51000  ↔  203.0.113.7:61001  ──►  93.184.x.x:443
192.168.1.11:51000  ↔  203.0.113.7:61002  ──►  93.184.x.x:443

NAT bought the IPv4 internet two extra decades. The bill:

End-to-end addressing is gone. Inbound connections require port forwarding, UPnP/NAT-PMP, or a rendezvous server.
Peer-to-peer becomes hard. Two NATed hosts cannot connect directly without hole punching — hence STUN, TURN, ICE, and most of WebRTC's machinery.
Protocols that embed addresses break. Active FTP and SIP carry IPs in their payloads, so the NAT must deep-inspect and rewrite them ("ALGs") — fragile, and a recurring source of bizarre bugs.
State is finite and mortal. Idle mappings are reclaimed, often within minutes for UDP; long-lived idle TCP connections die silently unless keepalives are shorter than the NAT's timeout.
CGNAT destroys the IP as identity. Thousands of subscribers share one address, so rate limiting, geolocation, abuse blocking and audit logging all become coarse and unfair — and ports run out, producing failures that look like packet loss.

A NAT is not a firewall, though it accidentally behaves like one inbound. Say what you mean and configure a stateful firewall explicitly. In cloud terms, a NAT gateway is precisely this device, and the reason private-subnet instances reach the internet but cannot be reached from it.

IPv6

128 bits as eight groups of four hex digits, with two compression rules (canonical form is RFC 5952): drop leading zeros in a group, and replace the single longest run of all-zero groups with ::, once.

1
2 2001:0db8:0000:0000:0000:ff00:0042:8329  →  2001:db8::ff00:42:8329

Prefix	Meaning
`2000::/3`	global unicast — the routable internet
`fe80::/10`	link-local; every IPv6 interface has one, always
`fc00::/7` (in practice `fd00::/8`)	unique local addresses, RFC 4193 — the rough analogue of RFC 1918
`ff00::/8`	multicast (`ff02::1` all-nodes, `ff02::2` all-routers)
`::1/128`, `::/128`	loopback, unspecified
`2001:db8::/32`	documentation, RFC 3849

Structural differences that matter: no broadcast (multicast replaces it, removing a class of broadcast storms); the subnet is /64, effectively always, because SLAAC requires it — sites get a /48 or /56, giving 65,536 or 256 subnets, so IPv6 subnetting is about organising prefixes rather than conserving addresses; a fixed 40-byte header with options moved to extension headers, so routers do less work per packet; and routers never fragment (RFC 8200).

SLAAC

Stateless Address Autoconfiguration (RFC 4862) lets a host configure itself with no server:

1
2
3
4
5
6 1. Host generates a link-local address: fe80::/64 + interface identifier.
2. Duplicate Address Detection — a Neighbor Solicitation for its own address; silence means free.
3. Host sends a Router Solicitation to ff02::2.
4. Router Advertisement returns the /64 prefix, default route, lifetimes, and the M/O flags.
5. Global address = advertised /64 prefix + interface identifier.

The interface identifier was originally derived from the MAC via EUI-64, leaking a stable hardware identity onto every network you joined. Modern stacks default to temporary privacy addresses (RFC 8981) plus a stable-but-opaque per-network identifier (RFC 7217), so a host typically holds several IPv6 addresses at once. The RA's M flag means "also use DHCPv6 for addresses", O means "use DHCPv6 for DNS and other information"; DHCPv6 itself is RFC 8415.

Dual stack and why adoption matters

A dual-stack host prefers IPv6 when a AAAA resolves. Happy Eyeballs v2 (RFC 8305) makes that safe: start the IPv6 connection, give it a short head start, race IPv4 behind it — so a broken IPv6 path costs milliseconds rather than a timeout. On IPv6-only networks, NAT64 (RFC 6146) with DNS64 (RFC 6147) synthesises addresses for IPv4-only destinations and translates at the edge.

Adoption matters for economic reasons before technical ones. The IANA free pool exhausted in 2011 and addresses now trade on a secondary market at real cost. Networks without enough IPv4 deploy CGNAT, which is expensive to run, degrades the user experience, and destroys the address as a unit of accountability. Large mobile operators run IPv6-only cores with 464XLAT because it is simply cheaper. Meanwhile IPv6 restores the end-to-end model, quietly removing whole categories of workaround. The pragmatic position is not "IPv6 or IPv4" — it is dual stack now, IPv6-primary as soon as dependencies allow, and never designing anything new that assumes 32-bit addresses.

Routing

A routing table maps destination prefixes to next hops.

1
2
3
4
5
6 ip route show
# default via 192.0.2.1 dev eth0 proto dhcp metric 100
# 192.0.2.0/24 dev eth0 proto kernel scope link src 192.0.2.15
# 10.0.0.0/8 via 192.0.2.254 dev eth0
ip route get 203.0.113.80     # ask the kernel which entry actually wins

Longest-prefix match

When several entries match, the most specific wins — longest prefix, regardless of table order. Given 0.0.0.0/0 → 203.0.113.1, 10.0.0.0/8 → 10.1.0.1, 10.1.0.0/16 → 10.1.0.2 and 10.1.5.0/24 → 10.1.5.1:

Destination	Matches	Winner	Next hop
`10.1.5.9`	/0, /8, /16, /24	/24	`10.1.5.1`
`10.1.9.9`	/0, /8, /16	/16	`10.1.0.2`
`10.9.9.9`	/0, /8	/8	`10.1.0.1`
`8.8.8.8`	/0	/0	`203.0.113.1`

This one rule explains most "why did my traffic go that way" mysteries, cloud ones included: a 0.0.0.0/0 entry pointing at a NAT gateway never beats a more specific entry pointing at a VPC endpoint or peering connection.

Static vs dynamic

Static routes are configured by a human or by IaC, do not react to failure, and scale to a handful of entries — right for stub networks, cloud route tables and default routes. Dynamic routing is a protocol between routers that reconverges around failures and scales to a million prefixes — right for anything with redundant paths.

Interior: OSPF

OSPF (OSPFv2 is RFC 2328; OSPFv3 carries IPv6) is a link-state IGP. Every router floods link-state advertisements describing its own links, so every router builds an identical map of the area and independently runs Dijkstra over it, with cost derived from bandwidth. Large deployments split into areas joined by a backbone (area 0) so flooding and SPF computation stay bounded. Convergence is fast — sub-second with tuned timers — because everyone sees the same topology. IS-IS occupies the same slot in large carriers; EIGRP is Cisco's distance-vector alternative.

Exterior: BGP

BGP-4 (RFC 4271) is the protocol that makes the internet one network. It is path-vector, running between Autonomous Systems: each AS advertises which prefixes it can reach and the AS path to get there, and each applies its own policy to decide what to accept, prefer and re-advertise. Policy, not shortest path, is the point — best-path selection walks a fixed list:

1
2
3
4
5
6
7
8 1. highest LOCAL_PREF     ← "which exit do I want to use"       (local policy)
2. shortest AS_PATH       ← fewest ASes traversed
3. lowest ORIGIN type
4. lowest MED             ← "which entrance do you want me to use"
5. eBGP preferred over iBGP
6. lowest IGP metric to the next hop
7. deterministic tiebreakers (oldest route, lowest router ID)

Step 1 comes first for a reason: money. LOCAL_PREF is where "prefer settlement-free peering over paid transit" is expressed. The default-free zone currently holds on the order of a million IPv4 prefixes and keeps growing — check the CIDR Report for the live figure rather than trusting a number in a blog post.

Hijacks and route leaks

BGP was designed among mutually trusting operators and authenticates nothing about who may announce what.

Prefix hijack — an AS announces a prefix it does not hold. Because a more specific announcement wins under longest-prefix match, announcing a /24 out of someone's /16 pulls their traffic globally. In 2008 Pakistan Telecom announced a more specific YouTube prefix intending a domestic block and removed YouTube from much of the internet for roughly two hours. In 2018 an announcement covering Amazon Route 53 address space redirected DNS for a cryptocurrency wallet and enabled real theft.
Route leak — an AS re-advertises routes learned from one peer or provider to another, violating the intended relationship. Usually misconfiguration rather than attack, it typically funnels a large share of internet traffic through a network far too small to carry it, producing global congestion. RFC 7908 classifies six distinct leak types.

Defences, none complete: RPKI ROAs record which AS may originate a prefix and Route Origin Validation (RFC 6811) lets routers drop RPKI-invalid announcements — this stops naive origin hijacks but not path forgery or most leaks; prefix filters and IRR data on customer sessions, with max-prefix limits to shut a session that suddenly announces 500,000 routes; and MANRS as the industry norms package. The honest summary: the routing layer is held together by filtering discipline and mutual reputation, incrementally reinforced by cryptography. Treat global reachability as best-effort.

Chapter 5 — Transport layer

Terminology

Port — a 16-bit endpoint number within a host. 0–1023 well-known, 1024–49151 registered, above that dynamic (Linux defaults to 32768–60999, see net.ipv4.ip_local_port_range).
Socket — one endpoint. A TCP connection is identified by the 4-tuple: source IP, source port, destination IP, destination port.
cwnd — congestion window: what the sender believes the network can absorb. rwnd — receive window: what the receiver says it can buffer.
RTO — retransmission timeout: how long to wait for an ACK before resending.

UDP vs TCP

	UDP (RFC 768)	TCP (RFC 9293, originally RFC 793)
Connection	none	three-way handshake
Reliability	none — lost is lost	acknowledgement and retransmission
Ordering	none	strict byte-stream order
Flow / congestion control	none; the app must supply it	both, mandatory
Header	8 bytes	20 bytes minimum
Message boundaries	preserved	none; it is a byte stream
Good for	DNS, DHCP, real-time media, games, QUIC	HTTP, SSH, SMTP, databases, file transfer

UDP is not "unreliable TCP"; it is the raw packet service with ports and a checksum. Choosing it means taking responsibility for whatever reliability you need — exactly the trade QUIC makes in order to implement better reliability than TCP can.

Handshake and teardown

1
2
3
4
5
6
7
8
9 Client                                   Server        Teardown (each direction closes
  | SYN         seq=x                       |          independently, so half-close is legal)
  |---------------------------------------->|            | FIN ---------------->|
  |         SYN,ACK  seq=y, ack=x+1         |            |<--------------- ACK  |
  |<----------------------------------------|            |<--------------- FIN  |
  | ACK         seq=x+1, ack=y+1            |            | ACK ---------------->|
  |---------------------------------------->|            | … TIME_WAIT (2×MSL) …|
  |            ESTABLISHED                   |

The handshake costs one full RTT before any application byte moves — which is why connection reuse matters so much and why TLS 1.3 and QUIC work so hard to collapse round trips. Both sides pick a random Initial Sequence Number; randomising it is a security requirement, since predictable ISNs allow off-path connection spoofing.

A half-open connection consumes server state, so flooding SYNs from spoofed sources fills the accept queue — the classic SYN flood. SYN cookies are the standard defence: the server encodes the connection state into the ISN it returns and keeps no state until the client's ACK proves receipt (net.ipv4.tcp_syncookies).

TIME_WAIT holds the closer for twice the maximum segment lifetime (60 s on Linux) so stray delayed segments cannot be mistaken for part of a new connection reusing the same 4-tuple.

Common pitfall: thousands of sockets in TIME-WAIT on a busy client is normal and healthy, not a leak. Thousands in CLOSE-WAIT is a bug — the application received a FIN and never called close().

Sequence numbers and retransmission

TCP numbers bytes, not packets. The ACK field carries the next byte expected, acknowledging everything before it cumulatively; SACK lets the receiver additionally report non-contiguous blocks it holds, so the sender resends only what is actually missing.

Loss is detected two ways. Fast retransmit treats three duplicate ACKs as a gap and resends immediately; modern Linux adds RACK-TLP (RFC 8985), which infers loss from transmit timestamps and probes the tail of a flight, greatly reducing costly timeouts. Timeout (RTO) is computed from a smoothed RTT estimate and its variance (RFC 6298): RTO = SRTT + max(G, 4 × RTTVAR), initial value 1 second, exponential backoff on repeated failure. An RTO is expensive — it stalls the flow and resets the congestion window to its minimum.

Flow control

The receiver advertises rwnd in every ACK; the sender may never have more unacknowledged data outstanding than that. This protects a slow receiver from a fast sender — purely end-to-end, unrelated to network capacity. The field is 16 bits, capping the window at 65,535 bytes, which Chapter 1 showed is far below the BDP of any long fast path. The window scale option (RFC 7323) multiplies it by a negotiated power of two, up to a gigabyte. It is negotiated only in the SYN, so a middlebox that strips unknown TCP options from SYNs silently caps every connection through it at 6 Mbit/s on a transatlantic path.

Congestion control

Flow control protects the receiver; congestion control protects the network, whose capacity nobody advertises. The sender may have at most min(cwnd, rwnd) bytes in flight. The classic algorithm (RFC 5681):

Slow start — begin small (10 segments is the common initial window) and double cwnd every RTT. Despite the name it is the aggressive phase; it is "slow" only relative to blasting at line rate immediately.
Congestion avoidance — past ssthresh, switch to additive increase: roughly one MSS per RTT, linear.
On loss — multiplicative decrease. Reno halves cwnd on fast retransmit; a full timeout collapses it to the initial window.

This AIMD shape is why loss hurts far more on long paths: recovery is one segment per RTT, so an 80 ms path recovers eight times more slowly than a 10 ms one.

	CUBIC	BBR
Signal	packet loss	measured bottleneck bandwidth and minimum RTT
Growth	cubic function of time since the last loss event	paced sending at the estimated delivery rate
On long fat paths	recovers far faster than Reno	short queues, tolerates non-congestive loss
Weakness	fills buffers, aggravating bufferbloat and latency	fairness against loss-based flows depends on version
Status	Linux default since 2.6.19; RFC 8312, updated by RFC 9438	developed at Google, widely deployed, actively revised

1
2
3 sysctl net.ipv4.tcp_available_congestion_control
sysctl net.ipv4.tcp_congestion_control          # what this host uses

Only the sender's algorithm matters per direction, so a server operator can change this unilaterally. Measure before and after on your own paths — the honest answer to "is BBR faster" is "it depends on your RTT, your loss profile, and what you are competing with."

Head-of-line blocking

TCP delivers bytes in order. If segment 5 of 10 is lost, segments 6–10 sit in the receiver's buffer, already arrived and undeliverable, until the retransmission lands. For a single file that is correct. For HTTP/2, which multiplexes many independent streams over one TCP connection, it is a serious flaw: loss affecting one image stalls every other response on the connection. This is the single biggest reason HTTP/3 abandoned TCP.

Ports and sockets

1
2
3
4 ss -tulpn                 # listening TCP/UDP sockets with owning processes
ss -tan state established # established connections
ss -s                     # summary counts by state

A listening socket is (protocol, local IP, local port); an established connection is the full 4-tuple — which is why one server port serves hundreds of thousands of clients simultaneously, since each connection differs in the remote half. Conversely a client opening many connections to one destination is limited by its ephemeral port range for that destination, around 28,000 on default Linux settings, and the cause of "cannot assign requested address" under load testing.

Chapter 6 — Application layer

Terminology

Resolver — the client-side component turning names into addresses. A stub resolver asks; a recursive resolver does the work.
Authoritative server — holds the real published data for a zone, a contiguous portion of the namespace under one administration.
TTL — how long a record may be cached, in seconds.
SNI — Server Name Indication: the hostname sent in the TLS handshake so a shared IP can select the right certificate.

DNS

DNS (RFC 1034, RFC 1035) is a distributed, hierarchical, aggressively cached database on UDP port 53, falling back to TCP port 53 for large responses, with EDNS(0) (RFC 6891) negotiating larger UDP payloads. A stub resolver knows almost nothing and expects a final answer; a recursive resolver (your ISP's, or 1.1.1.1) does the walking and caches; an authoritative server answers only for its own zones and otherwise returns a referral one level down.

Resolving `www.example.com`

1
2
3
4
5
6
7
8
9
10
11 0. Stub checks /etc/hosts, then the local cache (systemd-resolved, nscd, browser).
1. Stub → recursive resolver: "A record for www.example.com?"   (RD=1)
2. Resolver checks its cache. Assume cold.
3. Resolver → a root server (13 named identities a–m.root-servers.net, each anycast
   to hundreds of sites):     ← referral "ask the .com servers" + glue records
4. Resolver → a .com TLD server:
                              ← referral "ask ns1/ns2.example.com" + glue
5. Resolver → ns1.example.com (authoritative):
                              ← answer: www.example.com. 300 IN A 203.0.113.80  (AA=1)
6. Resolver caches for the TTL and returns to the stub (RA=1, AA=0).

1
2
3
4
5 dig +trace www.example.com     # perform the walk yourself, referral by referral
dig @1.1.1.1 example.com A     # bypass the system resolver
dig example.com SOA            # zone metadata, including negative-cache TTL
dig example.com +nssearch      # ask each authoritative server directly

Records worth knowing

A/AAAA are addresses. CNAME is an alias and cannot coexist with other records at the same name, so never at a zone apex. MX lists mail exchangers with preferences. TXT carries SPF, DKIM, DMARC and ownership proofs. NS names a zone's authoritative servers, SOA carries the serial, refresh/retry/expire timers and the negative-caching TTL (RFC 2308). PTR is reverse lookup, SRV gives host, port, priority and weight, CAA restricts which CAs may issue. DS, DNSKEY, RRSIG and NSEC/NSEC3 are DNSSEC material, and HTTPS/SVCB deliver service parameters — ALPN, port, address hints — in one lookup.

TTL and caching

TTL is the contract that makes DNS scale and the trap that makes migrations painful. A record with TTL 86400 may be cached for a day by every resolver worldwide and you cannot recall it. The discipline is non-negotiable: lower the TTL to 60–300 seconds at least one full old-TTL period before a planned change, make the change, verify, then raise it again. Negative answers are cached too, governed by the SOA minimum — which is why a typo'd record can look broken for a while after you fix it.

DNSSEC and encrypted transports

DNS answers are unauthenticated by default, which historically enabled cache poisoning: race a forged answer to a resolver and pollute it for the whole TTL. Source-port randomisation and query-name mixing (RFC 5452 and successors) raised the cost but authenticate nothing. DNSSEC (RFC 4033/4034/4035) signs record sets, publishes the public key as DNSKEY, and has the parent zone publish a DS hash of it — a chain of trust from the root down, letting a validating resolver prove an answer is genuine or that a name genuinely does not exist. It provides authenticity and integrity, not confidentiality, and its cost is operational: key rollovers and expired signatures fail closed.

	Port	Spec	Note
DoT — DNS over TLS	853	RFC 7858	dedicated port, so operators can see and police DNS
DoH — DNS over HTTPS	443	RFC 8484	indistinguishable from web traffic; filters lose visibility
DoQ — DNS over QUIC	853/UDP	RFC 9250	lower handshake cost than DoT

These encrypt the stub-to-resolver hop only. They hide nothing from the resolver itself, which makes "which resolver do you trust" a policy question as much as a technical one.

HTTP/1.1 → HTTP/2 → HTTP/3

	HTTP/1.1 (RFC 9112)	HTTP/2 (RFC 9113)	HTTP/3 (RFC 9114)
Framing	text, one request at a time per connection	binary frames, multiplexed streams	binary frames over QUIC streams
Concurrency	~6 TCP connections per origin	many streams on one TCP connection	many streams on one QUIC connection
Header compression	none	HPACK (RFC 7541)	QPACK (RFC 9204)
Transport	TCP	TCP	QUIC over UDP
HOL blocking	per connection, request level	gone in HTTP, remains in TCP	gone at both levels
Encryption	optional	in practice mandatory	mandatory, built in

HTTP/1.1's workaround for concurrency was six connections per origin, which drove the sprite sheets, domain sharding and file concatenation that dominated front-end practice for a decade. HTTP/2 made those anti-patterns — but only moved head-of-line blocking down a layer, where TCP still enforces in-order delivery. HTTP/3 completes the idea.

QUIC

QUIC (RFC 9000, TLS integration in RFC 9001, loss recovery in RFC 9002) is a transport built on UDP, in user space, with TLS 1.3 fused into it. Moving to UDP was not a preference for datagrams; it was the only way to deploy a new transport at all, because middleboxes everywhere inspect and mangle TCP and will not pass an unknown IP protocol number. UDP is the escape hatch, and user-space implementation lets the protocol evolve at application release cadence rather than kernel upgrade cadence.

What it fixes: no transport head-of-line blocking, since QUIC streams are independent and a lost packet stalls only its own stream; connection migration, because a QUIC connection is identified by a connection ID rather than the 4-tuple, so moving from Wi-Fi to cellular changes the IP but not the connection and the session survives with no reconnect and no lost upload; fewer round trips, with transport and cryptographic handshakes combined into 1-RTT for a new connection and 0-RTT on resumption; encryption by default including most of the transport header, so middleboxes cannot ossify it the way they ossified TCP; and unambiguous packet numbers that are never reused for retransmissions, removing TCP's retransmission ambiguity.

Costs, stated plainly: higher CPU per byte than kernel TCP (improving with offload), some networks block or throttle UDP, and 0-RTT data is replayable so it must carry only idempotent requests. Discovery is via the Alt-Svc header or an HTTPS DNS record — the client starts on TCP and upgrades.

TLS 1.3

TLS 1.3 (RFC 8446) is both faster and smaller than TLS 1.2 (RFC 5246), and both properties come from the same decision: delete the insecure options rather than negotiate them.

1
2
3
4
5
6
7
8
9
10
11 Client                                                        Server
  ClientHello + supported_versions, key_share (a guess), SNI, ALPN
  ------------------------------------------------------------------>
                                             ServerHello + key_share
                     {EncryptedExtensions, Certificate,
                      CertificateVerify, Finished}
  <------------------------------------------------------------------
  {Finished} + application data
  ------------------------------------------------------------------>
      one round trip; everything in {braces} is already encrypted

Changes from 1.2: 1-RTT instead of 2-RTT, because the client guesses the key-exchange group and sends its share in the first flight. Forward secrecy is mandatory — RSA key transport and static Diffie-Hellman are gone, so a future compromise of the server key cannot decrypt recorded sessions. The cipher suite list is short and AEAD-only; CBC modes, RC4, static RSA, compression, renegotiation and custom DH groups were all removed, and most TLS vulnerabilities of the 2010s targeted exactly those. More of the handshake is encrypted, including the certificate. 0-RTT resumption exists, with the same replay caveat as QUIC.

Certificates and chains of trust

An X.509 certificate (RFC 5280) binds a public key to identities. Validation checks, in order: the signature chain reaches a root in the client's trust store; every certificate is within its validity period; the requested hostname matches a Subject Alternative Name; key usage permits this use; and the certificate is not revoked.

1
2
3 openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null \
  | openssl x509 -noout -subject -issuer -dates -ext subjectAltName

Symptom	Real cause
"certificate has expired"	expired leaf — or a client with a wrong system clock
works in `curl`, fails in a browser or JVM	the server omits the intermediate; `curl` may have it cached, others do not
"certificate name mismatch"	wrong SNI, or a SAN list missing the hostname
"unknown authority" inside a container	the image has no CA bundle installed
works for one vhost, not another	SNI not sent, so the server returned its default certificate

Supporting machinery: CAA records restrict which CAs may issue for a domain; Certificate Transparency (RFC 6962, updated by RFC 9162) publishes every issued certificate to append-only logs so mis-issuance is detectable; ACME (RFC 8555) automates issuance and renewal, which matters more every year as certificate lifetimes shorten. Automate renewal or you will eventually have an outage caused by a calendar.

Chapter 7 — Delivery and scale

Terminology

Reverse proxy — a server that accepts client connections and forwards them to backends on the client's behalf. The origin is the authoritative server behind all caches.
PoP — point of presence: a CDN's edge location.
Cache key — the request attributes identifying a cache entry: usually method, host, path and query, plus whatever Vary names.
Anycast — announcing one IP address from many locations so routing delivers each client to a nearby one.

Load balancing: L4 vs L7

	L4 (transport)	L7 (application)
Decides on	IP addresses and ports	URL path, headers, cookies, method, host
Sees payload	no	yes
TLS	passes through	usually terminates
Per-connection cost	very low	higher; it parses HTTP
Can do	connection distribution, non-HTTP protocols	path routing, header rewriting, retries, canaries, rate limits, WAF
AWS equivalent	Network Load Balancer	Application Load Balancer

A TCP connection handled at L4 pins to one backend for its whole life. At L7 each request can go somewhere different, which is what makes canary releases and per-endpoint routing possible.

Algorithm	Behaviour	Use when
Round robin	strict rotation	backends identical, requests uniform
Weighted round robin	rotation with capacity weights	heterogeneous instance sizes
Least connections	fewest active connections wins	request durations vary widely
Least response time	combines connections and latency	backend performance is uneven
Source-IP hash	same client → same backend	crude affinity without cookies
Consistent hashing	a key maps to a backend; adding one moves ~1/N of keys	caches and sharded stores
Power of two choices	sample two at random, pick the less loaded	large fleets; near least-connections quality, far less state

Trick: plain round robin plus a slow endpoint equals a queue on every backend. Least-connections is the safer default whenever request cost is not uniform, because it naturally routes away from anything that has become slow.

Health checks and affinity

Active checks probe each backend on an interval — cheap, predictable, the standard. Passive checks observe real traffic and eject a backend after N consecutive errors — faster to react, but needs traffic to learn. A TCP connect proves the process is listening; an HTTP /healthz that touches the database proves it can actually serve.

Design the endpoint carefully. A deep check that fails when a shared dependency degrades marks every backend unhealthy at once and takes the service fully down — a self-inflicted outage worse than degraded service. The usual compromise is a shallow liveness check deciding restarts and a deeper readiness check deciding traffic, with a floor guaranteeing the balancer never removes the last healthy backends.

Session affinity pins a client to one backend via a balancer-set cookie or a source-IP hash. It is a workaround for server-side state and it costs real properties: uneven load, lost sessions when a backend dies, and much harder deploys. Prefer stateless backends with session state in a shared store; use affinity only where an in-memory cache or a stateful long-lived connection genuinely requires it.

Reverse proxies, CDNs, anycast

A reverse proxy (nginx, HAProxy, Envoy, Caddy) is the load balancer's more general sibling, typically also handling TLS termination, HTTP version translation (HTTP/3 to the client, HTTP/1.1 to a legacy backend), compression, buffering, timeouts, retries, circuit breaking and access control. Concentrating these in one tier means applications do not reimplement them per service.

Watch out: behind a proxy the backend sees the proxy's address as the client. Propagate the real one with X-Forwarded-For / Forwarded (RFC 7239) — and trust it only from known proxy addresses. A blindly trusted X-Forwarded-For is a client-controlled header, and it will be used to bypass your IP allowlists and rate limits.

A CDN is a global fleet of reverse-proxy caches. A client resolves your hostname to a nearby PoP, which serves from cache or fetches from the origin, often via an origin shield that collapses many PoPs' misses into one origin request. You gain latency (bytes start closer), origin offload (the origin serves misses only), and DDoS absorption (the attack lands on distributed edge capacity). You must manage invalidation and cache-key correctness — anything cached wrongly is now wrong in hundreds of places at once.

Anycast announces the same prefix from many locations and lets BGP deliver each client to the routing-closest instance. It is how the DNS root servers, public resolvers and every CDN's edge addresses work: no DNS TTL delay when steering traffic, automatic failover when a site withdraws its announcement, and DDoS split across sites by the routing system itself. The limits: "closest" means shortest AS path, not lowest latency, and a route change mid-connection can move a client to a different site and break a long-lived TCP connection. That last problem is exactly what QUIC's connection migration is designed to survive, which is one reason CDNs adopted HTTP/3 early.

Cache-Control decides whether any of this helps

A CDN can only cache what you tell it to. Caching semantics are RFC 9111.

Directive	Effect
`max-age=N`	any cache may reuse for N seconds
`s-maxage=N`	overrides `max-age` for shared caches (CDN, proxy) only
`public` / `private`	shared caches may store / browser only
`no-cache`	may store, but must revalidate before every reuse
`no-store`	must not be written to any cache at all
`must-revalidate`	once stale, never serve without revalidating
`immutable`	do not revalidate within the freshness window, even on reload
`stale-while-revalidate=N`	serve stale for N seconds while refreshing in the background (RFC 5861)
`stale-if-error=N`	serve stale for N seconds if the origin is failing (RFC 5861)

Validators refresh cheaply: ETag with If-None-Match, or Last-Modified with If-Modified-Since, producing a 304 Not Modified with no body. Vary names request headers forming part of the cache key — Vary: Accept-Encoding is correct and necessary, Vary: User-Agent fragments the cache into near-uselessness, Vary: * disables it.

1
2
3
4
5
6
7
8
9
10 Fingerprinted static assets (app.9f3b2c.js)
  Cache-Control: public, max-age=31536000, immutable    → forever; a deploy renames the file
HTML documents
  Cache-Control: no-cache  + ETag                       → always revalidate; 304s are near-free
Personalised API responses
  Cache-Control: private, no-store                      → never in a shared cache
Public, slow-changing API responses
  Cache-Control: public, s-maxage=60, stale-while-revalidate=300
                                                        → CDN absorbs load; nobody waits

Common pitfall: no-cache does not mean "do not cache" — it means "revalidate before reuse". The directive that means store nothing is no-store. Confusing them either leaks private data into shared caches or throws away all the benefit of validation.

Chapter 8 — Practical diagnosis

What each tool actually proves

Tool	Proves	Does not prove
`ping host`	ICMP echo reached the host and returned; RTT and loss	that any service is up; ICMP is often deprioritised or blocked, so loss here may be fictional
`traceroute host`	the forward path, via expiring TTL and ICMP Time Exceeded	the return path, which may differ entirely; nor real per-hop latency
`mtr host`	sustained per-hop loss and latency over time	that middle-hop loss is real — routers rate-limit ICMP generation; only loss at the final hop counts
`dig name`	what DNS answers, from a chosen server, with TTLs	what your application resolves — that depends on its own cache and config
`nslookup name`	the same, and it ships on Windows by default	as much detail as `dig`; prefer `dig` where available
`ss -tulpn` / `netstat -an`	which local sockets exist, in which state, owned by which process	anything about the remote end
`nc -vz host port`	a TCP connection to that port completed	that the service speaks the protocol you expect
`curl -v -w ...`	the full HTTP transaction with a timing breakdown	client-side rendering behaviour
`openssl s_client`	the chain, protocol version and cipher actually negotiated	whether your client's trust store accepts it
`tcpdump -i any -n ...`	exactly which packets crossed this interface	anything about packets dropped before this interface
`ip route get IP`	which routing entry the kernel will use	that the next hop is reachable
`iperf3 -c host`	achievable throughput between two hosts you control	anything about a third party's capacity

Two rules govern all of them. Symmetry is not guaranteed — forward and return paths may differ, so a one-way problem is normal and traceroute shows one direction only. And ICMP is a second-class citizen — routers generate it at low priority and rate-limit it, so intermediate-hop loss in mtr is usually an artefact.

The troubleshooting ladder

Work upward; each rung assumes the ones below passed. Stopping at the first failure is the entire technique, and most wasted debugging time comes from starting at layer 7.

1 — Physical and link. ip -br link (is it UP with carrier?), ethtool eth0 | grep -E 'Speed|Duplex|Link detected', iw dev wlan0 link for Wi-Fi signal and rate. Down or no carrier means cable, port, driver or radio, and nothing above can work.

2 — Layer 2 neighbours. ip neigh show. A gateway stuck INCOMPLETE means ARP is not being answered: wrong VLAN, wrong subnet mask, port security, or the gateway is genuinely down.

3 — Layer 3 addressing and routing.

1
2
3
4
5 ip addr show          # address and prefix — is the mask right?
ip route show         # is there a default route?
ping -c3 192.0.2.1    # the gateway
ping -c3 1.1.1.1      # the internet by IP, bypassing DNS entirely

The critical split: if ping 1.1.1.1 works but names fail, it is DNS — skip to rung 4. If it fails, stay at layer 3.

4 — DNS.

1
2
3
4
5 dig example.com A +short           # via the system resolver
dig @1.1.1.1 example.com A +short  # via a known-good public resolver
resolvectl status                  # which resolver is the system actually using?
grep -v '^#' /etc/hosts            # a stale override here explains many mysteries

Different answers from different resolvers means propagation delay after a change, split-horizon DNS, or a stale/poisoned cache. Compare the TTLs.

5 — Layer 4 reachability. nc -vz example.com 443, curl -v --connect-timeout 5 https://example.com. Read the failure mode, because it names the cause:

Result	Means
Connection refused (RST)	reached the host; nothing is listening on that port
Connection timed out	SYN silently dropped: firewall, security group, or a black-holed route
Connects then hangs	reached the service and it is not responding — or an MTU black hole

6 — TLS. openssl s_client -connect example.com:443 -servername example.com </dev/null, and check date -u, since clock skew invalidates every certificate check. Verify expiry, SAN list, chain completeness and negotiated version.

7 — Application.

1
2
3
4 curl -v -o /dev/null -s -w '\
 dns:%{time_namelookup}s connect:%{time_connect}s tls:%{time_appconnect}s\
 ttfb:%{time_starttransfer}s total:%{time_total}s code:%{http_code}\n' https://example.com/

This one command decomposes "slow" into a layer:

Dominant figure	Diagnosis
`dns`	resolver latency, cold cache, or a failing name server
`connect` − `dns`	network RTT and TCP handshake — a distance or path problem
`tls` − `connect`	handshake cost — check for TLS 1.2, missing resumption, an oversized chain
`ttfb` − `tls`	the server is slow. The network has done its job; go look at the application
`total` − `ttfb`	transfer time — bandwidth, window sizing, or an uncompressed response

8 — When it is "slow" rather than "broken".

1
2
3
4
5
6 ping -c 50 example.com          # RTT distribution and loss
mtr -rwzc 100 example.com       # sustained per-hop view; read the LAST hop
ping -M do -s 1472 example.com  # MTU probe (Chapter 3)
iperf3 -c your-own-host         # raw throughput between hosts you control
ss -ti                          # per-socket cwnd, rtt, retransmits — the real story

ss -ti is the underused one. It reports each connection's rtt, cwnd and retrans directly, turning Chapter 5's congestion-control discussion into observable numbers: a low cwnd with a climbing retrans count is loss-induced throughput collapse, not a bandwidth shortage. Capture packets only once the ladder has narrowed the problem — a capture without a hypothesis is a haystack:

1
2
3 tcpdump -i any -n host example.com and port 443 -w /tmp/capture.pcap
tcpdump -i any -n 'tcp[tcpflags] & (tcp-syn|tcp-rst) != 0'   # handshakes and resets only

Practice Exercises

Exercise 1 — Subnet arithmetic by hand

For each, give the network address, broadcast, usable host range and usable host count. No calculators.
192.168.10.77/26 · 10.13.200.45/21 · 172.20.130.200/27 · 198.51.100.9/30

Solution

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16 192.168.10.77/26   mask 255.255.255.192, block 64 in octet 4
  network 192.168.10.64   broadcast 192.168.10.127
  usable  192.168.10.65 – 192.168.10.126        hosts 2^6 − 2 = 62
 
10.13.200.45/21    mask 255.255.248.0, block 8 in octet 3 (200 is a multiple of 8)
  network 10.13.200.0     broadcast 10.13.207.255
  usable  10.13.200.1 – 10.13.207.254           hosts 2^11 − 2 = 2046
 
172.20.130.200/27  mask 255.255.255.224, block 32; boundaries … 160, 192, 224 → 200 is in 192…223
  network 172.20.130.192  broadcast 172.20.130.223
  usable  172.20.130.193 – 172.20.130.222       hosts 2^5 − 2 = 30
 
198.51.100.9/30    mask 255.255.255.252, block 4; boundaries 0, 4, 8, 12 → 9 is in 8…11
  network 198.51.100.8    broadcast 198.51.100.11
  usable  198.51.100.9 – 198.51.100.10          hosts 2^2 − 2 = 2

Exercise 2 — Design a VLSM plan

From 10.50.0.0/20, allocate non-overlapping blocks for 1,000 workstations, 400 VoIP phones, 200 servers, 60 printers, 25 cameras and four point-to-point router links. Leave the remainder contiguous.

Solution

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17 10.50.0.0/20 = 10.50.0.0 – 10.50.15.255 (4096 addresses). Largest first, each self-aligned:
 
10.50.0.0/22     workstations  1022 usable   10.50.0.0   – 10.50.3.255
10.50.4.0/23     VoIP           510 usable   10.50.4.0   – 10.50.5.255
10.50.6.0/24     servers        254 usable   10.50.6.0   – 10.50.6.255
10.50.7.0/26     printers        62 usable   10.50.7.0   – 10.50.7.63
10.50.7.64/27    cameras         30 usable   10.50.7.64  – 10.50.7.95
10.50.7.96/30    link 1           2 usable   10.50.7.96  – 10.50.7.99
10.50.7.100/30   link 2           2 usable   10.50.7.100 – 10.50.7.103
10.50.7.104/30   link 3           2 usable   10.50.7.104 – 10.50.7.107
10.50.7.108/30   link 4           2 usable   10.50.7.108 – 10.50.7.111
free             10.50.7.112 – 10.50.15.255  (2192 addresses, including a clean 10.50.8.0/21)
 
Alignment check: /22 starts at a multiple of 4 in octet 3 (0 ✓, and /23 at a multiple of 2: 4 ✓);
/26 at a multiple of 64 in octet 4 (0 ✓); /27 at a multiple of 32 (64 ✓);
each /30 at a multiple of 4 (96, 100, 104, 108 ✓).

Exercise 3 — Longest-prefix match

Given 0.0.0.0/0 → 203.0.113.1, 10.0.0.0/8 → 10.0.0.1, 10.2.0.0/16 → 10.2.0.1, 10.2.30.0/24 → 10.2.30.1, 172.16.0.0/12 → 10.0.0.9, name the next hop for 10.2.30.40, 10.2.99.1, 10.7.0.1, 172.16.0.5 and 1.1.1.1.

Solution

1
2
3
4
5
6
7
8 10.2.30.40   /0, /8, /16, /24              → /24  → 10.2.30.1
10.2.99.1    /0, /8, /16                   → /16  → 10.2.0.1
10.7.0.1     /0, /8                        → /8   → 10.0.0.1
172.16.0.5   /0 and 172.16.0.0/12          → /12  → 10.0.0.9   (172.16–172.31 only)
1.1.1.1      /0 only                       → /0   → 203.0.113.1
 
Table order is irrelevant; only prefix length decides.

Exercise 4 — Bandwidth-delay product

A 10 Gbit/s path has an RTT of 120 ms. (a) What is the BDP in bytes? (b) What window is needed to fill it? (c) What is the maximum single-connection throughput without window scaling?

Solution

1
2
3
4
5
6
7
8
9 (a) 10,000,000,000 × 0.120 = 1,200,000,000 bits = 150,000,000 bytes ≈ 143 MiB
 
(b) The window must be at least the BDP — ~150 MB in flight. The 16-bit field maxes at
    65,535, so a window scale factor of at least 2^12 is required
    (65,535 × 4096 ≈ 268 MB of headroom).
 
(c) 65,535 × 8 ÷ 0.120 = 524,280 ÷ 0.120 ≈ 4,369,000 bits/s ≈ 4.37 Mbit/s
    A 10 Gbit/s path delivering 4.4 Mbit/s per connection — a ~2,300× shortfall.

Exercise 5 — Read a DNS delegation

Run dig +trace www.iana.org. For each stage record which server was asked, whether the response was a referral or an answer, and the TTL on the final record. Run it again immediately against your system resolver and explain why the TTL is now lower.

Exercise 6 — Prove the failure mode

On a host you control, produce each outcome deliberately and record what nc -vz and curl -v report: (1) connection refused, with nothing listening; (2) connection timed out, by adding then removing a firewall DROP rule; (3) a TLS certificate name mismatch, by connecting via --resolve or requesting a hostname the certificate does not cover.

Exercise 7 — Decompose a slow page

Run the curl -w timing block from Chapter 8 ten times against a site hosted far from you, and record which component dominates. Then run it against a CDN-fronted asset on the same site and explain the difference in terms of Chapters 1 and 7.

Solution

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15 Distant, non-cached origin:
  dns      small after the first run (cached)
  connect  ≈ one RTT — dominated by distance
  tls      ≈ one more RTT with TLS 1.3, two with TLS 1.2
  ttfb     connect + tls + server think time + one more RTT for the response
  total    ttfb + transfer
 
CDN-fronted asset: connect and tls both fall sharply, because the TCP and TLS handshakes
now terminate at a nearby PoP rather than the origin — the RTT term in the BDP formula
dropped, and every handshake round trip got cheaper by the same factor. ttfb falls further
on a cache HIT, because the origin is never consulted; confirm with the CDN's cache-status
response header (names vary: Age, X-Cache, CF-Cache-Status).
 
Conclusion: the CDN did not add bandwidth. It reduced RTT, and every round trip is priced in RTT.

Cheat-sheet

What	How
Usable hosts in a prefix	`2^(32 − prefix) − 2`
Block size in the interesting octet	`256 − mask octet`
Network address	round the address down to the block boundary
Broadcast address	next boundary − 1; octets to the right become 255
Bandwidth-delay product	`bandwidth (bits/s) × RTT (s)`; ÷ 8 for bytes
MSS on a 1500 MTU	`1460` (IPv4 + TCP), `1440` (IPv6 + TCP)
RFC 1918 private ranges	`10/8`, `172.16/12` (`.16`–`.31` only), `192.168/16`
CGNAT shared space	`100.64.0.0/10`
Route selection rule	longest prefix match wins; table order is irrelevant
BGP first tiebreaker	highest `LOCAL_PREF` — policy before path length
Origin-hijack defence	RPKI ROAs plus Route Origin Validation (RFC 6811)
TCP connection identity	the 4-tuple: src IP, src port, dst IP, dst port
`TIME-WAIT` vs `CLOSE-WAIT`	`TIME-WAIT` is normal; many `CLOSE-WAIT` means the app never called `close()`
Lower a DNS TTL before a migration	at least one full old-TTL period in advance
`no-cache` vs `no-store`	revalidate before reuse vs never store at all; fingerprinted assets get `public, max-age=31536000, immutable`
Test path MTU	`ping -M do -s 1472 host` (1472 + 8 + 20 = 1500)
Is it DNS?	`ping 1.1.1.1` works but names fail → yes, it is DNS
Refused vs timeout	RST = nothing listening; timeout = firewall dropping
Decompose "slow"	`curl -w` timing block: dns / connect / tls / ttfb / total
Per-socket truth	`ss -ti` for live `rtt`, `cwnd` and `retrans`
Read `mtr` correctly	only loss at the final hop is real

Where to go deeper

Apply Chapters 3 and 4 in a cloud — AWS Course — 01 · Network First is this material with a control plane: VPC CIDRs are RFC 1918 planning, subnets are the subnetting arithmetic above, route tables are longest-prefix match, security groups are stateful firewalls, NACLs are stateless ones, and NAT gateways are Chapter 4's NAT.
Build a lab. Three Linux network namespaces joined by veth pairs, plus ip route and iptables, teach more in an afternoon than any diagram. Add FRRouting to run real OSPF and BGP between them.
Read one capture properly. Load a single HTTPS page load into Wireshark and account for every packet: handshake, TLS, HTTP/2 frames, ACK pattern, FIN. Doing this once changes how you read everything else.
Follow the specs. RFC 9293 (TCP), RFC 9000 (QUIC), RFC 8446 (TLS 1.3) and RFC 9110/9111 (HTTP semantics and caching) are unusually readable modern documents.
Watch routing live. RIPE RIS and RouteViews publish real BGP data; RIPE Atlas lets you run measurements from thousands of vantage points for free.

References

Planned future extensions: topics that deserve their own posts rather than another section here.

Subnetting drills — a generated worksheet of 50 problems with answers, plus the binary-AND method written out for people who prefer it to block sizes.
BGP in a lab — FRRouting across network namespaces: eBGP sessions, AS paths, LOCAL_PREF policy, deliberate route leaks, and RPKI validation with a local validator.
Wireshark, one capture at a time — annotated walkthroughs of a TLS 1.3 handshake, an HTTP/2 multiplexed page load, and a QUIC connection migration.
Kubernetes networking — the CNI model, Services, kube-proxy versus eBPF datapaths, Ingress and Gateway API, and where these fundamentals reappear under new names.
Network security in depth — DDoS classes and mitigation, WAFs, zero trust and mTLS between services, IPsec and WireGuard, and 802.1X on the wired edge.
Real-time media — WebRTC, jitter buffers, RTP/RTCP, adaptive bitrate, and why the metrics that matter for a video call are the opposite of those for a file transfer.

Learning Outcome

Chapter 1 — Fundamentals

Terminology

Circuit switching vs packet switching

Bandwidth is not speed

Bandwidth-delay product

Why layers

Chapter 2 — OSI and TCP/IP

Terminology

The two models

One packet, encapsulated

OSI is a teaching tool; TCP/IP is what runs

Chapter 3 — Link layer

Terminology

The Ethernet frame

MAC addresses

What a switch actually does

ARP

VLANs

MTU, and why 1500

Wi-Fi is not wireless Ethernet

Chapter 4 — Network layer

Terminology

IPv4 addressing and CIDR

Subnetting, step by step

Worked example 1 — 192.168.10.77/26

Worked example 2 — 10.13.200.45/21

Worked example 3 — VLSM allocation

Special and private ranges

NAT and what it costs

IPv6

SLAAC

Dual stack and why adoption matters

Routing

Longest-prefix match

Static vs dynamic

Interior: OSPF

Exterior: BGP

Hijacks and route leaks

Chapter 5 — Transport layer

Terminology

UDP vs TCP

Handshake and teardown

Sequence numbers and retransmission

Flow control

Congestion control

Head-of-line blocking

Ports and sockets

Chapter 6 — Application layer

Terminology

DNS

Resolving www.example.com

Records worth knowing

TTL and caching

DNSSEC and encrypted transports

HTTP/1.1 → HTTP/2 → HTTP/3

QUIC

TLS 1.3

Certificates and chains of trust

Chapter 7 — Delivery and scale

Terminology

Load balancing: L4 vs L7

Health checks and affinity

Reverse proxies, CDNs, anycast

Cache-Control decides whether any of this helps

Chapter 8 — Practical diagnosis

What each tool actually proves

The troubleshooting ladder

Practice Exercises

Exercise 1 — Subnet arithmetic by hand

Exercise 2 — Design a VLSM plan

Exercise 3 — Longest-prefix match

Exercise 4 — Bandwidth-delay product

Exercise 5 — Read a DNS delegation

Exercise 6 — Prove the failure mode

Exercise 7 — Decompose a slow page

Cheat-sheet

Where to go deeper

References

Worked example 1 — `192.168.10.77/26`

Worked example 2 — `10.13.200.45/21`

Resolving `www.example.com`