Every call over a network can be slow, can fail, or can fail in a way where you do not know if it worked. This post is a practical kit for those cases: timeouts on every call, retries that do not make an outage worse, idempotency keys so a retry never charges a card twice, rate limits that protect a server, circuit breakers, and queues. All code is runnable Python. It sits in Part C of the series, after Caching and before Testing.

Series map: 01 Text Encoding → … → 13 Caching → 14 API Reliability (this post) → 15 Testing → … → 24 Desk Health. Part C · Building safely.

Learning Outcome

  • Name the four ways a remote call fails, and why "unknown outcome" is the hard one.
  • Put a connect timeout and a read timeout on every call in requests and httpx.
  • Decide what is safe to retry, using HTTP method rules from RFC 9110 and the status code.
  • Write a retry decorator with exponential backoff and full jitter, and respect Retry-After.
  • Make POST safe to retry with an Idempotency-Key header and a server-side store with a TTL.
  • Build a token-bucket rate limiter that returns 429 Too Many Requests with Retry-After.
  • Explain circuit-breaker states, queue load levelling, dead-letter queues and idempotent consumers.

Chapter 1 — How remote calls fail

A function call inside one process either returns or throws. A call over a network has more outcomes:

Failure What you see Example
Slow The call takes 30 s instead of 30 ms Database is overloaded, a GC pause, a long queue
Down Connection refused, DNS error, connect timeout Service crashed, wrong port, network cut
Partial failure Some calls work, some fail One of five servers is broken; one region is down
Unknown outcome A timeout after the request was sent POST /payments timed out. Was the card charged?

The last one is the hard case. The request may have reached the server and run. Only the reply was lost. If you retry blindly, you may do the action twice. If you do not retry, the action may never happen. Most of this post is about handling this case safely.

Terminology

Term Meaning
Timeout The longest you wait for one step (connect, read) before giving up
Deadline The total time budget for the whole operation, including retries
Backoff Waiting longer before each new retry
Jitter Adding randomness to the wait, so clients do not retry at the same moment
Safe method A method the client does not expect to change anything (GET, HEAD)
Idempotent Doing it twice has the same intended effect as doing it once
Idempotency key A unique ID the client sends so the server can spot a repeat
Rate limit A cap on how many requests a client may send per time window
Circuit breaker A switch that stops calls to a failing service for a while
At-least-once A message is delivered one or more times — duplicates are possible
Dead-letter queue (DLQ) A side queue for messages that failed too many times

Chapter 2 — Timeouts on every network call

A call with no timeout can hang forever. One hung call holds a thread, a connection and memory. Enough of them and the whole service stops. Rule: every network call gets a timeout.

Connect vs read timeouts

  • Connect timeout — how long to wait to open the TCP connection. If it fails, the request was never sent, so a retry is always safe.
  • Read timeout — how long to wait for the server's reply after sending. If it fails, the server may have done the work.

requests

The requests library has no default timeout. Its docs say requests do not time out unless you set one.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
import requests
 
# (connect, read) in seconds
r = requests.get("https://httpbin.org/delay/1", timeout=(3.05, 10))
r.raise_for_status()
print(r.status_code)
 
try:
    requests.get("https://httpbin.org/delay/5", timeout=(3.05, 2))
except requests.exceptions.ConnectTimeout:
    print("could not connect — safe to retry")
except requests.exceptions.ReadTimeout:
    print("sent, but no reply in time — outcome unknown")

Watch out: the requests read timeout is the time between bytes from the server, not the total download time. A server that sends one byte every 9 seconds never trips a 10-second read timeout. Neither timeout is a wall-clock limit. For a hard total limit, use a deadline (below).

The requests docs suggest a connect timeout slightly larger than a multiple of 3 (for example 3.05), because 3 seconds is the classic TCP retransmission window.

httpx

httpx has a default timeout of 5 seconds and four separate settings: connect, read, write and pool (waiting for a free connection). Each raises its own exception: ConnectTimeout, ReadTimeout, WriteTimeout, PoolTimeout.

1
2
3
4
5
6
7
8
9
10
11
import httpx
 
timeout = httpx.Timeout(10.0, connect=3.0)   # read/write/pool = 10 s, connect = 3 s
with httpx.Client(timeout=timeout) as client:
    try:
        print(client.get("https://httpbin.org/get").status_code)
    except httpx.ConnectTimeout:
        print("connect timeout — safe to retry")
    except httpx.ReadTimeout:
        print("read timeout — outcome unknown for non-idempotent calls")

From the shell

In Windows PowerShell 5.1, curl is an alias for Invoke-WebRequest. Type curl.exe to get real curl.

1
2
3
# Git Bash / Linux / macOS
curl --connect-timeout 3 --max-time 10 https://httpbin.org/get
 
1
2
3
4
# Windows PowerShell 5.1
curl.exe --connect-timeout 3 --max-time 10 https://httpbin.org/get
Invoke-RestMethod -Uri https://httpbin.org/get -TimeoutSec 10

Deadlines

A timeout limits one attempt. A deadline limits the whole job. If a user request has 2 seconds in total, and 1.5 s are already spent, the next downstream call should get at most 0.5 s. Pass the remaining time down (as an argument or a header) so inner calls never outlive the outer request.

Trick: pick timeouts from real latency data. A common starting point is a bit above the downstream's p99 latency. Too long and a slow dependency blocks you; too short and you fail healthy calls.

Chapter 3 — What is safe to retry

Method rules from RFC 9110

RFC 9110 (HTTP Semantics) defines two properties:

Method Safe Idempotent
GET, HEAD, OPTIONS, TRACE yes yes
PUT, DELETE no yes
POST, PATCH no no

(PATCH is defined in RFC 5789, which says it is neither safe nor idempotent.)

Idempotent means: the intended effect of sending the request once is the same as sending it many times. PUT /users/7 {"name": "Ada"} twice leaves the same state. POST /orders twice may create two orders.

RFC 9110 is direct about retries: a client SHOULD NOT automatically retry a non-idempotent request unless it knows the request is really idempotent, or knows the original was never received. That is exactly what idempotency keys (Chapter 5) give you.

Watch out: these are promises your server must keep. A GET /reset-password that sends an email is not safe, whatever the RFC says. A DELETE that returns 404 the second time is still idempotent — the state is the same, even if the reply differs.

Status codes (common practice)

Response Retry? Why
Connect error / connect timeout Yes Request never reached the server
Read timeout, connection reset Only if idempotent or has an idempotency key Server may have run it
408, 429, 502, 503, 504 Yes, with backoff Temporary. For 429/503 read Retry-After
500 Maybe, a few times, idempotent only Could be a bug that will fail again
400, 401, 403, 404, 409, 422 No Retrying the same request gives the same error

The popular urllib3 Retry class (requests uses urllib3 under the hood) follows the method rule by default: its DEFAULT_ALLOWED_METHODS are DELETE, GET, HEAD, OPTIONS, PUT, TRACE — no POST. It also respects Retry-After by default for status codes 413, 429 and 503.

Chapter 4 — Retries with exponential backoff and jitter

If a server is overloaded and every client retries at once, the retries keep it overloaded. Two fixes:

  1. Exponential backoff — wait longer after each failure: base * 2**attempt, up to a cap.
  2. Jitter — add randomness, so clients spread out instead of retrying in waves.

Marc Brooker's AWS Architecture Blog post "Exponential Backoff And Jitter" (2015) compared several variants. Its formulas:

Variant Sleep
No jitter min(cap, base * 2 ** attempt)
Full Jitter random(0, min(cap, base * 2 ** attempt))
Equal Jitter half + random(0, half) where half = base * 2 ** attempt / 2
Decorrelated Jitter min(cap, random(base, sleep * 3))

In that simulation, Full Jitter used the least total client work, Decorrelated Jitter was close, and both beat no jitter by a wide margin. Its conclusion: jittered backoff should be the standard approach for remote clients.

A retry decorator with full jitter

The range each retry can wait under full jitter, from 0 up to min(5 s, 0.2 s × 2^attempt)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
# retry_jitter.py — retry decorator with exponential backoff + full jitter
import functools
import random
import time
 
 
class RetryableError(Exception):
    """Raise this for failures that are safe to try again."""
 
    def __init__(self, message, retry_after=None):
        super().__init__(message)
        self.retry_after = retry_after  # seconds, e.g. from a Retry-After header
 
 
def retry(max_attempts=4, base=0.2, cap=5.0, retry_on=(RetryableError,)):
    """Retry with 'full jitter': sleep = random(0, min(cap, base * 2**attempt))."""
    def decorator(fn):
        @functools.wraps(fn)
        def wrapper(*args, **kwargs):
            for attempt in range(max_attempts):
                try:
                    return fn(*args, **kwargs)
                except retry_on as exc:
                    if attempt == max_attempts - 1:
                        raise                      # out of attempts: give up
                    sleep = random.uniform(0, min(cap, base * 2 ** attempt))
                    hint = getattr(exc, "retry_after", None)
                    if hint is not None:           # the server knows best
                        sleep = max(sleep, hint)
                    print(f"attempt {attempt + 1} failed: {exc}; sleep {sleep:.2f}s")
                    time.sleep(sleep)
        return wrapper
    return decorator
 
 
calls = {"n": 0}
 
@retry(max_attempts=5)
def flaky_call():
    calls["n"] += 1
    if calls["n"] < 3:
        raise RetryableError("503 Service Unavailable")
    return "ok"
 
 
if __name__ == "__main__":
    print(flaky_call(), "after", calls["n"], "calls")   # ok after 3 calls

Deciding what to retry

Turn Chapter 3's rules into one function, and raise RetryableError only when it says yes:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# classify.py — map an httpx failure to "retry or not"
import httpx
 
RETRY_STATUS = {408, 429, 500, 502, 503, 504}
IDEMPOTENT = {"GET", "HEAD", "OPTIONS", "PUT", "DELETE"}
 
 
def should_retry(method, exc=None, status=None, has_idempotency_key=False):
    safe_to_repeat = method in IDEMPOTENT or has_idempotency_key
    if isinstance(exc, (httpx.ConnectError, httpx.ConnectTimeout, httpx.PoolTimeout)):
        return True                         # request never reached the server
    if isinstance(exc, (httpx.ReadTimeout, httpx.RemoteProtocolError)):
        return safe_to_repeat               # it may have run
    return status in RETRY_STATUS and safe_to_repeat
 
 
print(should_retry("POST", exc=httpx.ReadTimeout("slow")))                            # False
print(should_retry("POST", exc=httpx.ReadTimeout("slow"), has_idempotency_key=True))  # True
print(should_retry("POST", exc=httpx.ConnectError("refused")))                       # True
print(should_retry("GET", status=503), should_retry("GET", status=404))              # True False

Trick: httpx.MockTransport(handler) lets you fake a server in tests, so you can check retry logic offline (see Dev Essentials 15 · Testing).

In real code, use a tested library. tenacity has wait_random_exponential, which its docs say matches the "Full Jitter" algorithm:

1
2
3
4
5
6
7
8
from tenacity import retry, retry_if_exception_type, stop_after_attempt, wait_random_exponential
 
@retry(wait=wait_random_exponential(multiplier=0.2, max=5),
       stop=stop_after_attempt(4),
       retry=retry_if_exception_type(ConnectionError))
def call_service():
    ...

Retry budgets

Retries multiply. Google's Site Reliability Engineering book ("Addressing Cascading Failures") gives the example of three layers that each make 4 attempts: one user action becomes 4 × 4 × 4 = 64 attempts on the database — while it is already overloaded. So pick one layer to own retries. The chapter "Handling Overload" adds two limits:

  • a per-request budget of up to three attempts, then the error goes up to the caller;
  • a per-client budget: retry only while retries are below 10% of requests.

And stop retrying when the deadline is near.

Common pitfall: retrying POST with the urllib3 recipe from the requests docs, which sets allowed_methods={'POST'}. That is only correct if the endpoint is idempotent or uses idempotency keys.

Chapter 5 — Idempotency keys

An idempotency key turns a risky POST into a safe retry. The client makes a unique ID (a UUID v4) for one logical operation and sends it on the first try and every retry. The server remembers the key and the result. A repeat gets the saved result instead of running again.

How Stripe does it

Stripe's API docs describe a widely copied design:

  • Header: Idempotency-Key. All POST requests accept it. GET and DELETE do not need it — they are idempotent already.
  • Keys are up to 255 characters; Stripe suggests V4 UUIDs. Do not put personal data in a key.
  • Stripe saves the status code and body of the first request that started executing — success or failure, even 500.
  • If parameters differ from the original, Stripe returns an error.
  • Requests that fail validation, or that clash with a request still running, are not saved, so you can retry them.
  • Keys can be removed once they are at least 24 hours old; a reused key after that is a new request.

The IETF draft

The IETF HTTPAPI working group wrote "The Idempotency-Key HTTP Header Field" (draft-ietf-httpapi-idempotency-key-header). As of September 2026, the latest version is -07 (October 2025) and the Datatracker lists it as an expired Internet-Draft — not an RFC. It is still a useful reference. It suggests:

Situation Status
Key missing on an endpoint that requires it 400 Bad Request
Same key, different request payload 422 Unprocessable Content
Same key while the first request is still running 409 Conflict

It also recommends a UUID or a similar random value, and says the server should publish its key expiry policy.

Server design

  1. Scope the key to the caller: store client_id + key, so two customers cannot collide.
  2. Store a fingerprint (hash of method, path and body) to catch reuse with a different payload.
  3. Mark the key in progress before running the handler. A second request with the same key gets 409.
  4. Save the final status and body. Replays return the saved response.
  5. Expire keys after a TTL (for example 24 hours).
  6. The "insert if absent" in step 3 must be atomic. In one process a dict is enough. With many servers, use Redis SET key value NX EX 86400 or a unique index in a database (see Dev Essentials 13 · Caching for Redis).

A FastAPI middleware (in-memory)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
# idem_app.py — Idempotency-Key middleware for FastAPI (in-memory, one process)
# pip install fastapi uvicorn httpx
import hashlib
import itertools
import time
 
from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse, Response
 
app = FastAPI()
TTL_SECONDS = 24 * 3600
STORE: dict[str, dict] = {}   # key -> {"fingerprint", "status", "body", "headers", "expires"}
order_ids = itertools.count(1)
 
 
@app.middleware("http")
async def idempotency(request: Request, call_next):
    if request.method not in ("POST", "PATCH"):
        return await call_next(request)            # GET/PUT/DELETE: idempotent already
 
    key = request.headers.get("Idempotency-Key")
    if not key:
        return JSONResponse({"error": "Idempotency-Key header required"}, status_code=400)
 
    body = await request.body()
    fingerprint = hashlib.sha256(request.method.encode() + request.url.path.encode() + body).hexdigest()
    client_id = request.headers.get("Authorization", "anonymous")   # scope key to caller
    store_key = f"{client_id}:{key}"
 
    now = time.time()
    for k in [k for k, v in STORE.items() if v["expires"] < now]:
        del STORE[k]                               # drop expired keys (TTL)
 
    saved = STORE.get(store_key)
    if saved:
        if saved["fingerprint"] != fingerprint:
            return JSONResponse({"error": "key reused with a different request"}, status_code=422)
        if saved["status"] is None:
            return JSONResponse({"error": "original request still in progress"}, status_code=409)
        return Response(saved["body"], status_code=saved["status"],
                        headers={**saved["headers"], "Idempotent-Replayed": "true"})
 
    STORE[store_key] = {"fingerprint": fingerprint, "status": None, "expires": now + TTL_SECONDS}
    try:
        response = await call_next(request)
    except Exception:
        del STORE[store_key]                       # crashed: let the client retry
        raise
    body_out = b"".join([chunk async for chunk in response.body_iterator])
    headers = {"content-type": response.headers.get("content-type", "application/json")}
    STORE[store_key].update(status=response.status_code, body=body_out, headers=headers)
    return Response(body_out, status_code=response.status_code, headers=headers)
 
 
@app.post("/orders", status_code=201)
async def create_order(order: dict):
    return {"order_id": next(order_ids), **order}
 
 
if __name__ == "__main__":
    from fastapi.testclient import TestClient
    c = TestClient(app)
    h = {"Idempotency-Key": "7f1c2a4e-1b7d-4a55-9d7e-3b1f0f7c9a11"}
    for body, headers in [({"sku": "BOOK-1", "qty": 1}, h),    # first try
                          ({"sku": "BOOK-1", "qty": 1}, h),    # a retry
                          ({"sku": "BOOK-1", "qty": 9}, h),    # same key, new body
                          ({"sku": "BOOK-1", "qty": 1}, {})]:  # no key
        r = c.post("/orders", json=body, headers=headers)
        print(r.status_code, r.json(), r.headers.get("Idempotent-Replayed"))

Output (tested with FastAPI 0.115 and Starlette 0.41):

1
2
3
4
5
201 {'order_id': 1, 'sku': 'BOOK-1', 'qty': 1} None
201 {'order_id': 1, 'sku': 'BOOK-1', 'qty': 1} true
422 {'error': 'key reused with a different request'} None
400 {'error': 'Idempotency-Key header required'} None

The retry got order_id: 1 again — no second order. A second request with the same key that arrives while the first is still running gets 409, because the key is stored as "in progress" before the handler runs.

Flowchart of the Idempotency-Key middleware: pass through, 400, 422, 409, replay the saved reply, or run and save a new reply

Watch out: this store lives in one Python process. It is lost on restart and not shared between workers (uvicorn --workers 4 means four separate dicts). In production use Redis or a database table with a unique key. Also note one design choice: this code saves 5xx replies too (like Stripe). Some teams prefer to delete the key on 5xx so the client can retry. Pick one and document it.

Trick: the client must create the key once per logical action (for example when the user clicks "Pay"), store it, and reuse it for every retry. A new key per retry defeats the purpose.

Chapter 6 — Rate limiting

A rate limit protects a server from too much traffic — from bugs, abuse, or one noisy customer. Four common algorithms:

Algorithm How it works Good at Weak at
Token bucket Bucket holds up to capacity tokens, refills at rate/s. Each request takes one Allows short bursts, smooth long-run limit Needs care to share across servers
Leaky bucket Requests join a queue that drains at a fixed rate Very smooth output Bursts wait or are dropped
Fixed window Count per clock window (e.g. per minute) Simplest; one counter Up to 2× the limit around a window edge
Sliding window Count over the last N seconds (log or weighted counters) Fixes the edge problem More memory or maths

A token bucket

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
# token_bucket.py — a thread-safe token bucket
import threading
import time
 
 
class TokenBucket:
    def __init__(self, rate: float, capacity: float):
        self.rate = rate              # tokens added per second (long-run limit)
        self.capacity = capacity      # bucket size (largest burst)
        self.tokens = capacity        # start full
        self.updated = time.monotonic()
        self.lock = threading.Lock()
 
    def try_acquire(self, cost: float = 1.0) -> tuple[bool, float]:
        """Return (allowed, seconds_until_enough_tokens)."""
        with self.lock:
            now = time.monotonic()
            self.tokens = min(self.capacity, self.tokens + (now - self.updated) * self.rate)
            self.updated = now
            if self.tokens >= cost:
                self.tokens -= cost
                return True, 0.0
            return False, (cost - self.tokens) / self.rate
 
 
if __name__ == "__main__":
    bucket = TokenBucket(rate=2, capacity=5)    # 2 requests/s, bursts of 5
    for i in range(8):
        ok, wait = bucket.try_acquire()
        print(i, "allowed" if ok else f"rejected, retry in {wait:.2f}s")
    time.sleep(1.0)                             # 1 s later: about 2 new tokens
    print("after 1s:", bucket.try_acquire())

Requests 0-4 pass (the burst), 5-7 are rejected with about 0.5 s to wait, and after one second a request passes again. Use time.monotonic(), not time.time() — the wall clock can jump.

429 and Retry-After

429 Too Many Requests is defined in RFC 6585. The response MAY include Retry-After. RFC 9110 defines Retry-After as either an HTTP date or a number of seconds.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# rate_limited_app.py — one bucket per API key (TokenBucket from above)
import math
from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse
 
app = FastAPI()
buckets: dict[str, TokenBucket] = {}
 
@app.middleware("http")
async def rate_limit(request: Request, call_next):
    client = request.headers.get("X-API-Key") or (request.client.host if request.client else "?")
    allowed, wait = buckets.setdefault(client, TokenBucket(rate=1, capacity=3)).try_acquire()
    if not allowed:
        return JSONResponse({"error": "rate limit exceeded"}, status_code=429,
                            headers={"Retry-After": str(math.ceil(wait))})
    return await call_next(request)

Add any route and call it five times quickly with TestClient as alice: you get [200, 200, 200, 429, 429] with Retry-After: 1, while bob still gets 200 — each key has its own bucket.

The RateLimit header fields (draft)

Many APIs send their own X-RateLimit-* headers. The IETF HTTPAPI working group is standardising two fields in draft-ietf-httpapi-ratelimit-headers. The latest version is -11 (May 2026), still an active draft, so names may change:

1
2
3
RateLimit-Policy: "burst";q=100;w=60,"daily";q=1000;w=86400
RateLimit: "default";r=50;t=30

q is the quota, w the window in seconds, r the remaining quota, t the seconds until it resets. The draft says that if Retry-After is also present, Retry-After takes precedence.

Chapter 7 — Circuit breakers

When a dependency is down, each call waits for a timeout before failing. A circuit breaker notices repeated failures and fails fast for a while. The pattern was made popular by Michael Nygard's book Release It!; Martin Fowler's "CircuitBreaker" article (2014) is a clear short summary.

State Behaviour
Closed Normal. Calls go through. Count failures
Open Too many failures. Fail at once without calling. Wait reset_timeout
Half-open After the wait, let a trial call through. Success → closed. Failure → open again

A breaker and retries work together: retries handle short blips, the breaker stops retries from hammering a service that is clearly down. Exercise 4 builds a small one. Real breakers also add a lock, count failures over a time window, and often use a failure rate instead of a count.

Trick: when the breaker is open, return something useful if you can — a cached value, a default, or a clear "try again later" message — instead of a stack trace.

Chapter 8 — Queues, at-least-once delivery and the outbox

Load levelling

If work does not need an answer right now (send an email, resize an image), put it on a queue. The API replies 202 Accepted at once; workers take jobs at their own pace. A traffic spike makes the queue longer, not the servers slower. On AWS this is SQS (see AWS Course 05 · Serverless & Events).

At-least-once means duplicates

SQS standard queues keep copies of each message on several servers. The SQS docs say you may rarely receive a message again, and tell you to design your application to be idempotent. Many queues work this way. A worker can also crash after doing the work but before it deletes the message, so the message comes back.

So every consumer must be idempotent. The usual method: each message carries a unique ID; the consumer records processed IDs and skips any ID it has seen. Exercise 6 builds this with SQLite.

Watch out: the side effect and the "processed" record must be saved together, in one database transaction. If you save the effect, crash, and never save the ID, the redelivered message runs again.

Dead-letter queues

Some messages always fail (bad data, a bug). Without a limit, they loop forever. In SQS, a redrive policy sets maxReceiveCount: after that many receives, the message moves to the dead-letter queue. AWS advises setting it high enough to allow real retries, and giving the DLQ a longer retention period than the source queue. Put an alarm on DLQ depth — a message there means a person must look.

The outbox pattern

A classic bug: save an order to the database, then publish an "order created" event. If the process dies between the two steps, the event is lost. The outbox pattern writes the event into an outbox table in the same transaction as the order. A separate relay reads the outbox and publishes. The relay may publish twice, so consumers must still be idempotent. MongoDB Course 20 · Outbox Pattern builds this step by step.

Practice Exercises

Exercise 1 — Classify the failure

For each case, say "retry", "retry with idempotency key only", or "do not retry": (a) GET /users/7 → connect timeout; (b) POST /payments → read timeout; (c) PUT /users/7 → 503 with Retry-After: 2; (d) POST /orders → 422.

Solution

(a) Retry — the request never reached the server, and GET is safe anyway. (b) Retry only with the same idempotency key — the charge may have happened. (c) Retry after at least 2 seconds — PUT is idempotent and 503 is temporary. (d) Do not retry — the request itself is wrong; the same body will fail again.

Exercise 2 — Worst-case wait

With base = 0.2, cap = 5 and max_attempts = 5, what is the longest total sleep with full jitter?

Solution

There are 4 sleeps (after attempts 0-3). The upper bounds are min(5, 0.2 * 2**n) for n = 0..3: 0.2 + 0.4 + 0.8 + 1.6 = 3.0 s. The average is half of that, 1.5 s, because full jitter is uniform from 0.

Exercise 3 — Add a deadline

Change the retry decorator so it also takes deadline (seconds) and stops when the next sleep would pass it.

Solution

Add deadline=10.0 to the parameters, set end = time.monotonic() + deadline at the top of wrapper, and change the give-up test to:

1
2
3
if attempt == max_attempts - 1 or time.monotonic() + sleep > end:
    raise

Compute sleep before this test, so the check uses the real planned wait.

Exercise 4 — A small circuit breaker

Write a CircuitBreaker class with call(fn). It opens after failure_threshold failures in a row, fails fast with CircuitOpenError while open, and lets one trial call through after reset_timeout seconds.

Solution 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
import time
 
 
class CircuitOpenError(Exception):
    pass
 
 
class CircuitBreaker:
    def __init__(self, failure_threshold=3, reset_timeout=5.0):
        self.failure_threshold = failure_threshold
        self.reset_timeout = reset_timeout
        self.failures, self.state, self.opened_at = 0, "closed", 0.0
 
    def call(self, fn, *args, **kwargs):
        if self.state == "open":
            if time.monotonic() - self.opened_at < self.reset_timeout:
                raise CircuitOpenError("failing fast; dependency is down")
            self.state = "half-open"               # let one trial call through
        try:
            result = fn(*args, **kwargs)
        except Exception:
            self.failures += 1
            if self.state == "half-open" or self.failures >= self.failure_threshold:
                self.state, self.opened_at = "open", time.monotonic()
            raise
        self.state, self.failures = "closed", 0    # success resets everything
        return result
 
 
def down():
    raise ConnectionError("payment service down")
 
cb = CircuitBreaker(failure_threshold=3, reset_timeout=0.5)
for i in range(5):
    try:
        cb.call(down)
    except Exception as exc:
        print(i, type(exc).__name__, cb.state)
# 0-1 ConnectionError closed | 2 ConnectionError open | 3-4 CircuitOpenError open
time.sleep(0.6)
print(cb.call(lambda: "charged"), cb.state)        # charged closed

Exercise 5 — Fixed window edge

A fixed-window limiter allows 100 requests per minute. How many requests can a client get through in 2 seconds, and how?

Solution

Up to 200: 100 in the last second of one window (e.g. 12:00:59) and 100 in the first second of the next (12:01:00). A token bucket with capacity = 100 or a sliding window avoids this.

Exercise 6 — Idempotent consumer with a database

Rewrite handle() with SQLite so the balance update and the processed ID are saved in one transaction.

Solution 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
import sqlite3
 
db = sqlite3.connect(":memory:")
db.executescript("""
CREATE TABLE balance (user TEXT PRIMARY KEY, amount INTEGER);
CREATE TABLE processed (message_id TEXT PRIMARY KEY);
INSERT INTO balance VALUES ('alice', 0);
""")
 
def handle(msg):
    try:
        with db:  # one transaction: commit both or neither
            db.execute("INSERT INTO processed VALUES (?)", (msg["id"],))
            db.execute("UPDATE balance SET amount = amount + ? WHERE user = ?",
                       (msg["amount"], msg["user"]))
    except sqlite3.IntegrityError:
        print("skip duplicate", msg["id"])   # primary key clash = already done
 
for m in [{"id": "m-1", "user": "alice", "amount": 50},
          {"id": "m-1", "user": "alice", "amount": 50}]:
    handle(m)
print(db.execute("SELECT amount FROM balance").fetchone())   # (50,)

The unique key on message_id makes the duplicate check atomic.

Cheat-sheet

Topic Rule of thumb
Timeouts Every call. requests: timeout=(3.05, 10). httpx: httpx.Timeout(10, connect=3)
requests default No timeout at all — always pass one
httpx default 5 s
Safe to retry Connect errors; idempotent methods (GET, HEAD, OPTIONS, PUT, DELETE)
Not safe POST/PATCH after a read timeout — unless it has an idempotency key
Retryable statuses 408, 429, 502, 503, 504 (maybe 500)
Never retry 400, 401, 403, 404, 409, 422
Full jitter sleep = random(0, min(cap, base * 2 ** attempt))
Retry budget ≤ 3 attempts per request; retries < 10% of traffic; retry at one layer only
Idempotency key UUID v4 per logical action; same key on every retry
Server store client + key → fingerprint, status, body; TTL ~24 h; atomic insert
Draft status codes Missing key 400, different payload 422, still running 409
Rate limit reply 429 + Retry-After (seconds or HTTP date)
Token bucket rate = long-run limit, capacity = burst size
Circuit breaker Closed → Open (fail fast) → Half-open (one trial)
Queues At-least-once → idempotent consumers; DLQ via maxReceiveCount
Outbox Write event + data in one transaction; relay publishes later

What's next

Dev Essentials 15 · Testing Strategy covers unit, integration and end-to-end tests, mocks, and Testcontainers — including how to test code like the retry decorator and middleware above without real waits or real servers.

References

Note: this post is in progress. Planned additions: a Redis-backed idempotency store, a sliding-window limiter, hedged requests, and bulkheads (separate pools per dependency).