Many ideas you use every day started as a research paper. Consistent hashing, MapReduce and the Transformer all came from papers. Reading one front to back, word by word, is slow and tiring. There is a better way. This post teaches the three-pass method from S. Keshav's short 2007 paper. It also covers how to find papers, read figures and math, judge quality, and keep notes you can use again. It opens Part E, which is about skills outside code.

Series map: 01 Text Encoding → … → 19 RAG → 20 Reading Papers (this post) → 21 Estimating → … → 24 Desk Health. Part E · Growth outside code.

Learning Outcome

  • Explain the three passes and how long each one should take.
  • Answer the five Cs after a 5-10 minute first pass.
  • Run a small literature survey with Keshav's three steps.
  • Find papers with Google Scholar, arXiv, Semantic Scholar, the ACM Digital Library and dblp.
  • Read figures and tables first, and turn math notation into code.
  • Judge a paper: peer review, venue, preprint status, replication.
  • Keep notes with a copy-ready template.

Chapter 1 — What a paper looks like

Most computer science papers follow the same shape. Once you know the shape, you know where to look.

Part What it tells you Read in pass
Title + abstract The claim in one paragraph 1
Introduction The problem, why it matters, the contributions (often a bullet list) 1
Background / related work What came before, and how this paper differs 2 (skim in 1)
Design / method How the idea works 2, 3
Implementation What was built 2, 3
Evaluation Experiments, figures, tables 2
Discussion / lessons Limits, surprises 2
Conclusion The claim again, plus future work 1
References The paper's family tree 1 (glance)

Terminology

  • Peer review — other experts read the paper before it is accepted. They check that it is new, sound and clear. They do not re-run everything.
  • Venue — where the paper appeared: a conference (SOSP, OSDI, NeurIPS), a journal, or a workshop. In computer science, top conferences often matter more than journals.
  • Proceedings — the book (now a website) of all papers from one conference.
  • Preprint — a version posted before or without peer review, often on arXiv.
  • Artifact — the code, data or scripts behind a paper.
  • DOI — a permanent ID for a paper, like 10.1145/1273445.1273458.
  • Survey paper — a paper that summarises a whole area. Gold for a newcomer.

Chapter 2 — The three-pass method

The source is S. Keshav, "How to Read a Paper", ACM SIGCOMM Computer Communication Review 37(3), July 2007, pages 83-84. It is two pages long. Keshav has updated it over the years; later versions (for example, one dated February 17, 2016) keep the same method.

The key idea: read a paper in up to three passes, not once from start to end. Each pass has its own goal. You can stop after any pass.

First pass — the bird's-eye view (about 5-10 minutes)

Keshav's steps:

  1. Carefully read the title, abstract and introduction.
  2. Read the section and sub-section headings, and ignore everything else.
  3. Glance at the math (if any) to see what theory it rests on.
  4. Read the conclusions.
  5. Glance over the references and tick off the ones you have already read.

At the end, you should be able to answer the five Cs:

C Question
Category What type of paper is this? A measurement paper? An analysis of an existing system? A description of a research prototype?
Context Which other papers is it related to? What theory does it use?
Correctness Do the assumptions seem valid?
Contributions What are the main contributions?
Clarity Is it well written?

Now you decide. Maybe the paper does not interest you. Maybe you lack the background. Maybe the assumptions look wrong. Stopping here is fine. Keshav says the first pass is enough for papers outside your area that may matter some day.

Trick: the same logic works in reverse when you write. Most readers make only one pass. Clear headings and a tight abstract decide whether anyone reads further. (See Dev Essentials 22 · Design Docs.)

Second pass — the content (up to about an hour)

Read with more care, but skip details such as proofs. Write key points and questions in the margin as you go. Two special jobs:

  1. Look hard at the figures, diagrams and graphs. Are the axes labelled? Are there error bars, so the result is statistically meaningful? Keshav notes that mistakes like these separate rushed work from excellent work.
  2. Mark relevant references you have not read. They are your reading list for background.

After this pass you should be able to summarise the main idea, with evidence, to someone else. Keshav says this pass takes up to an hour for an experienced reader.

Sometimes you still will not understand the paper. The topic may be new, the terms unfamiliar, or the writing poor. You then have three choices: (a) put it aside, (b) come back after reading background material, or (c) push on to the third pass.

Third pass — virtual re-implementation (hours)

The key is to virtually re-implement the paper: make the same assumptions as the authors and re-create the work in your head (or on paper, or in code). Then compare your version with theirs. The gaps show you both the new ideas and the hidden weak points.

  • Challenge every assumption in every statement.
  • Ask how you would present each idea.
  • Write down ideas for future work.

Time: the 2007 version says about four or five hours for beginners and about an hour for an experienced reader. The 2016 version is less optimistic: many hours for beginners and more than an hour or two even for experienced readers.

At the end, you should be able to rebuild the paper's structure from memory, and point out implicit assumptions, missing citations and problems with the experiments or analysis.

The three passes at a glance

Pass Time (Keshav) You read You come out with
1 5-10 min Title, abstract, intro, headings, conclusions, reference list The five Cs; a go / no-go decision
2 Up to ~1 hour Whole paper, skip proofs; study figures A summary with evidence; a list of unread references
3 Hours Everything, in depth A mental re-implementation; strengths, weak points, future work

Keshav's three passes: time, what you read and do, and what you come out with, including the five Cs

Common pitfall: starting with pass 3 on page 1. You drown in details before you know what the paper claims. Always do pass 1 first, even on a paper you "must" read in depth.

Chapter 3 — A literature survey in three steps

Keshav's paper also shows how to survey a new field. You may need to read tens of papers.

  1. Seed. Use an academic search engine (Keshav names Google Scholar and CiteSeer) with good keywords. Find three to five recent, highly cited papers. Do one pass on each. Read their related-work sections. If one of them points to a recent survey paper, read that and you are nearly done.
  2. Find the core. Look for shared citations and repeated author names across the seed papers. These are the key papers and people. Visit the key researchers' pages and see where they publish. That tells you the top venues of the field.
  3. Scan the venues. Go to those conferences' recent proceedings. A quick scan finds recent strong work. These papers, plus the key papers from step 2, are the first version of your survey. Do two passes on them. If they all cite a key paper you missed, read it and repeat.

Step 2 is easy to automate once you have the reference lists. Paste a few references from each seed paper (short keys are fine):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# shared_citations.py
from collections import Counter
 
references = {
    "paper A": ["Dynamo 2007", "Bigtable 2006", "Chord 2001", "Paxos 1998"],
    "paper B": ["Dynamo 2007", "Cassandra 2010", "Chord 2001", "Raft 2014"],
    "paper C": ["Dynamo 2007", "Spanner 2012", "Paxos 1998", "Raft 2014"],
}
 
counts = Counter(ref for refs in references.values() for ref in set(refs))
for ref, n in counts.most_common():
    if n >= 2:
        print(f"{n} of {len(references)} seed papers cite: {ref}")
 
1
2
3
4
5
3 of 3 seed papers cite: Dynamo 2007
2 of 3 seed papers cite: Chord 2001
2 of 3 seed papers cite: Paxos 1998
2 of 3 seed papers cite: Raft 2014

The paper that every seed cites goes to the top of your reading list.

Chapter 4 — Finding papers (state in September 2026)

Tool Good for Notes
Google Scholar Broad search; "Cited by" to move forward in time; "All versions" to find a free copy Citation counts include non-peer-reviewed items
arXiv Preprints in CS, math, physics and more Moderated, not peer reviewed (see Chapter 7)
Semantic Scholar Search, citation graph, a free API Run by the Allen Institute for AI (Ai2)
ACM Digital Library ACM conferences and journals (SOSP, SIGCOMM, CHI, …) Fully open access since January 1, 2026
dblp A clean list of a researcher's papers, sorted by venue Great for Keshav's step 2
OpenReview Some ML conferences (for example ICLR) publish the reviews Reading reviews teaches you what experts question
Hugging Face Papers Trending ML papers, often with code links Papers with Code was shut down in July 2025; its domain now redirects here

To get a free copy legally, try, in order: the venue itself (many are open access, including USENIX papers), the arXiv version, the authors' home pages, your library, or a polite email to the author.

Searching arXiv from Python takes a few lines. The arXiv API returns Atom XML:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
"""arxiv_search.py - search arXiv from the terminal.  pip install requests"""
import sys
import xml.etree.ElementTree as ET
 
import requests
 
ATOM = "{http://www.w3.org/2005/Atom}"
API = "https://export.arxiv.org/api/query"
 
 
def search_arxiv(query, max_results=5):
    resp = requests.get(
        API,
        params={"search_query": query, "start": 0,
                "max_results": max_results, "sortBy": "relevance"},
        headers={"User-Agent": "paper-notes/0.1"},
        timeout=30,
    )
    resp.raise_for_status()
    root = ET.fromstring(resp.content)
    for entry in root.findall(ATOM + "entry"):
        title = " ".join(entry.find(ATOM + "title").text.split())
        published = entry.find(ATOM + "published").text[:10]
        link = entry.find(ATOM + "id").text
        authors = [a.find(ATOM + "name").text for a in entry.findall(ATOM + "author")]
        first = authors[0] + (" et al." if len(authors) > 1 else "")
        print(f"{published}  {first}\n  {title}\n  {link}\n")
 
 
if __name__ == "__main__":
    q = sys.argv[1] if len(sys.argv) > 1 else 'ti:"attention is all you need"'
    search_arxiv(q)
 
1
2
3
python arxiv_search.py
python arxiv_search.py 'ti:mapreduce AND cat:cs.DC'
 
1
2
3
python arxiv_search.py
python arxiv_search.py 'ti:mapreduce AND cat:cs.DC'

First result of the default query: 2017-06-12 Ashish Vaswani et al. — Attention Is All You Need — http://arxiv.org/abs/1706.03762v7. Field prefixes: ti: title, au: author, abs: abstract, cat: category, all: everything.

Watch out: the arXiv API manual asks you to wait about 3 seconds between calls. If you loop over many queries, add time.sleep(3).

Chapter 5 — Read figures and tables first

In pass 2, the figures often tell the story faster than the text. Use this checklist on every graph:

  • Axes: labelled? Units? Linear or log scale? Does the y-axis start at zero, or is a small gap made to look big?
  • What is compared: which baseline? Is it a fair, well-tuned one, or a weak straw man?
  • Spread: error bars, confidence intervals, or at least several runs? One run proves little.
  • Sample: how many machines, users, datasets or trials?
  • Range: does the plot stop right before things get interesting?
  • Caption: can you understand the figure from the caption alone?

For tables:

  • Bold usually marks the best number. Check how big the gap to second place is, and whether it is within noise.
  • An ablation table removes one part at a time. It shows which parts of the idea actually matter.
  • Look for missing rows: a dataset or setting the method might lose on.

Trick: before reading the evaluation text, write one sentence per figure: "This shows that ___." Then read the text and compare. Your version and the authors' version will not always match.

Chapter 6 — Handling math notation

Math is a compact language. You do not need to follow every line on pass 2. Some habits help:

  1. Build a notation table as you read. Each symbol, its meaning, and its shape or unit.
  2. Read the words around the equation. Authors usually say in plain words what it does.
  3. Plug in small numbers. Two items, three dimensions. Work it out by hand.
  4. Skip proofs on pass 2. Trust the statement first; check it in pass 3.
  5. Turn it into code. Code forces you to decide every shape.

Common symbols:

Symbol Usually means
Σ, ∏ Sum, product over a range
∈, ∀, ∃ "is in", "for all", "there exists"
‖x‖ Length (norm) of a vector
argmax f(x) The x that makes f(x) biggest
E[X] Expected (average) value
x ~ D x is drawn from distribution D
:= "is defined as"
∝, ≈ Proportional to, approximately equal
O(n log n) Grows no faster than n log n
Xᵀ Transpose of matrix X

Example: the core formula of "Attention Is All You Need" (Section 3.2.1) is

1
2
Attention(Q, K, V) = softmax( Q Kᵀ / sqrt(d_k) ) V

Notation table first: Q is queries (one row per query), K is keys, V is values, d_k is the key size, and softmax turns each row of scores into weights that sum to 1. Now code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
import numpy as np
 
 
def softmax(x, axis=-1):
    x = x - x.max(axis=axis, keepdims=True)   # subtract max for numerical safety
    e = np.exp(x)
    return e / e.sum(axis=axis, keepdims=True)
 
 
def attention(Q, K, V):
    d_k = K.shape[-1]
    scores = Q @ K.T / np.sqrt(d_k)          # (n_queries, n_keys)
    weights = softmax(scores, axis=-1)       # each row sums to 1
    return weights @ V, weights
 
 
rng = np.random.default_rng(0)
Q = rng.normal(size=(2, 4))   # 2 queries, d_k = 4
K = rng.normal(size=(3, 4))   # 3 keys,    d_k = 4
V = rng.normal(size=(3, 5))   # 3 values,  d_v = 5
 
out, w = attention(Q, K, V)
print(out.shape)              # (2, 5)
print(w.sum(axis=1))          # [1. 1.]

Writing this answers questions the text leaves open: K must have the same width as Q, and the output has one row per query. (Dev Essentials 17 · How LLMs Work explains what attention does in a model.)

Chapter 7 — Judging quality

A paper is a claim with evidence, not a fact. Weigh it.

  • Peer review is a filter, not a proof. Reviewers check that the work is new, sound and clear. They rarely re-run the experiments.
  • Venue. A paper at a top venue passed a hard filter. A workshop paper is often early work. Neither guarantees the result holds.
  • Preprints. arXiv states that its "moderation process is not a peer-review process". A preprint may be excellent, or wrong. Check whether a later peer-reviewed version exists.
  • Replication. Can others get the same result? Many CS conferences now run artifact evaluation. ACM badges mark this: Artifacts Available (code is public), Results Reproduced (another team got the main results using the authors' artifacts) and Results Replicated (another team got them without the authors' artifacts).
  • Retractions. The Retraction Watch database has been free and open since Crossref acquired it in September 2023. Check it for papers you plan to rely on.
  • Citation counts show attention, not truth. They also lag: a strong new paper can have few citations.
  • Industry papers (such as Dynamo and MapReduce) describe systems at a scale you cannot rebuild. Judge them by the data they show and by what later systems learned.

Red flags: vague baselines, a single run, no limitations section, claims in the abstract that the evaluation does not support, and code "available on request".

Chapter 8 — A paper-notes template

Copy this into your notes tool. One file per paper.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
# <Paper title>
- Authors / year / venue:
- Link (DOI or arXiv):
- Date read:            Passes done: 1 / 2 / 3
- Why read it now:
 
## Five Cs (after pass 1)
- Category:
- Context (related papers, theory):
- Correctness (do assumptions look valid?):
- Contributions (1-3 bullets):
- Clarity (1-5):
 
## Summary (after pass 2, max 5 sentences)
 
## Key figures / tables
- Fig/Table N: "This shows that ..."
 
## Notation table
| Symbol | Meaning | Shape / unit |
|--------|---------|--------------|
 
## Assumptions and weak points
 
## References to read next
- [ ]
 
## Ideas / how this applies to current work

Trick: keep the notes in a git repository as Markdown. Search across all your notes later with grep or Select-String (see Dev Essentials 05 · Command Line).

Chapter 9 — Three practice papers for this blog

These three connect to other series here. All are free to read.

Paper Venue Why read it Connects to
DeCandia et al., "Dynamo: Amazon's Highly Available Key-value Store" SOSP 2007 Consistent hashing, quorums, vector clocks, hinted handoff: a tour of distributed-systems ideas MongoDB Course (replication, sharding), AWS Course
Dean and Ghemawat, "MapReduce: Simplified Data Processing on Large Clusters" OSDI 2004 A clean idea, a clear system, honest experience section Python Course, MongoDB Course (aggregation)
Vaswani et al., "Attention Is All You Need" NIPS 2017 (now called NeurIPS) The Transformer; base of today's LLMs Dev Essentials 17 · How LLMs Work, 19 · RAG

First-pass hints:

  • Dynamo. Read the intro, then find the table early in the architecture section that pairs each problem with the technique used. That table is the paper in miniature. Note: Amazon DynamoDB is a later, different system. Its own paper (USENIX ATC 2022) explains how its design differs from Dynamo.
  • MapReduce. Start from the execution-overview figure (Figure 1). Then read the fault-tolerance part of the implementation section.
  • Attention. Look at the model-architecture figure (Figure 1) and the results table that compares quality and training cost. Keep Chapter 6's notation table next to you.

Practice Exercises

Exercise 1 — A timed first pass

Pick one of the three papers. Set a 10-minute timer. Do only Keshav's five first-pass steps. Stop when the timer rings.

Solution

A good first pass reads the title, abstract and intro in full, only the headings of the body, glances at any math, reads the conclusion and scans the references. If you read a full section of the body, you went too deep. The output is a go / no-go decision, not understanding.

Exercise 2 — Write the five Cs

Using the template in Chapter 8, write the five Cs for the paper from Exercise 1. One or two lines each.

Solution

Example for MapReduce:

  • Category: description of a working system, with a performance evaluation and experience report.
  • Context: parallel and distributed processing; the paper compares itself with earlier parallel programming systems in its related-work section.
  • Correctness: assumes cheap commodity machines that fail often, and tasks that can be split into independent map and reduce steps. Reasonable for batch jobs at Google.
  • Contributions: a simple programming model; a runtime that handles splitting, scheduling and machine failures; lessons from real use.
  • Clarity: high; short sections and one clear overview figure.

Exercise 3 — Critique a figure

Take any results figure from your chosen paper. Answer the Chapter 5 checklist for it.

Solution

Answer each point in writing: axes and units, scale, baseline, spread (error bars or number of runs), sample size, range, and whether the caption stands alone. A useful last line: "The figure supports / partly supports / does not support the claim that ___."

Exercise 4 — Notation to code

Write a notation table for the attention formula, then change the code in Chapter 6 so Q has 3 queries. Predict the output shape before running.

Solution

Q = rng.normal(size=(3, 4)). The scores are (3, 3) (3 queries by 3 keys), the weights are (3, 3), and the output is (3, 5): one row per query, d_v columns. w.sum(axis=1) prints three ones.

Exercise 5 — Mini survey

Pick a topic (for example "vector databases"). Find three recent, highly cited papers. Put 5-10 references from each into shared_citations.py.

Solution

The references cited by two or more seeds are your core reading list. Look up their authors on dblp to see which venues come up again and again. Those are the venues to scan in Keshav's step 3.

Exercise 6 — Judge a preprint

Find an arXiv paper in your area from the last year. Is there a peer-reviewed version? Is the code public? Has anyone reproduced it?

Solution

Check the arXiv "Comments" and "Journal reference" fields, then search the title on Google Scholar or Semantic Scholar for a venue version. Look for a code link and for later papers that report re-running it. If none exist, treat the claims as unconfirmed.

Cheat-sheet

Task Do this
Decide if a paper is worth it Pass 1, 5-10 min: title, abstract, intro, headings, conclusions, references
After pass 1 Write the five Cs: category, context, correctness, contributions, clarity
Understand the content Pass 2, up to ~1 h: skip proofs, study figures, mark unread references
Understand in depth Pass 3, hours: virtually re-implement; challenge every assumption
Survey a field 3-5 seed papers → shared citations and authors → top venues' proceedings
Search Google Scholar, arXiv, Semantic Scholar, ACM DL, dblp, OpenReview
Read a graph Axes, scale, baseline, error bars, sample size, range
Read math Notation table, small numbers, then code
Judge quality Venue, peer review vs preprint, artifacts, replication, retractions

What's next

Dev Essentials 21 · Estimating covers how to estimate software tasks: breaking work down, giving ranges instead of single numbers, and why estimates slip.

References

Note: this post is in progress. Planned additions: a worked second pass on the Dynamo paper, a Semantic Scholar API example for citation graphs, and tips for reading papers outside computer science.