A large language model (LLM) reads text as tokens and predicts the next token, again and again. That one idea explains most of the surprises developers meet: why a prompt costs what it costs, why a long chat "forgets" how it started, why the same question gets two different answers, and why a model can state a false fact with full confidence. This post builds the mental model step by step: tokens, the transformer, training, the context window, sampling and embeddings. Then it covers hallucinations and three API features you will meet everywhere. It opens Part D, and the next two posts build on it. The ideas apply to every vendor; the runnable examples use Python, NumPy, sentence-transformers and the Anthropic Python SDK.

Series map: 01 Text Encoding → … → 16 CI/CD → 17 How LLMs Work (this post) → 18 Coding with AI Agents → … → 24 Desk Health. Part D · AI tools.

Learning Outcome

Explain what a token is, why tokens are not words, and why code and non-English text often cost more.
Train a toy BPE tokenizer, and count real tokens with Anthropic's token-counting endpoint.
Describe next-token prediction and attention at an intuition level.
Name the training stages: pretraining, instruction tuning, and preference training (RLHF, AI feedback).
Explain the context window, what fills it, and what happens when a request is too big.
Use temperature, top-k and top-p correctly, and know which current models still accept them.
Turn text into embeddings and compare them with cosine similarity.
Explain why hallucinations happen, and reduce them.
Recognise prompt caching, tool use and extended thinking in an API.

Chapter 1 — Tokens and tokenizers

Setup

You need Python 3.10 or newer (the anthropic 1.x SDK requires it) and, for the API examples, a key from the Anthropic Console.

1
2
3
4
5 # Git Bash (Windows) / macOS / Linux
python -m venv .venv && source .venv/Scripts/activate   # macOS/Linux: .venv/bin/activate
pip install anthropic numpy sentence-transformers
export ANTHROPIC_API_KEY="sk-ant-..."

1
2
3
4
5 # Windows PowerShell 5.1
python -m venv .venv; .\.venv\Scripts\Activate.ps1
pip install anthropic numpy sentence-transformers
$env:ANTHROPIC_API_KEY = "sk-ant-..."

Terminology

Term	Meaning
Token	One chunk of text from the model's fixed list: a word, part of a word, a space plus a word, a symbol, or a byte.
Vocabulary	The fixed list of all tokens a model knows. Each token has an integer ID.
Tokenizer	The program that turns text into token IDs and back. It is fixed before training.
BPE	Byte Pair Encoding: build a vocabulary by merging the most frequent neighbouring pieces, over and over.

Tokens are not words

The model never sees letters or words. It sees a list of integers. A common word may be one token; a rare word may be three or four. Spaces, punctuation and line breaks use tokens too.

A rule of thumb from Anthropic's models page (vendor spec, September 2026): on the tokenizer Claude has used since Claude Opus 4.7, 1M tokens is roughly 555k English words, or 2.5M Unicode characters. Earlier models fit about 750k words in 1M tokens. So the same text gives a different count on a different model.

Tokens matter because almost every limit uses them: price (per million input and output tokens), the context window (Chapter 4), max_tokens (the cap on what the model writes) and rate limits.

BPE in one example

Byte Pair Encoding was brought into neural machine translation by Sennrich, Haddow and Birch (ACL 2016) to handle rare words as sequences of smaller units. Training: split every word into characters, count every neighbouring pair, merge the most frequent pair, repeat. Tokenizing: replay the merges in the same order. Many modern tokenizers run the same idea on UTF-8 bytes, so any text can be encoded (see Dev Essentials 01 · Text Encoding).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41 # bpe_toy.py
from collections import Counter
 
def merge_pair(symbols, pair):
    out, i = [], 0
    while i < len(symbols):
        if i < len(symbols) - 1 and (symbols[i], symbols[i + 1]) == pair:
            out.append(symbols[i] + symbols[i + 1])
            i += 2
        else:
            out.append(symbols[i])
            i += 1
    return tuple(out)
 
def train_bpe(corpus, num_merges):
    words = Counter(tuple(w) + ("</w>",) for w in corpus.split())   # chars + end marker
    merges = []
    for _ in range(num_merges):
        pairs = Counter()
        for word, freq in words.items():
            for a, b in zip(word, word[1:]):
                pairs[(a, b)] += freq
        if not pairs:
            break
        best = max(pairs, key=pairs.get)                              # most frequent pair
        merges.append(best)
        words = Counter({merge_pair(w, best): f for w, f in words.items()})
    return merges
 
def tokenize(word, merges):
    symbols = tuple(word) + ("</w>",)
    for pair in merges:
        symbols = merge_pair(symbols, pair)
    return list(symbols)
 
corpus = "low low low low lower lower newest newest newest widest widest low lowest"
merges = train_bpe(corpus, num_merges=8)
print("merges:", merges)
for w in ["low", "lowest", "newer", "slowly"]:
    print(f"{w:8} -> {tokenize(w, merges)}")

Output:

1
2
3
4
5
6 merges: [('l', 'o'), ('lo', 'w'), ('e', 's'), ('es', 't'), ('est', '</w>'), ('low', '</w>'), ('n', 'e'), ('ne', 'w')]
low      -> ['low</w>']
lowest   -> ['low', 'est</w>']
newer    -> ['new', 'e', 'r', '</w>']
slowly   -> ['s', 'low', 'l', 'y', '</w>']

low is common, so it becomes one token. newer never appeared, so it breaks into known pieces plus single letters. Common text becomes few tokens. Rare text becomes many.

Why code and non-English text often cost more

Merges are learned from the tokenizer's training data. Whatever is common there gets long tokens.

Non-English text. Petrov et al. (NeurIPS 2023) tokenized the same text translated into many languages and found lengths that differ by up to 15 times in some cases (independent measurement). A Chinese character is also 3 bytes in UTF-8, so a byte-level tokenizer starts from more pieces.
Code. The Codex paper (Chen et al., 2021) found the GPT-3 text tokenizer "not very effective for representing code", with whitespace the largest waste. Adding tokens for runs of spaces cut code token counts by about 30% (vendor research paper).

Measure your own data instead of trusting a rule of thumb. Anthropic's token-counting endpoint is free (rate-limited by usage tier) and returns an estimate for the exact model you name:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16 # count_tokens.py
import anthropic
 
client = anthropic.Anthropic()          # reads ANTHROPIC_API_KEY
MODEL = "claude-opus-5-5"
samples = {
    "English prose": "The quick brown fox jumps over the lazy dog. " * 20,
    "Python code": "def add(a, b):\n    return a + b\n\n" * 20,
    "Chinese": "敏捷的棕色狐狸跳過了懶狗。" * 20,
}
for name, text in samples.items():
    tokens = client.messages.count_tokens(
        model=MODEL, messages=[{"role": "user", "content": text}]
    ).input_tokens
    print(f"{name:14} chars={len(text):5}  tokens={tokens:5}  chars/token={len(text) / tokens:.2f}")

A higher chars/token means that kind of text is packed more tightly. Each count includes a few tokens of message framing, which is why the samples repeat 20 times. For scale, the docs show a tiny request (system "You are a scientist", message "Hello, Claude") counting as {"input_tokens": 14} on claude-opus-5-5.

Trick: count with both model IDs before you switch models. Anthropic's docs say Claude 4.7 and later models use a newer tokenizer that produces about 30% more tokens for the same text (vendor spec). Budgets and cost estimates move with it.

Chapter 2 — Next-token prediction and the transformer

Every LLM answer comes from the same loop:

Turn the prompt into token IDs.
Run them through the network. It returns one score (a logit) per vocabulary token.
Turn the scores into probabilities and pick one token (Chapter 5).
Append it and repeat. Stop at an end token, a stop sequence or max_tokens.

A toy model that only looks at the previous word shows the loop. It "trains" by counting:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21 # bigram_toy.py
import random
from collections import Counter, defaultdict
 
tokens = "the cat sat on the mat . the dog sat on the rug . the cat ate the fish .".split()
follow = defaultdict(Counter)
for a, b in zip(tokens, tokens[1:]):
    follow[a][b] += 1                         # "training": count what follows what
 
def next_token_probs(prev):
    total = sum(follow[prev].values())
    return {tok: n / total for tok, n in follow[prev].items()}
 
print(next_token_probs("the"))
random.seed(1)
out = ["the"]
for _ in range(7):                            # "generation": predict, sample, append
    probs = next_token_probs(out[-1])
    out.append(random.choices(list(probs), weights=list(probs.values()))[0])
print(" ".join(out))

Output:

1
2
3 {'cat': 0.3333333333333333, 'mat': 0.16666666666666666, 'dog': 0.16666666666666666, 'rug': 0.16666666666666666, 'fish': 0.16666666666666666}
the cat ate the cat sat on the

A real LLM looks at all earlier tokens, its probabilities come from billions of learned parameters, and it works on subword tokens. The loop is the same.

Attention, at intuition level

Almost every modern LLM is a transformer, from "Attention Is All You Need" (Vaswani et al., 2017). Its key part is self-attention: for each token, the model decides how much each earlier token matters right now. In "The animal didn't cross the street because it was tired", attention lets "it" pull in information from "animal". Each token makes a query (what am I looking for?), a key (what do I contain?) and a value (what do I pass on?):

$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$

For text generation, a causal mask stops each token from looking at later tokens. Real models stack many such layers, each with several attention "heads".

What this means for you:

Output is written one token at a time, which is why APIs offer streaming. On Anthropic's current price list, output tokens cost five times as much as input tokens.
The model cannot go back. An early wrong token shapes everything after it. Reasoning before answering helps (Chapter 8).
Long inputs cost more compute. The paper lists self-attention cost per layer as $O(n^2 \cdot d)$ for sequence length $n$ .

Chapter 3 — Training stages (high level)

Term	Meaning
Pretraining	Next-token prediction on a huge text corpus. Produces a base model.
Instruction tuning (SFT)	Supervised fine-tuning on example requests with good answers.
Reward model	A model trained to score answers, from human rankings.
RLHF / RLAIF	Reinforcement learning from human / AI feedback.

This is deliberately high level: labs combine and change these steps, and full recipes are not published.

Pretraining. The model learns to predict the next token over a very large text collection. It picks up grammar, facts, code patterns and reasoning habits, but does not reliably follow instructions yet.
Instruction tuning. Fine-tuning on demonstrations: a request, then a good response.
Preference training. InstructGPT (Ouyang et al., 2022) describes classic RLHF: supervised fine-tuning on labeller demonstrations, then a reward model trained on rankings of model outputs, then reinforcement learning against it. Constitutional AI (Bai et al., 2022) replaces much human labelling for harmlessness: the model critiques and revises its own answers against written principles, and a preference model trained on AI comparisons gives the reward (RLAIF).

In practice:

Knowledge has a cutoff. Anthropic lists a "reliable knowledge cutoff" per model (vendor spec): for example Feb 2025 for Claude Haiku 4.5 and Jun 2026 for Claude Opus 5.5.
The model knows nothing about your private code or data. You supply it through context: prompts, files, retrieved documents (post 19), tool results (post 18).

Chapter 4 — The context window

The context window is everything the model can read while writing a response, including the response itself. Anthropic's docs call it the model's "working memory". It counts the system prompt, every message (including tool results, images and documents), tool definitions, and this turn's output including any thinking.

The API is stateless. The model does not remember your last request, so a chat app resends the whole history every turn. Turn 10 pays for turns 1 to 9 again (Exercise 4 does the maths).

Sizes change often. As of September 2026 (vendor spec), current Claude Opus, Sonnet and Fable models have a 1M-token window and up to 128k output tokens per request; Claude Haiku 4.5 has 200k and 64k. When a request does not fit, the Anthropic API does this (vendor spec):

Situation	Result
Input alone is bigger than the window	`400 invalid_request_error` ("prompt is too long")
Input fits, but input + `max_tokens` does not (Claude 4.5 and newer)	Accepted; if generation hits the limit it stops with `stop_reason: "model_context_window_exceeded"`
Chat apps such as claude.ai	May drop the oldest turns ("first in, first out")

Other vendors differ, and some truncate silently. Check your API's docs.

Watch out: bigger is not automatically better. Anthropic's docs describe context rot: as the token count grows, accuracy and recall degrade. Choose what goes in with care.

Ways to stay inside the window: trim the oldest turns; summarise them (Anthropic's server-side version is called "compaction"); retrieve only the relevant chunks (post 19); or start a fresh session from a short written summary (post 18). A trimming helper using the free counting endpoint:

1
2
3
4
5
6
7
8
9 def trim_to_budget(client, model, system, messages, budget):
    """Drop the oldest user+assistant pair until the request fits `budget` input tokens."""
    msgs = list(messages)
    while True:
        used = client.messages.count_tokens(model=model, system=system, messages=msgs).input_tokens
        if used <= budget or len(msgs) <= 1:
            return msgs, used
        msgs = msgs[2:]            # the list still starts with a user turn

Chapter 5 — Sampling: temperature, top-k and top-p

Softmax turns logits $z_i$ into probabilities. Temperature $T$ divides the logits first:

$p_i = \frac{e^{z_i / T}}{\sum_j e^{z_j / T}}$

$T < 1$ sharpens the distribution, $T > 1$ flattens it, and as $T$ approaches 0 sampling approaches greedy decoding (always the top token). Watch it with made-up logits for the word after "The cat sat on the":

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18 # sampling_toy.py
import numpy as np
 
words = np.array(["mat", "rug", "sofa", "floor", "moon"])
logits = np.array([3.0, 2.2, 1.5, 1.0, -1.0])
 
def softmax(z, temperature=1.0):
    z = z / temperature
    e = np.exp(z - z.max())            # subtract the max so exp() cannot overflow
    return e / e.sum()
 
rng = np.random.default_rng(42)
for t in (0.2, 0.7, 1.0, 1.5):
    p = softmax(logits, t)
    picks = rng.choice(words, size=1000, p=p)
    counts = {str(w): int((picks == w).sum()) for w in words}
    print(f"T={t:<4} p={np.round(p, 2)}  {counts}")

Output:

1
2
3
4
5 T=0.2  p=[0.98 0.02 0.   0.   0.  ]  {'mat': 979, 'rug': 21, 'sofa': 0, 'floor': 0, 'moon': 0}
T=0.7  p=[0.67 0.21 0.08 0.04 0.  ]  {'mat': 662, 'rug': 210, 'sofa': 80, 'floor': 44, 'moon': 4}
T=1.0  p=[0.55 0.25 0.12 0.07 0.01]  {'mat': 560, 'rug': 236, 'sofa': 117, 'floor': 73, 'moon': 14}
T=1.5  p=[0.44 0.26 0.16 0.12 0.03]  {'mat': 444, 'rug': 253, 'sofa': 158, 'floor': 119, 'moon': 26}

Two filters cut the unlikely tail before sampling: top-k keeps the $k$ most likely tokens; top-p (nucleus sampling, Holtzman et al., ICLR 2020) keeps the smallest set of top tokens whose probabilities add up to at least $p$ .

The generation loop from prompt to next token, and a table showing how temperature reshapes the probabilities of the same made-up logits

Why the same prompt gives different answers: above temperature 0 every token is a random draw, and one different early token changes everything after it. Even at 0, Anthropic's API reference says results will not be fully deterministic.

What current Claude models accept (vendor spec, September 2026)

Models	Sampling parameters
Claude 4.7 and later (Opus 4.7, 4.8, 5, 5.5), Sonnet 5, Fable 5 and 5.1	A non-default `temperature`, `top_p` or `top_k` returns a 400 error. Steer with the prompt and `effort`.
Claude Haiku 4.5	`temperature` (0.0 to 1.0, default 1.0) or `top_p`, one at a time.

The Python SDK 1.x also removed these arguments from messages.create(); for a model that still accepts them, pass them in extra_body:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18 # temperature_demo.py
import anthropic
 
client = anthropic.Anthropic()
PROMPT = "Suggest one name for a grey cat. Reply with the name only."
 
for temperature in (0.0, 1.0):
    names = []
    for _ in range(5):
        msg = client.messages.create(
            model="claude-haiku-4-5-20251001",
            max_tokens=20,
            messages=[{"role": "user", "content": PROMPT}],
            extra_body={"temperature": temperature},   # SDK 1.x: sampling settings go here
        )
        names.append("".join(b.text for b in msg.content if b.type == "text").strip())
    print(f"temperature={temperature}: {names}")

At 0.0 expect the same name, or nearly, five times; at 1.0 expect more variety.

Watch out: Anthropic lists Claude Haiku 4.5's retirement as "not sooner than October 15, 2026". If the call returns not-found, the model is gone; the NumPy simulation still shows the effect.

Common pitfall: treating temperature=0 as a correctness switch. It picks the most likely token, not the true one. For stable output, use clear instructions, structured output and validation in code.

Chapter 6 — Embeddings

Term	Meaning
Embedding	A vector (list of numbers) that represents the meaning of a text.
Dimension	Vector length: 384 for `all-MiniLM-L6-v2`; 1024 by default for Voyage's current models (vendor spec).
Cosine similarity	How closely two vectors point the same way: 1 same, 0 unrelated, -1 opposite.
Normalised vector	Scaled to length 1. Then cosine similarity is just the dot product.

An embedding model is a separate, usually smaller model trained so that texts with similar meaning get vectors pointing in similar directions. It powers semantic search, RAG (post 19), clustering and duplicate detection.

$\cos(\mathbf{a}, \mathbf{b}) = \frac{\mathbf{a} \cdot \mathbf{b}}{\lVert \mathbf{a} \rVert \, \lVert \mathbf{b} \rVert}$

1
2
3
4
5
6
7
8
9 import numpy as np
 
def cosine(a, b):
    a, b = np.asarray(a, dtype=float), np.asarray(b, dtype=float)
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))
 
print(cosine([1, 2, 3], [2, 4, 6]), cosine([1, 0], [0, 1]), cosine([1, 0], [-1, 0]))
# 1.0 0.0 -1.0

sentence-transformers/all-MiniLM-L6-v2 is a small, free, Apache-2.0 model that downloads from Hugging Face on first use. In the library's quickstart, "The weather is lovely today." and "It's so sunny outside!" score 0.6660, while each scores only about 0.10 to 0.14 against "He drove to the stadium.". Few shared words, same meaning. A tiny semantic search:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18 # faq_search.py
import numpy as np
from sentence_transformers import SentenceTransformer
 
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
faq = [
    "How do I reset my password?",
    "Where can I download my invoice?",
    "How do I change the email on my account?",
    "What payment methods do you accept?",
]
faq_vecs = model.encode(faq, normalize_embeddings=True)          # shape (4, 384)
query_vec = model.encode(["I forgot my login password"], normalize_embeddings=True)[0]
 
scores = faq_vecs @ query_vec                  # dot product = cosine for unit vectors
for i in np.argsort(-scores):
    print(f"{scores[i]:.3f}  {faq[i]}")

Expect the password question at the top.

Watch out: the model card says input longer than 256 word pieces is truncated. Long documents must be split into chunks first (post 19).

Trick: embed documents and queries with the same model. Vectors from different models cannot be compared.

Anthropic does not offer its own embedding model; its docs point to Voyage AI as one provider (post 19 shows both options).

Chapter 7 — Why hallucinations happen and how to reduce them

A hallucination is fluent, confident text that is false or not supported by the given sources. Causes:

The goal is a plausible next token, not a true statement. A made-up citation can look very plausible.
Training and tests reward guessing. Kalai et al. (2025, "Why Language Models Hallucinate") argue that training and evaluation reward guessing over admitting uncertainty, like an exam with no penalty for wrong answers.
Missing knowledge: after the cutoff, private, or rare.
Missing or buried context: not in the prompt, or lost in a very long one.

Fixes. The first five follow Anthropic's "Reduce hallucinations" guide and work with any model:

Technique	How
Allow "I don't know"	"If the documents do not contain the answer, say so."
Ground in sources	Put the documents in the prompt; tell the model to use only them.
Quote first	For long documents, extract word-for-word quotes, then answer from them.
Cite and check	One citation per claim; retract any claim without a supporting quote.
Compare several runs	Answers that disagree across runs are a warning sign.
Verify with tools	Run the code and tests; check official docs for any API or flag you do not know.

1
2
3
4
5
6
7
8
9 Answer using only the documents below.
<documents>
<doc id="1">...</doc>
<doc id="2">...</doc>
</documents>
Cite the doc id for every claim, like [1].
If the documents do not contain the answer, reply "Not found in the documents."
Question: ...

The AI Prompting post covers prompt structure and evaluation in depth; post 18 applies "verify with tools" to coding.

Chapter 8 — Prompt caching, tool use and extended thinking

Prompt caching

Much of a request repeats: system prompt, tool list, a long document. Prompt caching stores the processed prefix and reuses it (the general idea is in Dev Essentials 13 · Caching). Other providers have similar features; Anthropic's details (vendor spec, September 2026):

The match is on the exact prefix, in the order tools → system → messages. One changed byte invalidates everything after it, so put stable content first.
A cache entry lives 5 minutes by default, refreshed on each use; a 1-hour option exists.
A 5-minute cache write costs 1.25× the base input price; a cache read costs 0.1× on most models.
The minimum cacheable prefix is 512 to 4,096 tokens depending on the model (512 on Claude Opus 5.5).

Mark the end of the stable part with cache_control, then check response.usage.cache_read_input_tokens on the next call:

1
2 system=[{"type": "text", "text": LONG_STYLE_GUIDE, "cache_control": {"type": "ephemeral"}}]

Tool use (function calling)

The model cannot run code or query your database by itself. With tool use you describe functions (name, description, JSON Schema for the input). The model may reply with a structured call instead of text; your code runs it and sends the result back.

1
2
3
4 {"name": "get_order_status",
 "description": "Look up the shipping status of one order by its ID.",
 "input_schema": {"type": "object", "properties": {"order_id": {"type": "string"}}, "required": ["order_id"]}}

On the Anthropic API the call arrives as a tool_use block with stop_reason: "tool_use", and you answer with a tool_result block. Put that exchange in a loop and you have an agent: model + tools + loop. That is where post 18 starts.

Extended thinking (reasoning)

Since the model cannot go back and fix tokens, it helps to let it reason before the final answer. On the Anthropic API (vendor spec, September 2026):

Current models use adaptive thinking: the model decides how much to think; you steer depth with output_config={"effort": ...} (low, medium, high, xhigh, max).
On Claude Opus 5.5 thinking is always on, and the default effort is medium.
Thinking tokens are billed as output, and count toward max_tokens and the context window.
Thinking text is omitted by default; display: "summarized" returns a readable summary.

1
2
3
4
5
6
7
8
9
10
11
12
13
14 # thinking_demo.py
import anthropic
 
client = anthropic.Anthropic()
response = client.messages.create(
    model="claude-opus-5-5",
    max_tokens=16000,
    thinking={"type": "adaptive", "display": "summarized"},
    output_config={"effort": "high"},
    messages=[{"role": "user", "content": "Is 2,027 a prime number? Check it carefully."}],
)
for block in response.content:               # thinking blocks come first
    print(block.type.upper(), ":", getattr(block, "thinking", None) or getattr(block, "text", ""))

Common pitfall: reading response.content[0].text. With thinking on, the first block is a thinking block. Select blocks by type.

Practice Exercises

Exercise 1 — Grow the toy tokenizer

Train bpe_toy.py with 12 merges, once on the original corpus and once with " newer newer newer" added. Compare tokenize("newer", merges).

Solution

1
2
3
4 base = "low low low low lower lower newest newest newest widest widest low lowest"
print(tokenize("newer", train_bpe(base, 12)))                         # ['new', 'e', 'r', '</w>']
print(tokenize("newer", train_bpe(base + " newer newer newer", 12)))  # ['newer</w>']

Once newer is frequent in the training text, its inner pairs get merged and the whole word becomes one token. Frequent strings get cheap; rare strings stay expensive.

Exercise 2 — Your own chars-per-token table

Add a paragraph in your first language, a JSON snippet and a line of minified JavaScript to count_tokens.py. Rank all samples by chars/token, then repeat with a second model ID.

Solution

1
2
3
4
5
6
7
8
9 samples["My language"] = "(paste a paragraph in your language here) " * 5
samples["JSON"] = '{"id": 42, "tags": ["a", "b"], "active": true}\n' * 20
samples["Minified JS"] = "function f(a,b){return a.map(x=>x*b).filter(Boolean)}" * 10
ratios = {name: len(text) / client.messages.count_tokens(
    model=MODEL, messages=[{"role": "user", "content": text}]).input_tokens
    for name, text in samples.items()}
for name in sorted(ratios, key=ratios.get, reverse=True):
    print(f"{ratios[name]:5.2f}  {name}")

With a different model, the absolute counts shift if the tokenizer differs.

Exercise 3 — Implement top-k and top-p

Write top_k_filter(p, k) and top_p_filter(p, top_p) that return a renormalised probability array. Test on [0.50, 0.25, 0.12, 0.08, 0.05].

Solution

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18 import numpy as np
 
def top_k_filter(p, k):
    keep = np.argsort(p)[::-1][:k]
    q = np.zeros_like(p); q[keep] = p[keep]
    return q / q.sum()
 
def top_p_filter(p, top_p):
    order = np.argsort(p)[::-1]
    cutoff = int(np.searchsorted(np.cumsum(p[order]), top_p)) + 1   # smallest set with sum >= top_p
    keep = order[:cutoff]
    q = np.zeros_like(p); q[keep] = p[keep]
    return q / q.sum()
 
p = np.array([0.50, 0.25, 0.12, 0.08, 0.05])
print(np.round(top_k_filter(p, 2), 3))    # [0.667 0.333 0.    0.    0.   ]
print(np.round(top_p_filter(p, 0.8), 3))  # [0.575 0.287 0.138 0.    0.   ]

With top-p 0.8, the first two tokens only reach 0.75, so the third is kept too.

Exercise 4 — Why long chats get expensive

A 500-token system prompt, 100-token user messages, 400-token answers, full history resent every turn. How many input tokens does a 10-turn chat use?

Solution

Turn $n$ sends the system prompt, $n$ user messages and $n-1$ earlier answers: $500 + 100n + 400(n-1)$ .

1
2 print(sum(500 + 100 * n + 400 * (n - 1) for n in range(1, 11)))   # 28500

28,500 input tokens for a chat whose visible text is 5,500 tokens. Total input grows roughly with the square of the number of turns. Caching, trimming and summarising are the fixes.

Exercise 5 — Rewrite a prompt that invites hallucination

Improve: "Tell me the release date and main features of version 4 of our internal billing library."

Solution

The model has never seen your internal library, so it can only guess. Give it the facts and permission to say no:

1
2
3
4
5
6
7 <changelog>
...paste CHANGELOG.md here...
</changelog>
Using only the changelog: give the release date of version 4.0.0 and list its main
features, each with a quote from the changelog. If version 4 is not in the changelog,
reply "Version 4 is not in the changelog."

Cheat-sheet

Topic	Key fact
Token	Chunk from a fixed vocabulary, not a word; count with the model you will call
Claude rule of thumb	~555k English words / ~2.5M characters per 1M tokens (since Opus 4.7, vendor spec)
BPE	Merge the most frequent neighbouring pair, repeat; common text = fewer tokens
Code / non-English	Often more tokens per character; up to 15x gaps between languages measured
Count tokens	`client.messages.count_tokens(model=..., messages=[...]).input_tokens`
Generation	Logits → softmax → pick one token → append → repeat
Attention	$\text{softmax}(QK^T/\sqrt{d_k})V$ with a causal mask
Training	Pretraining → instruction tuning → preference training (RLHF / RLAIF)
Context window	Request + output; API is stateless, so history is resent every turn
Too long (Claude)	400 "prompt is too long"; output overflow → `model_context_window_exceeded`
Temperature	Low = focused, high = varied; 0 is not fully deterministic
top-k / top-p	Keep k most likely / smallest set reaching probability p
Claude sampling	Rejected (non-default) on Claude 4.7+, Sonnet 5, Fable; Haiku 4.5 via `extra_body`
Embeddings	Text → vector; normalise, then dot product = cosine
Hallucinations	Ground, allow "I don't know", quote, cite, verify with tools
Caching	Exact prefix; stable first; check `usage.cache_read_input_tokens`
Tool use / thinking	Structured calls your code runs / reasoning tokens billed as output

What's next

Next is Dev Essentials 18 · Coding with AI Agents: "model + tools + a loop" in daily practice with Claude Code, including project memory in CLAUDE.md, skills, hooks, subagents, permission modes, and how to review what an agent writes. Post 19 then builds a complete RAG system on this chapter's embeddings.

References

Note: this post is in progress. Planned additions: a token-count comparison across several vendors' tokenizers, a streaming example, a structured-output (JSON Schema) example, and how images are counted as tokens.