Working SWE → AI engineer · Stop 1 of 15 lessons

What is an LLM, really?

The problem

Most engineers treat a language model as an oracle: text in, answer out. Then it invents a function that doesn’t exist, costs triple what you budgeted, and returns a different answer to the same prompt twice in a row — and you have no mental model to explain any of it. You don’t need the maths. You need the mechanism.

The intuition

Strip away the chat interface and an LLM is a single, boring function: a sequence of tokens goes in, and a probability distribution over the next token comes out. The model reads everything so far and answers one question — “what token is likely to come next?” — as a score for every token it knows.¹ It does not plan a sentence, look anything up, or decide to be truthful.

An “answer” is that function called in a loop: pick a next token, glue it onto the end of the input, run the function again, repeat until a stop signal. This is what autoregressive means, and it’s why the GPT and Claude families generate one token at a time.¹ Chat, “reasoning,” and “memory” are all this one loop with text arranged around it — nothing more. Hold onto that picture; almost every surprising behaviour later in the roadmap falls out of it.

The mechanism

Four moving parts turn “predict the next token” into the thing you type at.

Tokens, not words. The model never sees characters or words. A tokenizer first chops your text into tokens — common chunks of bytes, usually via byte-pair encoding, so frequent sequences become one token and rare ones split into several.² A rough rule of thumb for English is one token ≈ 4 characters ≈ three-quarters of a word, so ~100 tokens is about 75 words.³ Tokens are case- and space-sensitive: GPT and GPT (with a leading space) are different tokens, because most tokenizers fold the leading space into the token.² The same text tokenizes to different counts on different models — that, plus different per-token prices, is why the same prompt costs different amounts.³

One forward pass → a distribution. The token sequence runs through the network — a transformer, the architecture from Attention Is All You Need⁴ — which emits one raw score, a logit, for every token in the vocabulary. A softmax turns those scores into a probability distribution over the whole vocabulary: the model’s answer to “what’s next?”¹

Sampling — turning the distribution into one token. How you pick is the part you control.⁵ Greedy takes the single highest-probability token (most APIs treat temperature=0 as this). Temperature scales the logits before the softmax: T = 1 leaves the distribution unchanged, T < 1 sharpens it toward the top candidates, T > 1 flattens it and gives long-shots a chance — it never changes the ranking, only how confident the model acts.⁵ Top-k keeps the k highest tokens; top-p (nucleus) keeps the smallest set whose probabilities sum to p, so the pool grows when the model is unsure and shrinks when it’s confident — introduced by Holtzman et al.⁶

The context window, and no memory. Between two API calls the model remembers nothing — it is stateless; each request stands alone.⁷ The illusion of a conversation exists because the client re-sends the whole transcript every turn. The context window is the most tokens (prompt + output) the model can attend to at once — from ~2K in early GPT-style models to 128K in Llama 3.1 and beyond in current systems.⁸ The “KV cache” you’ll read about is a speed-up within one call, not memory across calls.⁷

Build it

You don’t need a GPU to feel the mechanism — the sampling step is a few lines of pure Python. Given the model’s raw logits, this is the exact transformation the runtime does before it picks a token. The one thing that changes between “deterministic” and “creative” output is a single argument, highlighted.

import math, random

def sample_next_token(logits, temperature=1.0, top_p=1.0):
  if temperature <= 0:            # greedy: argmax
      return max(range(len(logits)),
                 key=lambda i: logits[i])
  scaled = [z / temperature for z in logits]
  m = max(scaled)                 # softmax, max-shifted
  exps = [math.exp(z - m) for z in scaled]
  total = sum(exps)
  probs = [e / total for e in exps]
  order = sorted(range(len(probs)),
                 key=lambda i: probs[i],
                 reverse=True)
  kept, cum = [], 0.0
  for i in order:                 # nucleus (top-p)
      kept.append(i)
      cum += probs[i]
      if cum >= top_p:
          break
  z = sum(probs[i] for i in kept) # renormalise
  r, acc = random.random() * z, 0.0
  for i in kept:
      acc += probs[i]
      if acc >= r:
          return i
  return kept[-1]

tok = sample_next_token(logits, temperature=0.7)

The whole “answer” is this called in a loop — forward → sample → append — until the model emits its stop token or you hit max_tokens. That loop is the shape of every generation endpoint you’ll ever call.

Use it

Run the sampler on one fixed distribution at two temperatures and watch the mechanism: temperature=0 returns the same token every time; a higher temperature spreads the picks across the plausible candidates.

import math, random

def sample(logits, temperature):
  if temperature <= 0:
      return max(range(len(logits)), key=lambda i: logits[i])
  e = [math.exp(z / temperature) for z in logits]
  s = sum(e); p = [x / s for x in e]
  r, acc = random.random(), 0.0
  for i, pi in enumerate(p):
      acc += pi
      if acc >= r: return i
  return len(p) - 1

logits = [2.0, 1.0, 0.2, -1.0]          # 4 candidate tokens
print("greedy :", [sample(logits, 0) for _ in range(8)])
print("t=1.0  :", [sample(logits, 1.0) for _ in range(8)])

In real use you never touch logits — you turn four knobs, in rough order of leverage: model choice (the ceiling on everything), the prompt/context (the model’s entire world for that call), sampling (temperature + top_p + max_tokens), and context management (what you keep, summarise, or drop as the window fills). Reach for temperature≈0 on extraction, classification, and structured output; raise it toward 0.7–1.0 for drafting and brainstorming. Move one of temperature or top-p, not both — they interact.⁵

In production

“Deterministic” is a lie you’ll tell yourself. Even at temperature=0, identical prompts can return different outputs. Sampling is only one source of variation; floating-point non-associativity, GPU batching, and mixture-of-experts routing mean the same input isn’t guaranteed the same logits from one run to the next. Design evals and caches to tolerate near-duplicates — compare on meaning or a tolerance, never demand a byte-identical reply, or your regression suite will flake on the model’s nature rather than on real bugs.

Cost is a token bill, and output dominates. You pay per input token and per output token, and providers price output tokens higher than input. With the rule of thumb of ~4 characters per token,³ a 2,000-word document is roughly 2,600 tokens of input you pay for on every call that includes it — so a long system prompt or a big retrieved context is a recurring cost, not a one-off. Because generation is one token at a time, latency also scales with output length: a 20-token answer streams far faster than a 2,000-token one. Cap max_tokens deliberately, and don’t ask for a paragraph when you need a word. The one input-cost lever worth knowing early is prompt caching: providers can bill a repeated, stable prefix — a big system prompt or a fixed context — at a discount on later calls, so order your prompt with the unchanging part first and the request last.

The window is a hard budget. Prompt plus output must fit inside the context window — 128K tokens on a current mid-size model, but still finite.⁸ Blow past it and you get a truncation, an error, or silently dropped history, none of them fun to debug at 2 a.m. Track your token count the way you track memory in a tight loop: measure it, cap it, and decide on purpose what gets evicted when you approach the ceiling — the oldest turns, or a running summary of them.

Knowledge is frozen at training time. A model’s parametric knowledge stops at its training cutoff and does not update as the world changes.⁹ Anything fresh, private, or after the cutoff has to arrive in the context on the call — which is the entire reason retrieval-augmented generation exists, and the whole of the RAG track downstream. If a fact matters and it’s newer than the model, you must hand it to the model; hoping it “knows” is how stale answers ship.

Failure modes

Keep this

An LLM is one function — tokens in, a distribution over the next token out — called in a loop until it stops. You control four things: which model, what’s in the context, how you sample, and how you manage the window. It is stateless between calls, its knowledge is frozen at training time, and it optimises for plausible, not true.