ML coding rounds ask you to write core model pieces (attention, a KV-cached decode loop, a sampler, a backward pass, an optimizer, or a classic algorithm) from a blank file in NumPy or PyTorch, or to find planted bugs in a small transformer. Frontier labs use it to check that you can turn ideas into correct tensor code fast; OpenAI reports a miniGPT debug plus add-a-KV-cache round and Anthropic a MHA + KV cache + GQA hour. At senior staff the code is table stakes: you are also expected to prove it correct, explain the numerics and memory costs, and say how the same bug or design would behave at scale.
What L7-level depth looks like in this topic, the follow-up an interviewer would push on, and the resources that prepare you for it.
The best resources for a time-poor senior engineer, newest first.
Sebastian Raschka · Oct 2026 · ~25 h
Chapter notebooks implement causal multi-head attention, GPT blocks, pretraining, sampling with temperature/top-k, and bonus folders for KV cache, GQA, RoPE-based architectures (Llama, Qwen, Gemma 3) and LoRA (appendix E). Actively maintained in 2026.
Tatsunori Hashimoto, Percy Liang (Stanford) · Apr 2026 · ~120 h
Assignment 1 has you write a byte-level BPE tokenizer, a pre-norm transformer with RoPE and SwiGLU, cross-entropy, AdamW, a cosine schedule and a training loop against unit tests; it is the curriculum's end-to-end capstone and the single best rehearsal for the from-scratch rounds.
Stanford (Fei-Fei Li, Ehsan Adeli et al.) · Apr 2026 · ~40 h
Assignment 1 is kNN with no-loop vectorized L2 distances, a softmax classifier and a two-layer net with hand-written backward passes checked numerically; A2 adds batchnorm/dropout backward, A3 a captioning transformer. The no-loop kNN is almost exactly the reported OpenAI 1-NN question.
Andrej Karpathy · Nov 2025 · ~8 h
~300-line GPT model and training loop that is the de facto template for 'miniGPT' interview code; reading model.py closely is preparation for debugging someone else's version of it.
Sebastian Raschka · Jun 2025 · ~2 h
Walks through adding a KV cache to a from-scratch GPT: cache buffers, position tracking, reset, and a side-by-side script with and without the cache.
Andrej Karpathy · Jun 2024 · ~30 h
Code-along series from micrograd (scalar autograd) through manual backprop of an MLP with batchnorm ('becoming a backprop ninja'), to building GPT, the GPT tokenizer and reproducing GPT-2. The fastest way to make attention-from-a-blank-file muscle memory.
Mitchell Wortsman et al. (Google DeepMind) · Sep 2023 · ~2 h
Reproduces large-model instabilities (attention-logit growth, output-logit divergence) in small models and shows the fixes (qk-layernorm, z-loss) and LR sensitivity analysis.
Sasha Rush · Oct 2022 · ~3 h
Write the backward (VJP) for a series of tensor functions, checked automatically; directly drills matmul, sum, broadcasting and indexing gradients.
Sasha Rush · Apr 2022 · ~4 h
21 puzzles reimplementing NumPy/PyTorch functions using only broadcasting and arange, one line each.
Andrej Karpathy · Apr 2019 · ~1 h
The canonical checklist: inspect data, fix seeds, verify init loss (~ln(vocab)), overfit one batch, visualize just before the net, add complexity one step at a time.
Work through the stages in order. The last stage is the senior-staff layer: production case studies, papers to critique and large-scale trade-offs. Each resource's L7 line says what it adds at your level.
Write causal multi-head attention, RoPE and a KV-cached decode loop with shapes in comments, tested against SDPA · Week 1 (10–12 h)
Matmul/softmax-CE/layernorm backward in NumPy checked with finite differences; AdamW; loop-free kNN, k-means, logistic regression · Week 2 (10 h)
Build a 15-bug zoo, narrate a bisection, find 5 planted bugs in 45 min; implement temperature/top-k/top-p/beam · Week 3 (8–10 h)
BPE tokenizer, transformer LM, AdamW, schedule and training loop passing the official tests, trained on TinyStories · Weeks 4–5 (25–40 h)
Argue KV-cache budgets, attention variants, stability fixes and nondeterminism with numbers and primary sources · Week 6+ (12–15 h, selective)
Problems that recur in this track across 30 reports (16 firsthand), with the resources that prepare you for each.
Every question group collected for this topic from candidate reports, mapped to the resources that prepare you for it. 52 of 52 groups are covered, which is 100% of all report mentions. Hover an example to see the full question.
Everything collected for this topic, newest first. 2026 items are marked green. ★ marks a top pick.