Training at scale Study Guide

The senior-staff bar

What L7-level depth looks like in this topic, the follow-up an interviewer would push on, and the resources that prepare you for it.

From the interview reportsNo staff-level evidence either way; core for infra and research-engineer roles. “What is 5D parallelism?” and “how would you train a 100B+ model?” are reported questions.

First-principles memory and step-time estimatesA 100B dense model, 4k sequence, 4M-token global batch on 1,024 H100s: how many bytes per GPU under ZeRO-3 vs TP8×PP4×DP32, including activations, and what step time and MFU do you expect? Show the arithmetic.
How to Scale Your Model: A Systems View of LLMs on TPUs The Ultra-Scale Playbook: Training LLMs on GPU Clusters Reducing Activation Recomputation in Large Transformer Models Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM The Llama 3 Herd of Models NVIDIA H100 Tensor Core GPU Architecture Whitepaper
Choosing a parallel layout against the bandwidth hierarchyWhy does TP stay inside the NVLink domain, when does adding PP beat more FSDP sharding, and how does your answer change on a 72-GPU NVLink domain (GB200 NVL72) or on TPU ICI?
Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM How to Scale Your Model: A Systems View of LLMs on TPUs The Ultra-Scale Playbook: Training LLMs on GPU Clusters Distributed Hybrid Parallelism for Large Language Models: Comparative Study and System Design Guide Pathways: Asynchronous Distributed Dataflow for ML Getting Started with Fully Sharded Data Parallel (FSDP2)
Communication cost modeling and overlapDerive ring vs tree all-reduce cost for a 1 GB gradient on 1,024 GPUs; when is DP communication exposed; how would you hide MoE all-to-all, and what does NCCL actually do at small message sizes?
Demystifying NCCL: An In-depth Analysis of GPU Communication Protocols and Algorithms Berkeley CS267: Applications of Parallel Computers (Spring 2025)How to Scale Your Model: A Systems View of LLMs on TPUs DeepSeek-V3 Technical Report Collective communication: theory, practice, and experience Communication Efficient Algorithms for Top-k Selection Problems
Designing for failures and goodputLlama 3 saw 419 unexpected interruptions in 54 days on 16K GPUs. For your 1,024-GPU run, estimate failures per day, pick a checkpoint interval, and explain how you detect a silent straggler or NCCL hang.
The Llama 3 Herd of Models Revisiting Reliability in Large-Scale Machine Learning Research Clusters Robust LLM Training Infrastructure at ByteDance MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs Machine Learning Engineering Open Book 100,000 H100 Clusters: Power, Network Topology, Ethernet vs InfiniBand, Reliability, Failures, Checkpointing ByteCheckpoint: A Unified Checkpointing System for Large Foundation Model Development OPT-175B training logbook (chronicles)Small-scale proxies for large-scale Transformer training instabilities
Low-precision trade-offsWhat breaks going from bf16 to fp8 to fp4, why per-tensor vs block scaling, which tensors stay in higher precision, and how would you detect a precision-induced divergence 200B tokens in?
Scaling FP8 training to trillion-token LLMs Pretraining Large Language Models with NVFP4 DeepSeek-V3 Technical Report Supercharging Training using float8 and FSDP2 Floating-Point 8: An Introduction to Efficient, Lower-Precision AI Training
Kernel-level performance reasoningThis attention kernel reaches 40% of H100 peak. Is it memory- or compute-bound, what does the profiler show, and what would FlashAttention-3-style warp specialization or a Blackwell TMEM kernel change?
Making Deep Learning Go Brrrr From First Principles Inside NVIDIA GPUs: Anatomy of high performance matmul kernels FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision GPUs Go Brrr (ThunderKittens)Nsight Compute Profiling Guide Anthropic's original performance take-home
Co-design and choosing what not to buildWould you adopt DualPipe/zero-bubble scheduling, write custom kernels, or rely on torch.compile and buy more interconnect? Defend the engineering-cost vs MFU trade-off for your org.
Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures Zero Bubble Pipeline Parallelism State of torch.compile for training (August 2025)PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation TorchTitan: One-stop PyTorch native solution for production ready LLM pre-training Pathways: Asynchronous Distributed Dataflow for ML

Start here

The best resources for a time-poor senior engineer, newest first.

2026Course

Stanford CS336: Language Modeling from Scratch (Spring 2026) — GPUs/TPUs, Kernels/Triton, Parallelism lectures

Percy Liang, Tatsunori Hashimoto (Stanford) · Apr 2026 · ~10 h

Four lectures (GPUs/TPUs Apr 13, Kernels/Triton Apr 15, Parallelism Apr 20 and 22) cover memory accounting, roofline, Triton and every parallelism axis in one place; Assignment 2 has you profile a model, write FlashAttention-2 in Triton and build distributed training.

2026Practice

Anthropic's original performance take-home

Anthropic · Jan 2026 · ~6 h

Optimize a kernel on a simulated VLIW SIMD accelerator to minimize cycles; README gives the 18,532-cycle starting point of the 2-hour version and Claude/human benchmarks (e.g. <1,487 cycles).

2026Tutorial

Triton tutorials

OpenAI / Triton team · 2026 · ~6 h

Vector add, fused softmax, matmul, layer norm and fused attention in Triton, each with a benchmark against PyTorch.

2026Practice

TorchTitan

PyTorch / Meta · 2026 · ~8 h

PyTorch-native reference pre-training stack: run Llama-style models with FSDP2, TP, PP, CP, float8 and torch.compile from a TOML config on rented GPUs.

2025Reference

The Ultra-Scale Playbook: Training LLMs on GPU Clusters

Nouamane Tazi, Ferdinand Mom, Haojun Zhao, Phuc Nguyen, Mohamed Mekkouri, Leandro von Werra, Thomas Wolf (Hugging Face) · Feb 2025 · ~12 h

Walks through DP, ZeRO-1/2/3, TP, sequence/context parallel (ring attention), pipeline (AFAB, 1F1B, interleaved, zero-bubble, DualPipe), expert parallelism and the '5D' combination, with memory formulas and 4,000+ benchmarked configs.

2025Book

How to Scale Your Model: A Systems View of LLMs on TPUs

Jacob Austin, Sholto Douglas, Roy Frostig, Anselm Levskaya, et al. (Google DeepMind) · Feb 2025 · ~15 h

Rooflines, the cost of every collective, how to shard a transformer for training and inference, with worked problems; has a dedicated GPU chapter. Written by GDM people who run these jobs.

2024Paper

The Llama 3 Herd of Models

Llama Team, AI @ Meta · Jul 2024 · ~3 h

Section 3.3 is a full infra case study: 16K H100s, 4D parallelism (TP, CP, PP, FSDP) layout and MFU, network topology, and the 54-day reliability log (466 interruptions, 419 unexpected, ~90% effective training time).

2022Blog post

Making Deep Learning Go Brrrr From First Principles

Horace He · 2022 · ~1 h

Splits runtime into compute, memory bandwidth and overhead, and explains operator fusion and why most ops are memory-bound. The fastest route to the right mental model.

Learning path

Work through the stages in order. The last stage is the senior-staff layer: production case studies, papers to critique and large-scale trade-offs. Each resource's L7 line says what it adds at your level.

1
Foundations: memory and the GPU mental model

Do the 16-bytes-per-param math, explain compute- vs memory-bound, and know the H100 numbers. · Week 1 (8–10 h)
- 2022
  ★Making Deep Learning Go Brrrr From First Principles
  Horace He · 2022 · Blog post · Intro · ~1 h
  Splits runtime into compute, memory bandwidth and overhead, and explains operator fusion and why most ops are memory-bound. The fastest route to the right mental model.
  L7Baseline; read first. Its compute/bandwidth/overhead split is the opening move for any 'why is this slow' question.
  What Shapes Do Matrix Multiplications Like? (2024-04)Strangely, matmuls run faster with predictable data (power limits)
- 2025
  GPU Glossary
  Modal (Charles Frye et al.) · 2025 · Reference · Intro · ~3 h
  Linked glossary of GPU hardware and software terms (SM, warp, tensor core, HBM, L2, CUDA, PTX, roofline, occupancy) with diagrams.
  L7Baseline vocabulary; skim if known.
- 2026
  ★Stanford CS336: Language Modeling from Scratch (Spring 2026) — GPUs/TPUs, Kernels/Triton, Parallelism lectures
  Percy Liang, Tatsunori Hashimoto (Stanford) · Apr 2026 · Course · Core · ~10 h · video
  Four lectures (GPUs/TPUs Apr 13, Kernels/Triton Apr 15, Parallelism Apr 20 and 22) cover memory accounting, roofline, Triton and every parallelism axis in one place; Assignment 2 has you profile a model, write FlashAttention-2 in Triton and build distributed training.
  L7Assignment 2 gives you a real artifact (your own Triton FlashAttention + DDP/sharded optimizer) to cite instead of describing papers secondhand.
  YouTube playlist (Spring 2026)YouTube playlist (Spring 2025)
- 2019
  ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
  Samyam Rajbhandari et al. (Microsoft) · Oct 2019 · Paper · Core · ~2 h
  Partitioning optimizer state, gradients and parameters across data-parallel ranks; source of the 2+2+12 = 16 bytes/param accounting.
  L7Baseline; the memory math in section 3 is the foundation for every fit-in-memory question.
  DeepSpeed Zero Redundancy Optimizer tutorial (2025)DeepSpeed home
- 2026
  Understanding CUDA Memory Usage
  PyTorch · 2026 · Reference · Intro · ~2 h
  How to record and visualize memory snapshots: parameters, gradients, optimizer state, activations and allocator fragmentation over a training step.
  L7Turns the 16-bytes-per-param rule into something you've measured; useful for answering 'why did this OOM'.
  Understanding GPU Memory 1 (PyTorch blog, 2023)
- 2022
  NVIDIA H100 Tensor Core GPU Architecture Whitepaper
  NVIDIA · 2022 · Reference · Core · ~2 h
  Hopper specs (SMs, tensor-core throughput by dtype, HBM3 bandwidth, NVLink 4, TMA, thread-block clusters, FP8 Transformer Engine) — the numbers you plug into roofline estimates.
  L7Memorizing a few numbers (≈1 PFLOP/s dense bf16, ≈3.35 TB/s HBM, 900 GB/s NVLink) lets you do step-time estimates live.
  Blackwell architecture technical overview Benchmarking and Dissecting the Nvidia Hopper GPU Architecture (2024-02)Dissecting the NVIDIA Blackwell Architecture with Microbenchmarks (2025-07)
- 2025
  Floating-Point 8: An Introduction to Efficient, Lower-Precision AI Training
  NVIDIA · Jun 2025 · Blog post · Intro · ~1 h
  E4M3 vs E5M2, per-tensor delayed/current scaling, block scaling (MXFP8) and what stays in bf16/fp32 during FP8 training.
  L7Baseline for the fp8 vocabulary; fastest route in for someone coming from fp32/bf16.
  Transformer Engine docs Transformer Engine GitHub FP8 Formats for Deep Learning (2022-09)
- 2026
  Getting Started with Distributed Data Parallel
  PyTorch · 2026 · Tutorial · Intro · ~2 h
  Run DDP with torchrun, understand gradient bucketing and overlap of all-reduce with backward.
  L7Baseline; it's step one of the hands-on plan (DDP then FSDP on rented GPUs).
  PyTorch Distributed Overview (2026)torch.distributed API DeviceMesh recipe PyTorch Distributed: Experiences on Accelerating Data Parallel Training (2020-06)
- 2023
  Transformer Math 101
  Quentin Anthony, Stella Biderman, Hailey Schoelkopf (EleutherAI) · Apr 2023 · Blog post · Intro · ~1 h
  Compute (6ND), memory for weights, optimizer states, gradients and activations by precision, and how ZeRO/TP/PP divide them — the formulas behind 'does 7B or 70B fit'.
  L7Baseline; read once and redo the 7B/70B arithmetic yourself — interviewers expect the numbers without notes.
  LoRA (arXiv 2106.09685)QLoRA (arXiv 2305.14314)
2
Parallelism and collectives

Explain every parallelism axis, its collectives and memory effect, and combine them into 5D parallelism. · Weeks 2–3 (15–20 h)
- 2025
  ★The Ultra-Scale Playbook: Training LLMs on GPU Clusters
  Nouamane Tazi, Ferdinand Mom, Haojun Zhao, Phuc Nguyen, Mohamed Mekkouri, Leandro von Werra, Thomas Wolf (Hugging Face) · Feb 2025 · Reference · Core · ~12 h
  Walks through DP, ZeRO-1/2/3, TP, sequence/context parallel (ring attention), pipeline (AFAB, 1F1B, interleaved, zero-bubble, DualPipe), expert parallelism and the '5D' combination, with memory formulas and 4,000+ benchmarked configs.
  L7This is the direct source for 'what is 5D parallelism?'; its benchmark section lets you justify a layout with measured MFU instead of folklore.
  5D parallelism section picotron (companion code)
- 2025
  ★How to Scale Your Model: A Systems View of LLMs on TPUs
  Jacob Austin, Sholto Douglas, Roy Frostig, Anselm Levskaya, et al. (Google DeepMind) · Feb 2025 · Book · Advanced · ~15 h
  Rooflines, the cost of every collective, how to shard a transformer for training and inference, with worked problems; has a dedicated GPU chapter. Written by GDM people who run these jobs.
  L7The worked problems train exactly the back-of-envelope step-time and memory math needed to lay out a 100B run on 1,024 GPUs and defend it under questioning.
  Rooflines chapter Training parallelism chapter How to think about GPUs Sharded matrices and how to multiply them Worked example: training LLaMA 3 on TPUs
- 2019
  Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
  Mohammad Shoeybi et al. (NVIDIA) · Sep 2019 · Paper · Core · ~2 h
  The original intra-layer tensor-parallel split of attention and MLP with only two all-reduces per layer.
  L7Baseline; you must be able to draw the column-then-row split and count the collectives.
- 2018
  GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
  Yanping Huang et al. (Google) · Nov 2018 · Paper · Core · ~1 h
  Micro-batched pipeline parallelism with synchronous flushes and recomputation; source of the (p−1)/m bubble analysis. Read with PipeDream-Flush (in extra), which introduced the 1F1B schedule that bounds in-flight activations.
  L7Baseline; know the bubble formula cold.
  Memory-Efficient Pipeline-Parallel DNN Training (PipeDream-2BW / PipeDream-Flush) (2020-06)PipeDream (2018, async original)
- 2023
  Ring Attention with Blockwise Transformers for Near-Infinite Context
  Hao Liu, Matei Zaharia, Pieter Abbeel (UC Berkeley) · Oct 2023 · Paper · Advanced · ~2 h
  Context parallelism: shard the sequence across devices and pass KV blocks around a ring, overlapping communication with blockwise attention. DeepSpeed Ulysses (extra) is the all-to-all alternative; PyTorch's context-parallel tutorial (extra) implements both.
  L7Lets you compute when ring-attention comm hides behind compute (block size vs bandwidth) and compare it to Ulysses all-to-all.
  ring-flash-attention implementation Introduction to Context Parallel (2026)DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models (2023-09)
- 2020
  GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
  Dmitry Lepikhin et al. (Google) · Jun 2020 · Paper · Core · ~2 h
  Top-2 MoE with expert parallelism via all-to-all, capacity factor and auxiliary load-balancing loss, plus sharding annotations.
  L7Baseline for expert parallelism; know where the all-to-alls happen and what capacity factor trades off.
  Switch Transformers (2021)MegaBlocks (2022)Training MoEs at Scale with PyTorch (2024-06)
- 2026
  Getting Started with Fully Sharded Data Parallel (FSDP2)
  PyTorch · 2026 · Tutorial · Core · ~3 h
  FSDP2's per-parameter DTensor sharding with fully_shard, prefetching, mixed precision and checkpointing.
  L7Being able to describe exactly when FSDP all-gathers and reduce-scatters, and how prefetch hides it, is what lets you argue FSDP vs TP at a given scale.
  fully_shard API PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel (2023-04)
- 2026
  Large Scale Transformer model training with Tensor Parallel (TP)
  PyTorch · 2026 · Tutorial · Core · ~2 h
  Column/row-wise parallel linear layers, sequence parallel, loss parallel, and how TP composes with FSDP on a 2D DeviceMesh.
  L7Makes the Megatron split concrete in code, so you can explain the f/g collectives and why TP stays inside a node.
  torch.distributed.tensor.parallel API
- 2025
  Demystifying NCCL: An In-depth Analysis of GPU Communication Protocols and Algorithms
  Zhiyi Hu, Siyuan Shen, Tommaso Bonato, Sylvain Jeaugey et al. (ETH Zurich, NVIDIA) · Jul 2025 · Paper · Advanced · ~2 h
  How NCCL actually implements collectives: Simple/LL/LL128 protocols, ring vs tree algorithms, channels, intra- vs inter-node data paths, and their latency/bandwidth regimes.
  L7Lets you go beyond 'ring all-reduce costs 2(N−1)/N · S/B' to explain which algorithm NCCL picks and why at different message sizes — the depth the Anthropic collectives questions probe.
  nccl-tests (benchmark your cluster)NCCL User Guide (2026)NCCL source
- 2022
  Reducing Activation Recomputation in Large Transformer Models
  Vijay Korthikanti et al. (NVIDIA) · May 2022 · Paper · Advanced · ~2 h
  Derives per-layer activation memory (≈ sbh(34 + 5as/h) bytes), introduces Megatron sequence parallelism and selective recomputation.
  L7The activation-memory formula here is what lets you finish the memory budget beyond the 16 bytes/param, with numbers.
  Chen et al., Training Deep Nets with Sublinear Memory Cost (2016)
- 2007
  Collective communication: theory, practice, and experience
  Ernie Chan, Marcel Heimlich, Avi Purkayastha, Robert van de Geijn (UT Austin) · 2007 · Paper · Advanced · ~3 h
  The classic alpha-beta-gamma treatment of MPI collectives: lower bounds, minimum-spanning-tree (latency-optimal) vs bucket/ring (bandwidth-optimal) algorithms, and composing all-reduce from reduce-scatter + all-gather.
  L7Lets you design an all-reduce from scratch for a given message size and topology and prove its cost — exactly the MPI/all-reduce design round.
- 2015
  Communication Efficient Algorithms for Top-k Selection Problems
  Lorenz Hübschle-Schneider, Peter Sanders, Ingo Müller (KIT) · Feb 2015 · Paper · Advanced · ~2 h
  Distributed selection (median/k-th element), most-frequent-items and top-k with provably low communication: sampling-based pivots, local aggregation, and handling skewed inputs.
  L7Directly prepares the distributed mode/median prompt: lets you trade rounds against bytes moved and handle skewed vs uniform data with stated bounds.
3
GPU performance and kernels

Read a profile, reason with rooflines, and write one fused Triton kernel. · Weeks 3–4 (12–15 h)
- 2026
  ★Triton tutorials
  OpenAI / Triton team · 2026 · Tutorial · Core · ~6 h
  Vector add, fused softmax, matmul, layer norm and fused attention in Triton, each with a benchmark against PyTorch.
  L7Writing fused softmax and matmul yourself is the minimum to credibly discuss fusion and tiling; the attention tutorial is FlashAttention in ~200 lines.
- 2024
  Triton Puzzles
  Sasha Rush, Tejas Ramesh / GPU MODE · 2024 · Practice · Core · ~4 h
  Progressive Triton exercises from vector ops to fused softmax, flash attention, quantized matmul, run in an interpreter without a GPU.
  L7Gets you to 'one Triton kernel written' (curriculum step 3) quickly and without renting hardware.
- 2022
  How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance: a Worklog
  Simon Boehm · Dec 2022 · Blog post · Core · ~4 h
  Ten kernel iterations from naive to ~94% of cuBLAS: coalescing, shared-memory tiling, 1D/2D blocktiling, vectorized loads, warptiling, with measured GFLOP/s at each step.
  L7Gives you a concrete optimization ladder to narrate in a kernel-optimization interview (including the Anthropic take-home's mindset).
  Lei Mao: CUDA Matrix Multiplication (2022)Lei Mao's Log Book (2026)
- 2022
  FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
  Tri Dao et al. (Stanford) · May 2022 · Paper · Core · ~2 h
  Tiling plus online softmax so attention never materializes the N×N matrix in HBM; the clearest example of IO-aware kernel design.
  L7Baseline; you should be able to explain the online-softmax trick and why it's memory-bound-to-compute-bound.
  FlashAttention-2 (2023)flash-attention GitHub
- 2026
  PyTorch Profiler recipe
  PyTorch · 2026 · Tutorial · Intro · ~1 h
  Profile CPU/GPU time and memory per operator and export a Chrome/Perfetto trace.
  L7Baseline; skim if known. Reading a trace for exposed communication and idle GPU time is the practical skill.
- 2026
  GPU MODE lecture series
  GPU MODE community (Mark Saroufim, Andreas Köpf et al.) · 2026 · Course · Advanced · ~20 h · video
  Free weekly talks by kernel authors (CUTLASS, Triton internals, FlashAttention, ThunderKittens, NCCL, quantization, profiling); repo has slides and code for each lecture.
  L7Hearing practitioners explain why their kernels are built the way they are lets you discuss kernel design choices at the level of the people who wrote them.
  YouTube channel Resource stream GPU MODE reference kernels / leaderboard (2025)GPU MODE site
- 2007
  Optimizing Parallel Reduction in CUDA
  Mark Harris (NVIDIA) · 2007 · Tutorial · Core · ~1 h
  Seven successive reduction kernels (divergence, bank conflicts, unrolling, multiple elements per thread) with bandwidth measured at each step.
  L7Baseline for the 'implement a parallel reduction / prefix sum' round; modern answers add warp shuffles and CUB, so go one step past the slides.
  GPU Gems 3 ch. 39: Parallel Prefix Sum (Scan) with CUDA
4
Hands-on

Run DDP then FSDP2/TP on rented GPUs, and do the Anthropic take-home in a 2-hour timebox. · Week 5 (10–15 h)
- 2026
  ★TorchTitan
  PyTorch / Meta · 2026 · Practice · Core · ~8 h
  PyTorch-native reference pre-training stack: run Llama-style models with FSDP2, TP, PP, CP, float8 and torch.compile from a TOML config on rented GPUs.
  L7Running real multi-dimensional parallel configs and reading the traces is the hands-on step that turns paper knowledge into numbers you measured yourself.
  TorchTitan Update: New SOTA Models, Performance at Scale, TitanRL (2026-10)
- 2025
  picotron
  Hugging Face (Ferdinand Mom et al.) · 2025 · Practice · Core · ~8 h
  Minimal educational 4D-parallel (DP, TP, PP, CP) training framework, each axis in a few hundred lines, companion to the Ultra-Scale Playbook.
  L7Implementing or reading each axis in small code lets you derive sharded matmul and backprop across devices from memory — the OpenAI sharded-GEMM prompt.
  nanotron (production version)
- 2026
  ★Anthropic's original performance take-home
  Anthropic · Jan 2026 · Practice · Advanced · ~6 h
  Optimize a kernel on a simulated VLIW SIMD accelerator to minimize cycles; README gives the 18,532-cycle starting point of the 2-hour version and Claude/human benchmarks (e.g. <1,487 cycles).
  L7This is the actual Anthropic interview task; timeboxing it to 2 hours and reaching the README thresholds is the most direct preparation available.
- 2025
  LeetGPU
  LeetGPU · 2025 · Practice · Core · ~6 h
  In-browser CUDA/Triton/PyTorch challenges (reductions, matmul, softmax, attention) run on real GPUs with leaderboards.
  L7Timed kernel problems are the closest thing to a kernel-coding round; good for building speed.
  Challenges
- 2026
  Getting Started with Distributed Checkpoint (DCP)
  PyTorch · 2026 · Tutorial · Core · ~1 h
  Sharded save/load of FSDP/TP state, resharding on load to a different world size, and async checkpointing.
  L7Lets you describe the mechanics behind a checkpoint/resume design (sharded, async, reshardable) rather than 'save every N steps'.
  torch.distributed.checkpoint API torchft (fault-tolerant HSDP, DiLoCo)
5
Senior-staff depth

Lay out a 100B run on 1,024 GPUs with step-time estimates and failure handling, and critique production designs. · Weeks 6–7 (15–20 h)
- 2021
  Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM
  Deepak Narayanan et al. (NVIDIA, Stanford, Microsoft) · Apr 2021 · Paper · Advanced · ~3 h
  PTD-P: how to combine tensor, pipeline (interleaved 1F1B) and data parallelism on 3,072 GPUs at 52% MFU, with the analytical rules for choosing degrees.
  L7This is the canonical source for 'how would you train a 100B+ model' — its layout heuristics and bubble formula are what you reproduce on the whiteboard.
- 2024
  ★The Llama 3 Herd of Models
  Llama Team, AI @ Meta · Jul 2024 · Paper · Advanced · ~3 h
  Section 3.3 is a full infra case study: 16K H100s, 4D parallelism (TP, CP, PP, FSDP) layout and MFU, network topology, and the 54-day reliability log (466 interruptions, 419 unexpected, ~90% effective training time).
  L7The single best source for 'lay out a large run and handle failures' with real numbers; your own company's run, so expect follow-ups.
- 2024
  DeepSeek-V3 Technical Report
  DeepSeek-AI · Dec 2024 · Paper · Advanced · ~4 h
  Full training-infra disclosure for a 671B MoE on 2,048 H800s: DualPipe, cross-node all-to-all kernels within NVLink/IB limits, FP8 training with fine-grained (tile/block) scaling, no TP, and cost in GPU-hours.
  L7The best public case study for designing around a constrained interconnect; lets you critique choices (no TP, 16-way PP, 64-way EP) with reasons.
  DeepEP (expert-parallel comm library)Profile data (overlap traces)DeepGEMM (FP8 GEMM)MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production (2025-05)
- 2025
  Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures
  Chenggang Zhao et al. (DeepSeek-AI) · May 2025 · Paper · Advanced · ~2 h
  ISCA 2025 paper on hardware-model co-design: memory-bound MLA, MoE comm vs NVLink/IB bandwidth, FP8 accumulation precision limits, multi-plane network topology, and requests to hardware vendors.
  L7Lets you argue hardware/software co-design trade-offs and name what you'd ask a chip or network team for — senior-staff scope.
- 2024
  MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs
  Ziheng Jiang et al. (ByteDance, Peking University) · Feb 2024 · Paper · Advanced · ~2 h
  Production training on 12,288 GPUs at 55.2% MFU: algorithm/system co-design, comm overlap, data pipeline, and a diagnosis/fault-tolerance toolkit for stragglers and failures.
  L7Gives concrete MFU numbers and straggler/failure tooling to cite when designing goodput and observability for a large run.
- 2024
  Revisiting Reliability in Large-Scale Machine Learning Research Clusters
  Apostolos Kokolis et al. (Meta FAIR) · Oct 2024 · Paper · Advanced · ~2 h
  Eleven months of failure data from two Meta research clusters (24K+ GPUs): failure taxonomy, MTTF scaling with job size, and effective-training-time metrics.
  L7Lets you derive expected failures per day for an N-GPU job and size checkpoint intervals from measured MTTF.
  Characterization of Large Language Model Development in the Datacenter (2024-03)
- 2025
  Robust LLM Training Infrastructure at ByteDance
  Borui Wan et al. (ByteDance Seed) · Sep 2025 · Paper · Advanced · ~2 h
  SOSP 2025: ByteRobust on a 200K-GPU platform — fault demarcation/localization, hot updates, warm standby machines, fault-aware checkpointing; 97% effective training time over three months on 9,600 GPUs.
  L7Most current production account of fast detection and recovery; lets you design the failure-handling half of a training-system answer.
- 2024
  Zero Bubble Pipeline Parallelism
  Penghui Qi et al. (Sea AI Lab) · Jan 2024 · Paper · Advanced · ~2 h
  Splits backward into input-grad and weight-grad passes and schedules them to nearly eliminate pipeline bubbles; basis for DualPipe in DeepSeek-V3.
  L7Lets you go past 1F1B and argue when bubble-elimination schedules are worth their memory and complexity cost.
  DualPipe (DeepSeek)
- 2024
  Scaling FP8 training to trillion-token LLMs
  Maxim Fishman, Brian Chmiel, Ron Banner, Daniel Soudry (Intel Habana, Technion) · Sep 2024 · Paper · Advanced · ~2 h
  Finds FP8 instabilities that only appear after ~200B tokens (SwiGLU outlier amplification) and fixes them (Smooth-SwiGLU, FP8 Adam moments) for a 2T-token run.
  L7Lets you name a real long-horizon failure mode of low precision, not just the throughput upside.
- 2025
  Inside NVIDIA GPUs: Anatomy of high performance matmul kernels
  Aleksa Gordić · Sep 2025 · Blog post · Advanced · ~5 h
  H100 from memory hierarchy to SASS, then Hopper-era async kernels: TMA, swizzling, wgmma tensor-core instructions, producer/consumer warp specialization, persistent kernels, clusters.
  L7Bridges the gap from Boehm's pre-tensor-core worklog to how state-of-the-art Hopper GEMMs are built, so you can discuss modern kernels without hand-waving.
  Blog index Matrix Multiplication on Blackwell: Part 1 — Introduction (2026-01)
- 2024
  FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision
  Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, Tri Dao · Jul 2024 · Paper · Advanced · ~2 h
  Hopper-specific attention: warp specialization, ping-pong scheduling of softmax and GEMM, and FP8 with incoherent processing; ~75% of H100 peak.
  L7Lets you explain how hardware asynchrony (TMA, wgmma) changes kernel design, a typical L7 kernel-depth probe.
  Tri Dao blog post
- 2026
  Distributed Hybrid Parallelism for Large Language Models: Comparative Study and System Design Guide
  Hossam Amer, Rezaul Karim, Ali Pourranjbar, Weiwei Zhang et al. · Feb 2026 · Survey paper · Advanced · ~3 h
  Recent comparison of hybrid parallel strategies, collective operations, communication–computation overlap and cost-model-driven automatic parallelization.
  L7Its cost-model framing mirrors the 'spreadsheet' sharding-estimation interview: compare layouts by compute, memory and network and name the binding resource.
- 2022
  OPT-175B training logbook (chronicles)
  Meta AI (Susan Zhang et al.) · 2022 · Practice · Advanced · ~2 h
  Day-by-day log of training a 175B model on 992 A100s: loss spikes and divergences, restarts from checkpoints with lowered LR, hardware failures and the fixes tried.
  L7The most candid public record of debugging divergence and failures at scale; lets you walk through a divergence triage with real precedent.
  OPT paper (arXiv 2205.01068)
- 2023
  Small-scale proxies for large-scale Transformer training instabilities
  Mitchell Wortsman, Peter J. Liu, Lechao Xiao et al. (Google DeepMind) · Sep 2023 · Paper · Advanced · ~2 h
  Reproduces large-scale instabilities (attention-logit growth, output-logit divergence) in small models and tests mitigations (qk-layernorm, z-loss) and LR sensitivity.
  L7Lets you propose a cheap, principled way to debug a divergence at 100K-GPU scale instead of burning the cluster on reruns.
- 2024
  ByteCheckpoint: A Unified Checkpointing System for Large Foundation Model Development
  Borui Wan et al. (ByteDance, HKU) · Jul 2024 · Paper · Advanced · ~2 h
  Production checkpointing: parallelism-agnostic representation, load-time resharding, async tiered save to memory/storage, with measured stall reductions.
  L7Lets you design a checkpoint system that survives rack failures and resizes jobs, with numbers on save/load stall time.

Question coverage

Every question group collected for this topic from candidate reports, mapped to the resources that prepare you for it. 31 of 31 groups are covered, which is 100% of all report mentions. Hover an example to see the full question.

Reports	Question group and example	Prepare with
3	VLIW SIMD kernel optimizationAnthropic 3Take-home: optimize a kernel's cycle count on a simulated VLIW SIMD machine (Anthropic performance take-home)	Anthropic's original performance take-home How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance: a Worklog Stanford CS149: Parallel Computing (Fall 2025)The take-home itself; ILP/SIMD and memory-hierarchy reasoning for cutting cycles
3	model exceeds one acceleratorGoogle DeepMind 2, OpenAI 1Design the training setup for a model that doesn't fit on one accelerator, say 70B parameters on a pod.	How to Scale Your Model: A Systems View of LLMs on TPUs The Ultra-Scale Playbook: Training LLMs on GPU Clusters Getting Started with Fully Sharded Data Parallel (FSDP2)The Llama 3 Herd of ModelsSharding a 70B model: memory math, FSDP/TP/PP choice, real 4D layout
2	kernel optimization take-homeAnthropic 2Take-Home: Optimising a Kernel on a Simulated Vector Machine: You are given a simulator for a small vector processor and a kernel that runs …	Anthropic's original performance take-home How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance: a WorklogTimeboxed practice on the real task plus a kernel optimization ladder
1	5D parallelismUnspecified frontier 1What is 5D parallelism?	The Ultra-Scale Playbook: Training LLMs on GPU Clusters5D parallelism section: DP, TP, PP, CP, EP and how they combine
1	sharded matmul debuggingOpenAI 1Debug problems in a sharded matmul with communication and make sure the training loss converges	How to Scale Your Model: A Systems View of LLMs on TPUs picotron Stanford CS336: Language Modeling from Scratch (Spring 2026) — GPUs/TPUs, Kernels/Triton, Parallelism lecturesSharded-matmul chapter, small TP code to bug-hunt, Assignment 2 distributed training
1	MoE pretrainingMeta MSL 1Round on pretraining MoE training (design a scalable MoE pretraining pipeline)	DeepSeek-V3 Technical Report GShard: Scaling Giant Models with Conditional Computation and Automatic ShardingProduction MoE pretraining: EP all-to-all, load balancing, overlap, layouts
1	distributed mode/median collectivesAnthropic 1Compute distributed mode and median given 10 nodes and 1,000 elements, with network as the bottleneck — minimize data movement; collectives …	Communication Efficient Algorithms for Top-k Selection Problems Collective communication: theory, practice, and experience Berkeley CS267: Applications of Parallel Computers (Spring 2025)Communication-efficient selection/frequency algorithms and tree/ring collective costs
1	performance-bounded designAnthropic 1Onsite prompt (from candidate portal): system design on scalable, performance-bounded systems under realistic constraints, deriving bounds f…	How to Scale Your Model: A Systems View of LLMs on TPUs 100,000 H100 Clusters: Power, Network Topology, Ethernet vs InfiniBand, Reliability, Failures, Checkpointing NVIDIA H100 Tensor Core GPU Architecture WhitepaperDeriving roofline and network bounds from hardware numbers and cluster constraints
1	distributed mode/medianAnthropic 1Distributed coding: compute mode and median using supplied messaging interfaces, attending to partitioning and communication costs	Communication Efficient Algorithms for Top-k Selection Problems Collective communication: theory, practice, and experiencePartitioned median/mode with sampling, local aggregation and collective cost models
1	sharding cost estimationAnthropic 1Use a spreadsheet-like tool to estimate compute, memory, and network costs under different model-sharding choices, identify the limiting res…	How to Scale Your Model: A Systems View of LLMs on TPUs Distributed Hybrid Parallelism for Large Language Models: Comparative Study and System Design Guide Transformer Math 101Compute/memory/network cost formulas per sharding; find the binding resource
1	tensor vs pipeline parallelismMistral 1Model parallelism: compare tensor parallelism and pipeline parallelism	Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM The Ultra-Scale Playbook: Training LLMs on GPU Clusters Large Scale Transformer model training with Tensor Parallel (TP)TP vs PP: communication volume, bubbles, memory, where each fits
1	sharded matmul + backpropOpenAI 1ML coding: simplified GEMM — derive sharded matmul, then matmul + backprop across devices in numpy/pytorch, then find a fixed number of plan…	How to Scale Your Model: A Systems View of LLMs on TPUs picotron Megatron-LM: Training Multi-Billion Parameter Language Models Using Model ParallelismDerive sharded matmul forward/backward collectives; code it in small framework
1	MPI allreduce designGoogle DeepMind 1EM system design on MPI/HPC and allreduce (topic given in advance)	Collective communication: theory, practice, and experience Demystifying NCCL: An In-depth Analysis of GPU Communication Protocols and Algorithms Berkeley CS267: Applications of Parallel Computers (Spring 2025)MPI collective algorithms, alpha-beta costs, NCCL ring/tree implementation
1	distributed modeAnthropic 1Find the most frequent value across data distributed over 10 workers when network bandwidth is the bottleneck	Communication Efficient Algorithms for Top-k Selection Problems Collective communication: theory, practice, and experienceMost-frequent-items algorithms minimizing bytes moved; reduction trees
1	TP/PP bandwidth tradeoffsxAI 1Distributed training at scale: tensor parallelism, pipeline parallelism, bandwidth tradeoffs	Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM How to Scale Your Model: A Systems View of LLMs on TPUsTP inside NVLink, PP across nodes: bandwidth-driven layout rules
1	tensor-parallel MLP backwardOpenAI 1Tensor-parallel MLP gradients: what each machine computes, what must be communicated, and how partial gradients combine	Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism Large Scale Transformer model training with Tensor Parallel (TP)How to Scale Your Model: A Systems View of LLMs on TPUsColumn/row split MLP: per-rank compute, f/g all-reduces, gradient combination
1	parallel matmul strategiesxAI 1ML coding: given starter code with a simulated inter-device communicator, implement two parallel strategies for matrix multiplication (NumPy…	How to Scale Your Model: A Systems View of LLMs on TPUs picotronMultiple sharded-matmul strategies (all-gather vs reduce-scatter) with simulated comms
1	parallelism strategies comparedGoogle DeepMind 1, Meta MSL 1Compare tensor, pipeline, data, sequence and expert parallelism. When do you combine them?	The Ultra-Scale Playbook: Training LLMs on GPU Clusters Efficient Training of Large Language Models on Distributed Infrastructures: A SurveyEach parallelism axis compared, with when-to-combine guidance and benchmarks
1	fine-tuning memory math 7BMistral 1Do the GPU memory maths for full fine-tuning a 7B model in bf16 with Adam. Now with LoRA.	Transformer Math 101 ZeRO: Memory Optimizations Toward Training Trillion Parameter ModelsBytes per param for weights, grads, Adam; LoRA/QLoRA memory in extras
1	scaling training to thousands of GPUsMeta MSL 1What breaks when you scale LLM training from 8 GPUs to thousands, and how do modern stacks deal with it?	MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs The Llama 3 Herd of Models How Meta trains large language models at scaleWhat breaks at 10k GPUs: stragglers, failures, network, comm overlap
1	goodput under hardware failuresxAI 1You're training on tens of thousands of GPUs and hardware fails constantly. How do you keep goodput high?	Robust LLM Training Infrastructure at ByteDance Revisiting Reliability in Large-Scale Machine Learning Research Clusters The Llama 3 Herd of ModelsFailure taxonomies, MTTF scaling, detection and fast recovery for goodput
1	simulated accelerator take-homeAnthropic 1Performance-team timed take-home built on a simulated accelerator (profiling, scratchpad memory, SIMD/VLIW, multicore reasoning); Anthropic …	Anthropic's original performance take-home Stanford CS149: Parallel Computing (Fall 2025)The published challenge plus SIMD/VLIW, scratchpad and multicore fundamentals
1	noisy all_gather information theoryOpenAI 1Implement all_gather across nodes over noisy channels; derive rounds needed for target accuracy given channel noise; then improve the bound …	Information Theory, Inference, and Learning Algorithms Collective communication: theory, practice, and experienceRepetition codes and channel capacity bounds; all-gather algorithm structure
1	70B training on 2K GPUsMistral 1Design the training pipeline for a 70B dense model across 2K H100 GPUs with checkpoint recovery.	The Llama 3 Herd of Models Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM Getting Started with Distributed Checkpoint (DCP)Layout for dense 70B on thousands of H100s and sharded checkpoint recovery
1	checkpoint resume rack failuresxAI 1Design the checkpoint / resume system for a frontier model training run that can survive rack failures.	ByteCheckpoint: A Unified Checkpointing System for Large Foundation Model Development Getting Started with Distributed Checkpoint (DCP)Robust LLM Training Infrastructure at ByteDanceSharded async checkpoints, resharding on restart, warm spares for rack loss

Show the other 6 question groups

1	CUDA reduction / prefix sumxAI 1Implement a parallel-algorithm primitive in CUDA or multi-GPU setting (parallel reduction, prefix sum, or all-reduce).	Optimizing Parallel Reduction in CUDA Programming Massively Parallel Processors: A Hands-on Approach, 4th ed.GPU PuzzlesWorked CUDA reduction and scan kernels; textbook chapters; hands-on puzzles
1	divergence debugging at scalexAI 1Debug a hypothetical training-divergence scenario at 100K GPU scale.	OPT-175B training logbook (chronicles)Small-scale proxies for large-scale Transformer training instabilities Scaling FP8 training to trillion-token LLMsReal divergence logs, small-scale instability proxies, low-precision failure modes
1	train 100B+ model parallelismUnspecified frontier 1How would you train a 100B+ parameter model?	Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM How to Scale Your Model: A Systems View of LLMs on TPUs The Ultra-Scale Playbook: Training LLMs on GPU ClustersPTD-P layout rules and step-time math for 100B+ training
1	matmul roofline shardingAnthropic 1Performance Modelling: Matmuls, Roofline and Sharding (4 parts / 60 min)	How to Scale Your Model: A Systems View of LLMs on TPUs NVIDIA H100 Tensor Core GPU Architecture WhitepaperRooflines and sharded-matmul chapters, plus hardware numbers to plug in
1	optimize LLM trainingxAI 1How would you optimize training for a large language model?	The Ultra-Scale Playbook: Training LLMs on GPU Clusters TorchTitan: One-stop PyTorch native solution for production ready LLM pre-training Making Deep Learning Go Brrrr From First PrinciplesAblated levers: parallelism, fp8, compile, overlap, fusion, and why each helps
1	distributed training fault toleranceAnthropic 1Design a distributed training pipeline for a large language model. How would you handle fault tolerance?	Robust LLM Training Infrastructure at ByteDance The Llama 3 Herd of Models Getting Started with Distributed Checkpoint (DCP)Pipeline design with failure detection, checkpointing and restart

All resources

Everything collected for this topic, newest first. 2026 items are marked green. ★ marks a top pick.

Type

Level

Subtopic

2026
MIT 6.5940: TinyML and Efficient AI Computing (Fall 2026)
Song Han (MIT HAN Lab) · Sep 2026 · Course · Core · ~8 h · video
Covers quantization, pruning, efficient attention and distributed training lectures; best used for the low-precision number-format material that feeds the fp8/fp4 discussion.
L7Baseline for quantization vocabulary; skim if you already know int8/fp8 formats.
Fall 2026 lecture list not shown on the landing page; videos linked are from 2023.
YouTube playlist (Fall 2023)
2026
CMU 10-714: Deep Learning Systems (Fall 2026)
Tianqi Chen, Zico Kolter (CMU) · Sep 2026 · Course · Core · ~15 h
Build your own autodiff framework (needle) with CPU and CUDA backends: tiling, memory coalescing, matmul and convolution kernels.
L7Baseline; useful only if you want to see autodiff + a CUDA backend end to end. Skip if time-poor.
Public recordings are from the 2022 online offering; Fall 2026 lectures post on the site.
2026
★Stanford CS336: Language Modeling from Scratch (Spring 2026) — GPUs/TPUs, Kernels/Triton, Parallelism lectures
Percy Liang, Tatsunori Hashimoto (Stanford) · Apr 2026 · Course · Core · ~10 h · video
Four lectures (GPUs/TPUs Apr 13, Kernels/Triton Apr 15, Parallelism Apr 20 and 22) cover memory accounting, roofline, Triton and every parallelism axis in one place; Assignment 2 has you profile a model, write FlashAttention-2 in Triton and build distributed training.
L7Assignment 2 gives you a real artifact (your own Triton FlashAttention + DDP/sharded optimizer) to cite instead of describing papers secondhand.
Watch only lectures 5–8 for this track; the assignment is ~20 h if done fully.
YouTube playlist (Spring 2026)YouTube playlist (Spring 2025)
2026
Distributed Hybrid Parallelism for Large Language Models: Comparative Study and System Design Guide
Hossam Amer, Rezaul Karim, Ali Pourranjbar, Weiwei Zhang et al. · Feb 2026 · Survey paper · Advanced · ~3 h
Recent comparison of hybrid parallel strategies, collective operations, communication–computation overlap and cost-model-driven automatic parallelization.
L7Its cost-model framing mirrors the 'spreadsheet' sharding-estimation interview: compare layouts by compute, memory and network and name the binding resource.
2026
★Anthropic's original performance take-home
Anthropic · Jan 2026 · Practice · Advanced · ~6 h
Optimize a kernel on a simulated VLIW SIMD accelerator to minimize cycles; README gives the 18,532-cycle starting point of the 2-hour version and Claude/human benchmarks (e.g. <1,487 cycles).
L7This is the actual Anthropic interview task; timeboxing it to 2 hours and reaching the README thresholds is the most direct preparation available.
Timebox to 2 hours first, then continue untimed. Don't modify tests/.
2026
GPU MODE lecture series
GPU MODE community (Mark Saroufim, Andreas Köpf et al.) · 2026 · Course · Advanced · ~20 h · video
Free weekly talks by kernel authors (CUTLASS, Triton internals, FlashAttention, ThunderKittens, NCCL, quantization, profiling); repo has slides and code for each lecture.
L7Hearing practitioners explain why their kernels are built the way they are lets you discuss kernel design choices at the level of the people who wrote them.
Pick 4–6 lectures (profiling, Triton, CUTLASS/CuTe, FlashAttention, NCCL); don't binge.
YouTube channel Resource stream GPU MODE reference kernels / leaderboard (2025)GPU MODE site
2026
Machine Learning Engineering Open Book
Stas Bekman · 2026 · Book · Advanced · ~10 h
Practitioner handbook from BLOOM/IDEFICS training: hardware selection, network benchmarking, fault tolerance, debugging hangs, checkpointing and throughput tuning.
L7Supplies the operational detail (NCCL debugging, hang triage, checkpoint cadence) that separates someone who has run a large job from someone who has read about one.
Continuously updated repo; date is the current state.
Fault tolerance chapter Network benchmarks Training performance
2026
CUDA Programming Guide
NVIDIA · 2026 · Reference · Core · ~6 h
Official reference for the CUDA execution and memory model (thread blocks, warps, shared memory, async copies, clusters), rewritten for CUDA 13.
L7Baseline reference; read the programming-model and memory-hierarchy chapters only.
Legacy CUDA C++ Programming Guide CUDA C++ Best Practices Guide
2026
★Triton tutorials
OpenAI / Triton team · 2026 · Tutorial · Core · ~6 h
Vector add, fused softmax, matmul, layer norm and fused attention in Triton, each with a benchmark against PyTorch.
L7Writing fused softmax and matmul yourself is the minimum to credibly discuss fusion and tiling; the attention tutorial is FlashAttention in ~200 lines.
2026
CUTLASS and CuTe documentation
NVIDIA · 2026 · Reference · Advanced · ~8 h
CuTe layouts and tensor algebra, the abstraction under CUTLASS 3/4 and FlashAttention-3; also the CuTe DSL in Python.
L7Lets you explain how production Hopper/Blackwell GEMM and attention kernels are actually structured (layouts, TMA, warp specialization).
Only needed for kernel-heavy roles.
CUTLASS GitHub CuTe docs on GitHub
2026
Getting Started with Distributed Data Parallel
PyTorch · 2026 · Tutorial · Intro · ~2 h
Run DDP with torchrun, understand gradient bucketing and overlap of all-reduce with backward.
L7Baseline; it's step one of the hands-on plan (DDP then FSDP on rented GPUs).
PyTorch Distributed Overview (2026)torch.distributed API DeviceMesh recipe PyTorch Distributed: Experiences on Accelerating Data Parallel Training (2020-06)
2026
Getting Started with Fully Sharded Data Parallel (FSDP2)
PyTorch · 2026 · Tutorial · Core · ~3 h
FSDP2's per-parameter DTensor sharding with fully_shard, prefetching, mixed precision and checkpointing.
L7Being able to describe exactly when FSDP all-gathers and reduce-scatters, and how prefetch hides it, is what lets you argue FSDP vs TP at a given scale.
fully_shard API PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel (2023-04)
2026
Large Scale Transformer model training with Tensor Parallel (TP)
PyTorch · 2026 · Tutorial · Core · ~2 h
Column/row-wise parallel linear layers, sequence parallel, loss parallel, and how TP composes with FSDP on a 2D DeviceMesh.
L7Makes the Megatron split concrete in code, so you can explain the f/g collectives and why TP stays inside a node.
torch.distributed.tensor.parallel API
2026
Understanding CUDA Memory Usage
PyTorch · 2026 · Reference · Intro · ~2 h
How to record and visualize memory snapshots: parameters, gradients, optimizer state, activations and allocator fragmentation over a training step.
L7Turns the 16-bytes-per-param rule into something you've measured; useful for answering 'why did this OOM'.
Understanding GPU Memory 1 (PyTorch blog, 2023)
2026
PyTorch Profiler recipe
PyTorch · 2026 · Tutorial · Intro · ~1 h
Profile CPU/GPU time and memory per operator and export a Chrome/Perfetto trace.
L7Baseline; skim if known. Reading a trace for exposed communication and idle GPU time is the practical skill.
2026
Nsight Compute Profiling Guide
NVIDIA · 2026 · Reference · Advanced · ~3 h
How Nsight Compute measures kernels: roofline charts, memory workload analysis, warp stall reasons, occupancy.
L7Lets you diagnose a kernel by stall reason and memory throughput instead of guessing, which is how kernel-perf follow-ups are judged.
Nsight Systems (timeline profiler)
2026
JAX: Introduction to parallel programming (sharded computation)
Google JAX team · 2026 · Tutorial · Core · ~3 h
Mesh, NamedSharding/PartitionSpec, automatic (jit/GSPMD) vs explicit sharding vs manual shard_map.
L7Lets you discuss TPU-style sharding-annotation programming, the model GDM uses, alongside PyTorch's eager distributed style.
Manual parallelism with shard_map Distributed arrays and automatic parallelization
2026
Megatron-LM / Megatron Core
NVIDIA · 2026 · Reference · Advanced · ~4 h
NVIDIA's reference implementation of TP, PP (interleaved 1F1B), sequence/context parallel, expert parallel and FP8 used for most large GPU runs outside the big labs.
L7Reading its parallel-state and pipeline-schedule code lets you talk about real implementation constraints (virtual stages, microbatch counts, comm overlap).
Megatron Core user guide
2026
★TorchTitan
PyTorch / Meta · 2026 · Practice · Core · ~8 h
PyTorch-native reference pre-training stack: run Llama-style models with FSDP2, TP, PP, CP, float8 and torch.compile from a TOML config on rented GPUs.
L7Running real multi-dimensional parallel configs and reading the traces is the hands-on step that turns paper knowledge into numbers you measured yourself.
TorchTitan Update: New SOTA Models, Performance at Scale, TitanRL (2026-10)
2026
Getting Started with Distributed Checkpoint (DCP)
PyTorch · 2026 · Tutorial · Core · ~1 h
Sharded save/load of FSDP/TP state, resharding on load to a different world size, and async checkpointing.
L7Lets you describe the mechanics behind a checkpoint/resume design (sharded, async, reshardable) rather than 'save every N steps'.
torch.distributed.checkpoint API torchft (fault-tolerant HSDP, DiLoCo)
2025
Stanford CS149: Parallel Computing (Fall 2025)
Kayvon Fatahalian, Kunle Olukotun (Stanford) · Sep 2025 · Course · Core · ~12 h · video
The clearest course on how throughput hardware works: multicore/SIMD, GPU architecture and CUDA (Oct 14), evaluating DNNs on GPUs (Oct 21), hardware specialization and mapping AI to the datacenter (Oct 23–30).
L7Gives you the latency-hiding and data-movement intuition to reason about why a kernel or layout is slow, not just name the fix.
Fall 2025 videos are not public (site says so); slides are. Use the 2023 playlist for video.
YouTube playlist (2023 lectures, latest public videos)Fall 2024 site CMU 15-418/15-618: Parallel Computer Architecture and Programming (Fall 2026) (2026-08)Schedule and slides
2025
Inside NVIDIA GPUs: Anatomy of high performance matmul kernels
Aleksa Gordić · Sep 2025 · Blog post · Advanced · ~5 h
H100 from memory hierarchy to SASS, then Hopper-era async kernels: TMA, swizzling, wgmma tensor-core instructions, producer/consumer warp specialization, persistent kernels, clusters.
L7Bridges the gap from Boehm's pre-tensor-core worklog to how state-of-the-art Hopper GEMMs are built, so you can discuss modern kernels without hand-waving.
Blog index Matrix Multiplication on Blackwell: Part 1 — Introduction (2026-01)
2025
Defeating Nondeterminism in LLM Inference
Horace He (Thinking Machines Lab) · Sep 2025 · Blog post · Advanced · ~1 h
Shows that run-to-run variance comes from batch-size-dependent reduction order in kernels, and how to write batch-invariant matmul, RMSNorm and attention.
L7Lets you reason about floating-point reduction order in kernels and collectives — relevant to debugging loss divergence and reproducibility at scale.
2025
Robust LLM Training Infrastructure at ByteDance
Borui Wan et al. (ByteDance Seed) · Sep 2025 · Paper · Advanced · ~2 h
SOSP 2025: ByteRobust on a 200K-GPU platform — fault demarcation/localization, hot updates, warm standby machines, fault-aware checkpointing; 97% effective training time over three months on 9,600 GPUs.
L7Most current production account of fast detection and recovery; lets you design the failure-handling half of a training-system answer.
2025
Pretraining Large Language Models with NVFP4
NVIDIA · Sep 2025 · Paper · Advanced · ~2 h
4-bit (NVFP4) pretraining of a 12B model on 10T tokens on Blackwell: random Hadamard transforms, 2D block scaling, stochastic rounding, and which layers stay in higher precision.
L7Puts you at the 2025–2026 frontier of precision trade-offs and lets you argue what fp4 costs in accuracy and engineering.
2025
State of torch.compile for training (August 2025)
Edward Z. Yang (Meta PyTorch) · Aug 2025 · Blog post · Advanced · ~1 h
Candid status report from a PyTorch lead on where compile helps or hurts for large-scale training: compile time, graph breaks, distributed integration, and what frontier labs actually use.
L7Lets you take an informed position on 'compiler vs hand-written kernels' in a training stack, including the costs.
Introduction to torch.compile (2026)torch.compiler docs torch.compile, the missing manual (ezyang, Google Doc)
2025
Demystifying NCCL: An In-depth Analysis of GPU Communication Protocols and Algorithms
Zhiyi Hu, Siyuan Shen, Tommaso Bonato, Sylvain Jeaugey et al. (ETH Zurich, NVIDIA) · Jul 2025 · Paper · Advanced · ~2 h
How NCCL actually implements collectives: Simple/LL/LL128 protocols, ring vs tree algorithms, channels, intra- vs inter-node data paths, and their latency/bandwidth regimes.
L7Lets you go beyond 'ring all-reduce costs 2(N−1)/N · S/B' to explain which algorithm NCCL picks and why at different message sizes — the depth the Anthropic collectives questions probe.
nccl-tests (benchmark your cluster)NCCL User Guide (2026)NCCL source
2025
Floating-Point 8: An Introduction to Efficient, Lower-Precision AI Training
NVIDIA · Jun 2025 · Blog post · Intro · ~1 h
E4M3 vs E5M2, per-tensor delayed/current scaling, block scaling (MXFP8) and what stays in bf16/fp32 during FP8 training.
L7Baseline for the fp8 vocabulary; fastest route in for someone coming from fp32/bf16.
Transformer Engine docs Transformer Engine GitHub FP8 Formats for Deep Learning (2022-09)
2025
Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures
Chenggang Zhao et al. (DeepSeek-AI) · May 2025 · Paper · Advanced · ~2 h
ISCA 2025 paper on hardware-model co-design: memory-bound MLA, MoE comm vs NVLink/IB bandwidth, FP8 accumulation precision limits, multi-plane network topology, and requests to hardware vendors.
L7Lets you argue hardware/software co-design trade-offs and name what you'd ask a chip or network team for — senior-staff scope.
2025
★The Ultra-Scale Playbook: Training LLMs on GPU Clusters
Nouamane Tazi, Ferdinand Mom, Haojun Zhao, Phuc Nguyen, Mohamed Mekkouri, Leandro von Werra, Thomas Wolf (Hugging Face) · Feb 2025 · Reference · Core · ~12 h
Walks through DP, ZeRO-1/2/3, TP, sequence/context parallel (ring attention), pipeline (AFAB, 1F1B, interleaved, zero-bubble, DualPipe), expert parallelism and the '5D' combination, with memory formulas and 4,000+ benchmarked configs.
L7This is the direct source for 'what is 5D parallelism?'; its benchmark section lets you justify a layout with measured MFU instead of folklore.
5D parallelism section picotron (companion code)
2025
★How to Scale Your Model: A Systems View of LLMs on TPUs
Jacob Austin, Sholto Douglas, Roy Frostig, Anselm Levskaya, et al. (Google DeepMind) · Feb 2025 · Book · Advanced · ~15 h
Rooflines, the cost of every collective, how to shard a transformer for training and inference, with worked problems; has a dedicated GPU chapter. Written by GDM people who run these jobs.
L7The worked problems train exactly the back-of-envelope step-time and memory math needed to lay out a 100B run on 1,024 GPUs and defend it under questioning.
Do the end-of-chapter problems; reading alone is not enough.
Rooflines chapter Training parallelism chapter How to think about GPUs Sharded matrices and how to multiply them Worked example: training LLaMA 3 on TPUs
2025
KernelBench
Anne Ouyang, Simon Guo et al. (Stanford Scaling Intelligence Lab) · Feb 2025 · Practice · Advanced · ~6 h
250 PyTorch workloads (single ops, fused blocks, full models) to beat with custom kernels, with correctness and speedup harness.
L7A graded set of fusion targets to practice choosing what is worth a custom kernel vs torch.compile.
KernelBench paper (arXiv 2502.10517)
2025
Berkeley CS267: Applications of Parallel Computers (Spring 2025)
Aydin Buluç, James Demmel (UC Berkeley / LBNL) · Jan 2025 · Course · Advanced · ~15 h · video
HPC view of parallelism: communication-avoiding algorithms, collective cost models (latency/bandwidth alpha-beta), distributed dense linear algebra and ML applications — the theory behind the sharded-matmul and distributed mode/median questions.
L7Lets you derive communication lower bounds and alpha-beta costs for a sharded algorithm on the whiteboard, which is what the Anthropic distributed-coding and OpenAI sharded-GEMM prompts reward.
Course site says recordings need an @berkeley.edu login; the playlist page itself loads publicly. Spring 2026 site requires sign-in.
YouTube playlist (Spring 2025)
2025
GPU Glossary
Modal (Charles Frye et al.) · 2025 · Reference · Intro · ~3 h
Linked glossary of GPU hardware and software terms (SM, warp, tensor core, HBM, L2, CUDA, PTX, roofline, occupancy) with diagrams.
L7Baseline vocabulary; skim if known.
2025
picotron
Hugging Face (Ferdinand Mom et al.) · 2025 · Practice · Core · ~8 h
Minimal educational 4D-parallel (DP, TP, PP, CP) training framework, each axis in a few hundred lines, companion to the Ultra-Scale Playbook.
L7Implementing or reading each axis in small code lets you derive sharded matmul and backprop across devices from memory — the OpenAI sharded-GEMM prompt.
nanotron (production version)
2025
LeetGPU
LeetGPU · 2025 · Practice · Core · ~6 h
In-browser CUDA/Triton/PyTorch challenges (reductions, matmul, softmax, attention) run on real GPUs with leaderboards.
L7Timed kernel problems are the closest thing to a kernel-coding round; good for building speed.
Free tier; some features need an account.
Challenges
2024
DeepSeek-V3 Technical Report
DeepSeek-AI · Dec 2024 · Paper · Advanced · ~4 h
Full training-infra disclosure for a 671B MoE on 2,048 H800s: DualPipe, cross-node all-to-all kernels within NVLink/IB limits, FP8 training with fine-grained (tile/block) scaling, no TP, and cost in GPU-hours.
L7The best public case study for designing around a constrained interconnect; lets you critique choices (no TP, 16-way PP, 64-way EP) with reasons.
DeepEP (expert-parallel comm library)Profile data (overlap traces)DeepGEMM (FP8 GEMM)MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production (2025-05)
2024
Supercharging Training using float8 and FSDP2
IBM and Meta PyTorch teams · Nov 2024 · Blog post · Advanced · ~1 h
Measured float8 training speedups (up to ~50% over bf16) on Llama-style models up to 405B with FSDP2 and float8 all-gather, with loss-parity checks.
L7Gives you real throughput and convergence numbers to cite when asked whether fp8 is worth it and where the gains come from.
Float8 rowwise on 2K H200s (PyTorch blog, 2025-04)
2024
Revisiting Reliability in Large-Scale Machine Learning Research Clusters
Apostolos Kokolis et al. (Meta FAIR) · Oct 2024 · Paper · Advanced · ~2 h
Eleven months of failure data from two Meta research clusters (24K+ GPUs): failure taxonomy, MTTF scaling with job size, and effective-training-time metrics.
L7Lets you derive expected failures per day for an N-GPU job and size checkpoint intervals from measured MTTF.
Characterization of Large Language Model Development in the Datacenter (2024-03)
2024
TorchTitan: One-stop PyTorch native solution for production ready LLM pre-training
Wanchao Liang et al. (Meta) · Oct 2024 · Paper · Advanced · ~2 h
Composable FSDP2 + TP + PP + CP, float8, async TP, torch.compile and distributed checkpointing, with measured speedups on Llama 3.1 8B–405B up to 512 GPUs.
L7Gives ablation numbers for each technique so you can say what each parallelism/precision lever is worth.
2024
Scaling FP8 training to trillion-token LLMs
Maxim Fishman, Brian Chmiel, Ron Banner, Daniel Soudry (Intel Habana, Technion) · Sep 2024 · Paper · Advanced · ~2 h
Finds FP8 instabilities that only appear after ~200B tokens (SwiGLU outlier amplification) and fixes them (Smooth-SwiGLU, FP8 Adam moments) for a 2T-token run.
L7Lets you name a real long-horizon failure mode of low precision, not just the throughput upside.
2024
Efficient Training of Large Language Models on Distributed Infrastructures: A Survey
Jiangfei Duan et al. (Shanghai AI Lab and others) · Jul 2024 · Survey paper · Core · ~4 h
Broad map of LLM training infrastructure: accelerators, networks, storage, every parallelism scheme, compute/memory/communication optimizations, and fault tolerance, with references.
L7Use as an index to find the right primary paper when an interviewer names a technique you haven't seen.
2024
FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision
Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, Tri Dao · Jul 2024 · Paper · Advanced · ~2 h
Hopper-specific attention: warp specialization, ping-pong scheduling of softmax and GEMM, and FP8 with incoherent processing; ~75% of H100 peak.
L7Lets you explain how hardware asynchrony (TMA, wgmma) changes kernel design, a typical L7 kernel-depth probe.
Tri Dao blog post
2024
★The Llama 3 Herd of Models
Llama Team, AI @ Meta · Jul 2024 · Paper · Advanced · ~3 h
Section 3.3 is a full infra case study: 16K H100s, 4D parallelism (TP, CP, PP, FSDP) layout and MFU, network topology, and the 54-day reliability log (466 interruptions, 419 unexpected, ~90% effective training time).
L7The single best source for 'lay out a large run and handle failures' with real numbers; your own company's run, so expect follow-ups.
2024
ByteCheckpoint: A Unified Checkpointing System for Large Foundation Model Development
Borui Wan et al. (ByteDance, HKU) · Jul 2024 · Paper · Advanced · ~2 h
Production checkpointing: parallelism-agnostic representation, load-time resharding, async tiered save to memory/storage, with measured stall reductions.
L7Lets you design a checkpoint system that survives rack failures and resizes jobs, with numbers on save/load stall time.
2024
How Meta trains large language models at scale
Meta Engineering · Jun 2024 · Blog post · Advanced · ~1 h
Meta's account of moving from many small jobs to few very large ones: hardware reliability, fast recovery, efficient state preservation, optimal GPU connectivity.
L7You're a Meta engineer — being able to speak to your own company's training infra choices with specifics is a credibility signal.
Building Meta's GenAI Infrastructure (2024-03)RoCE networks for distributed AI training at scale (2024-08)
2024
100,000 H100 Clusters: Power, Network Topology, Ethernet vs InfiniBand, Reliability, Failures, Checkpointing
Dylan Patel et al. (SemiAnalysis) · Jun 2024 · Blog post · Advanced · ~2 h · paid
Cluster-level view: power budgets, rail-optimized fat-tree topologies, InfiniBand vs Ethernet, failure rates and checkpointing at 100k-GPU scale.
L7Lets you reason about the datacenter constraints (power, topology, failure rates) that bound a training design — org-level judgment beyond the model code.
Free preview; the later sections are paywalled.
2024
GPUs Go Brrr (ThunderKittens)
Benjamin Spector, Aaryan Singhal, Simran Arora, Chris Ré (Stanford Hazy Research) · May 2024 · Blog post · Advanced · ~2 h
What H100s actually reward (tensor cores, TMA, async wgmma, warp-group scheduling) and the tile-based ThunderKittens DSL built around it.
L7Gives you a defensible view on why tile-level abstractions win on modern GPUs and how they relate to Triton and CUTLASS.
ThunderKittens paper (arXiv 2410.20399)ThunderKittens GitHub
2024
PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation
Jason Ansel et al. (Meta) · Apr 2024 · Paper · Advanced · ~3 h
ASPLOS 2024 paper on TorchDynamo and TorchInductor: how graphs are captured, guarded and lowered to Triton.
L7Know how the compiler actually works so you can reason about when fusion is automatic and when a custom kernel is warranted.
2024
MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs
Ziheng Jiang et al. (ByteDance, Peking University) · Feb 2024 · Paper · Advanced · ~2 h
Production training on 12,288 GPUs at 55.2% MFU: algorithm/system co-design, comm overlap, data pipeline, and a diagnosis/fault-tolerance toolkit for stragglers and failures.
L7Gives concrete MFU numbers and straggler/failure tooling to cite when designing goodput and observability for a large run.
2024
Zero Bubble Pipeline Parallelism
Penghui Qi et al. (Sea AI Lab) · Jan 2024 · Paper · Advanced · ~2 h
Splits backward into input-grad and weight-grad passes and schedules them to nearly eliminate pipeline bubbles; basis for DualPipe in DeepSeek-V3.
L7Lets you go past 1F1B and argue when bubble-elimination schedules are worth their memory and complexity cost.
DualPipe (DeepSeek)
2024
Triton Puzzles
Sasha Rush, Tejas Ramesh / GPU MODE · 2024 · Practice · Core · ~4 h
Progressive Triton exercises from vector ops to fused softmax, flash attention, quantized matmul, run in an interpreter without a GPU.
L7Gets you to 'one Triton kernel written' (curriculum step 3) quickly and without renting hardware.
2024
llm.c
Andrej Karpathy · 2024 · Practice · Core · ~6 h
GPT-2 training in raw C/CUDA with hand-written kernels, mixed precision, multi-GPU (NCCL) and ZeRO-1.
L7A full training loop with no framework, so you can explain every byte and every kernel in a step.
2023
Ring Attention with Blockwise Transformers for Near-Infinite Context
Hao Liu, Matei Zaharia, Pieter Abbeel (UC Berkeley) · Oct 2023 · Paper · Advanced · ~2 h
Context parallelism: shard the sequence across devices and pass KV blocks around a ring, overlapping communication with blockwise attention. DeepSpeed Ulysses (extra) is the all-to-all alternative; PyTorch's context-parallel tutorial (extra) implements both.
L7Lets you compute when ring-attention comm hides behind compute (block size vs bandwidth) and compare it to Ulysses all-to-all.
ring-flash-attention implementation Introduction to Context Parallel (2026)DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models (2023-09)
2023
Small-scale proxies for large-scale Transformer training instabilities
Mitchell Wortsman, Peter J. Liu, Lechao Xiao et al. (Google DeepMind) · Sep 2023 · Paper · Advanced · ~2 h
Reproduces large-scale instabilities (attention-logit growth, output-logit divergence) in small models and tests mitigations (qk-layernorm, z-loss) and LR sensitivity.
L7Lets you propose a cheap, principled way to debug a divergence at 100K-GPU scale instead of burning the cluster on reruns.
2023
Transformer Math 101
Quentin Anthony, Stella Biderman, Hailey Schoelkopf (EleutherAI) · Apr 2023 · Blog post · Intro · ~1 h
Compute (6ND), memory for weights, optimizer states, gradients and activations by precision, and how ZeRO/TP/PP divide them — the formulas behind 'does 7B or 70B fit'.
L7Baseline; read once and redo the 7B/70B arithmetic yourself — interviewers expect the numbers without notes.
LoRA (arXiv 2106.09685)QLoRA (arXiv 2305.14314)
2022
How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance: a Worklog
Simon Boehm · Dec 2022 · Blog post · Core · ~4 h
Ten kernel iterations from naive to ~94% of cuBLAS: coalescing, shared-memory tiling, 1D/2D blocktiling, vectorized loads, warptiling, with measured GFLOP/s at each step.
L7Gives you a concrete optimization ladder to narrate in a kernel-optimization interview (including the Anthropic take-home's mindset).
Older (no tensor cores) but still the canonical worklog.
Lei Mao: CUDA Matrix Multiplication (2022)Lei Mao's Log Book (2026)
2022
Reducing Activation Recomputation in Large Transformer Models
Vijay Korthikanti et al. (NVIDIA) · May 2022 · Paper · Advanced · ~2 h
Derives per-layer activation memory (≈ sbh(34 + 5as/h) bytes), introduces Megatron sequence parallelism and selective recomputation.
L7The activation-memory formula here is what lets you finish the memory budget beyond the 16 bytes/param, with numbers.
Chen et al., Training Deep Nets with Sublinear Memory Cost (2016)
2022
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
Tri Dao et al. (Stanford) · May 2022 · Paper · Core · ~2 h
Tiling plus online softmax so attention never materializes the N×N matrix in HBM; the clearest example of IO-aware kernel design.
L7Baseline; you should be able to explain the online-softmax trick and why it's memory-bound-to-compute-bound.
FlashAttention-2 (2023)flash-attention GitHub
2022
Pathways: Asynchronous Distributed Dataflow for ML
Paul Barham et al. (Google) · Mar 2022 · Paper · Advanced · ~2 h
Google's single-controller runtime that schedules SPMD computations across TPU pods over ICI and DCN; used to train PaLM on 6,144 TPU v4 chips.
L7Lets you contrast Google's single-controller design with the multi-controller SPMD (torchrun) model and discuss what it buys for multi-pod training.
PaLM paper (2022) — training-infra sections GSPMD: General and Scalable Parallelization for ML Computation Graphs (2021-05)TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings (2023-04)TPU architecture (2026)
2022
Programming Massively Parallel Processors: A Hands-on Approach, 4th ed.
Wen-mei Hwu, David Kirk, Izzat El Hajj (Morgan Kaufmann) · 2022 · Book · Core · ~25 h · paid
The standard CUDA textbook (basis of UIUC ECE 408): memory coalescing, tiling, reductions, scans, convolution, sparse ops, and performance considerations, with exercises.
L7Baseline; read chapters 4–6 (architecture, memory, performance) and 10 (reduction) — enough to write and defend a tiled kernel.
Paid book. ECE 408's own course site could not be loaded (stale request); the GPU MODE reading group also follows this book.
PMPP lecture videos (YouTube channel linked from publisher companion page)Publisher companion page
2022
★Making Deep Learning Go Brrrr From First Principles
Horace He · 2022 · Blog post · Intro · ~1 h
Splits runtime into compute, memory bandwidth and overhead, and explains operator fusion and why most ops are memory-bound. The fastest route to the right mental model.
L7Baseline; read first. Its compute/bandwidth/overhead split is the opening move for any 'why is this slow' question.
Older but still the canonical intro.
What Shapes Do Matrix Multiplications Like? (2024-04)Strangely, matmuls run faster with predictable data (power limits)
2022
NVIDIA H100 Tensor Core GPU Architecture Whitepaper
NVIDIA · 2022 · Reference · Core · ~2 h
Hopper specs (SMs, tensor-core throughput by dtype, HBM3 bandwidth, NVLink 4, TMA, thread-block clusters, FP8 Transformer Engine) — the numbers you plug into roofline estimates.
L7Memorizing a few numbers (≈1 PFLOP/s dense bf16, ≈3.35 TB/s HBM, 900 GB/s NVLink) lets you do step-time estimates live.
Hosted PDF behind an NVIDIA viewer; may prompt for an email.
Blackwell architecture technical overview Benchmarking and Dissecting the Nvidia Hopper GPU Architecture (2024-02)Dissecting the NVIDIA Blackwell Architecture with Microbenchmarks (2025-07)
2022
GPU Puzzles
Sasha Rush · 2022 · Practice · Intro · ~3 h
14 notebook puzzles teaching CUDA thread/block indexing, shared memory, reductions, scans and matmul via Numba.
L7Baseline; fastest way to get the CUDA programming model into your fingers.
2022
OPT-175B training logbook (chronicles)
Meta AI (Susan Zhang et al.) · 2022 · Practice · Advanced · ~2 h
Day-by-day log of training a 175B model on 992 A100s: loss spikes and divergences, restarts from checkpoints with lowered LR, hardware failures and the fixes tried.
L7The most candid public record of debugging divergence and failures at scale; lets you walk through a divergence triage with real precedent.
OPT paper (arXiv 2205.01068)
2021
Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM
Deepak Narayanan et al. (NVIDIA, Stanford, Microsoft) · Apr 2021 · Paper · Advanced · ~3 h
PTD-P: how to combine tensor, pipeline (interleaved 1F1B) and data parallelism on 3,072 GPUs at 52% MFU, with the analytical rules for choosing degrees.
L7This is the canonical source for 'how would you train a 100B+ model' — its layout heuristics and bubble formula are what you reproduce on the whiteboard.
2020
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
Dmitry Lepikhin et al. (Google) · Jun 2020 · Paper · Core · ~2 h
Top-2 MoE with expert parallelism via all-to-all, capacity factor and auxiliary load-balancing loss, plus sharding annotations.
L7Baseline for expert parallelism; know where the all-to-alls happen and what capacity factor trades off.
Switch Transformers (2021)MegaBlocks (2022)Training MoEs at Scale with PyTorch (2024-06)
2019
ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
Samyam Rajbhandari et al. (Microsoft) · Oct 2019 · Paper · Core · ~2 h
Partitioning optimizer state, gradients and parameters across data-parallel ranks; source of the 2+2+12 = 16 bytes/param accounting.
L7Baseline; the memory math in section 3 is the foundation for every fit-in-memory question.
DeepSpeed Zero Redundancy Optimizer tutorial (2025)DeepSpeed home
2019
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
Mohammad Shoeybi et al. (NVIDIA) · Sep 2019 · Paper · Core · ~2 h
The original intra-layer tensor-parallel split of attention and MLP with only two all-reduces per layer.
L7Baseline; you must be able to draw the column-then-row split and count the collectives.
2018
GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
Yanping Huang et al. (Google) · Nov 2018 · Paper · Core · ~1 h
Micro-batched pipeline parallelism with synchronous flushes and recomputation; source of the (p−1)/m bubble analysis. Read with PipeDream-Flush (in extra), which introduced the 1F1B schedule that bounds in-flight activations.
L7Baseline; know the bubble formula cold.
Memory-Efficient Pipeline-Parallel DNN Training (PipeDream-2BW / PipeDream-Flush) (2020-06)PipeDream (2018, async original)
2015
Communication Efficient Algorithms for Top-k Selection Problems
Lorenz Hübschle-Schneider, Peter Sanders, Ingo Müller (KIT) · Feb 2015 · Paper · Advanced · ~2 h
Distributed selection (median/k-th element), most-frequent-items and top-k with provably low communication: sampling-based pivots, local aggregation, and handling skewed inputs.
L7Directly prepares the distributed mode/median prompt: lets you trade rounds against bytes moved and handle skewed vs uniform data with stated bounds.
2007
Collective communication: theory, practice, and experience
Ernie Chan, Marcel Heimlich, Avi Purkayastha, Robert van de Geijn (UT Austin) · 2007 · Paper · Advanced · ~3 h
The classic alpha-beta-gamma treatment of MPI collectives: lower bounds, minimum-spanning-tree (latency-optimal) vs bucket/ring (bandwidth-optimal) algorithms, and composing all-reduce from reduce-scatter + all-gather.
L7Lets you design an all-reduce from scratch for a given message size and topology and prove its cost — exactly the MPI/all-reduce design round.
Older but canonical; PDF hosted on a UT Austin course page (Concurrency and Computation: Practice and Experience, 2007).
2007
Optimizing Parallel Reduction in CUDA
Mark Harris (NVIDIA) · 2007 · Tutorial · Core · ~1 h
Seven successive reduction kernels (divergence, bank conflicts, unrolling, multiple elements per thread) with bandwidth measured at each step.
L7Baseline for the 'implement a parallel reduction / prefix sum' round; modern answers add warp shuffles and CUB, so go one step past the slides.
Older but still the canonical worked example.
GPU Gems 3 ch. 39: Parallel Prefix Sum (Scan) with CUDA
2003
Information Theory, Inference, and Learning Algorithms
David J. C. MacKay (Cambridge) · 2003 · Book · Core · ~4 h
Free textbook; chapters 1 and 9–10 cover repetition codes, binary symmetric channels and channel capacity — the tools for bounding rounds needed over a noisy channel.
L7Only for the noisy all_gather prompt: lets you derive repetition-count bounds from channel error rate and argue why coding beats naive repetition.
Older but canonical; read chapters 1, 9, 10 only.
Full book PDF

Not found or not verified. No public course site or lecture videos could be verified for UIUC ECE 408 (courses.grainger.illinois.edu returns a stale-request page); the PMPP book and its publisher-linked YouTube channel stand in. Fall 2025 CS149, Fall 2026 CMU 15-418 and Spring 2026 CS267 videos are not public (older public playlists listed instead; CS267 2025 recordings may need a Berkeley login). No 2025 broad arXiv survey of distributed LLM training was found; the 2024 survey plus a Feb 2026 hybrid-parallelism comparison are used. No public infra write-ups from Anthropic, OpenAI or xAI exist on training-cluster design; Meta, DeepSeek, ByteDance and Google material substitutes. Thakur et al.'s MPICH collectives paper is behind a bot wall (Chan et al. 2007 used instead). Aleksa Gordić's planned Blackwell matmul post was not found; Modular's Blackwell series covers it. Noisy all_gather (scale-q23) has no dedicated resource; it is covered by information-theory fundamentals plus collective-algorithm structure. The Anthropic Interview Notes repo cited by the questions was not added (third-party). Links checked 27 Oct 2026.

01 Practical coding 02 Algorithms 03 Fundamentals 04 ML coding 05 LLM internals 06 Deep dive & leadership 07 System design 08 Inference design 09 ML infra 10 Training at scale 11 Applied LLM 12 Research 13 Values & mission

Distributed training and GPU performance

The senior-staff bar

Start here

Stanford CS336: Language Modeling from Scratch (Spring 2026) — GPUs/TPUs, Kernels/Triton, Parallelism lectures

Anthropic's original performance take-home

Triton tutorials

TorchTitan

The Ultra-Scale Playbook: Training LLMs on GPU Clusters

How to Scale Your Model: A Systems View of LLMs on TPUs

The Llama 3 Herd of Models

Making Deep Learning Go Brrrr From First Principles

Learning path

Foundations: memory and the GPU mental model

Parallelism and collectives

GPU performance and kernels

Hands-on

Senior-staff depth

What the interviews ask

Question coverage

All resources