2026Book
Machine Learning Engineering Open Book
Stas Bekman · 2026 · ~20 h
Practitioner handbook from BLOOM/IDEFICS training: compute, storage, network benchmarking, fault tolerance, SLURM, debugging and checkpointing at scale.
The senior-staff bar
What L7-level depth looks like in this topic, the follow-up an interviewer would push on, and the resources that prepare you for it.
From the interview reportsThe Anthropic senior-staff infra loop's design round was a model downloader that recovers without a coordinator. Distributing large model files to thousands of machines recurs across staff reports.
Start here
The best resources for a time-poor senior engineer, newest first.
2026Reference
Best practices design patterns: optimizing Amazon S3 performance
Amazon Web Services · 2026 · ~0.5 h
Per-prefix request rates (3,500 PUT / 5,500 GET per second), parallel ranged GETs, retries and scaling across prefixes.
2026Reference
Mosaic Streaming
MosaicML (Databricks) · 2026 · ~2 h
Streaming dataset format with deterministic, elastic shuffling and mid-epoch resumption across changing numbers of nodes.
2025Book
How To Scale Your Model
Jacob Austin, Sholto Douglas et al. (Google DeepMind) · Feb 2025 · ~15 h
Roofline analysis, TPU/GPU networking, parallelism strategies and serving maths for LLMs, written by DeepMind researchers.
2024Paper
The Llama 3 Herd of Models (infrastructure sections)
Llama Team (Meta AI) · Jul 2024 · ~3 h
Section 3.3 covers the 16K-H100 cluster, storage (Tectonic, 2–7 TB/s checkpoint bursts), network and 466 job interruptions in 54 days with root causes.
2024Paper
MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs
Ziheng Jiang et al. (ByteDance), NSDI 2024 · Feb 2024 · ~3 h
ByteDance's 12,288-GPU training system: diagnostics, straggler detection, fast checkpoint/recovery and network tuning, with incident data.
2023Paper
Gemini: Fast Failure Recovery in Distributed Training with In-Memory Checkpoints
Zhuang Wang et al. (Rice, AWS), SOSP · Oct 2023 · ~2 h
Checkpoints to peer CPU memory with placement and traffic scheduling so recovery is 13x faster than remote storage.
2022Paper
Owl: Scale and Flexibility in Distribution of Hot Content
Jason Flinn et al. (Meta), OSDI · Jul 2022 · ~3 h
Meta's system for distributing hot binaries, models and packages to millions of hosts: P2P data plane with a centralized control plane and per-use-case policies.
2019Tool
Kraken: P2P Docker registry
Uber Engineering · Mar 2019 · ~3 h
Production P2P blob distribution (tens of TB, thousands of hosts) with content-addressed pieces, a tracker, origin cluster and bandwidth-aware peer selection.
2015Paper
Large-scale cluster management at Google with Borg
Abhishek Verma et al. (Google), EuroSys · Apr 2015 · ~2 h
Priorities, quotas, preemption, admission control and cell sizing in Google's cluster manager.
Learning path
Work through the stages in order. The last stage is the senior-staff layer: production case studies, papers to critique and large-scale trade-offs. Each resource's L7 line says what it adds at your level.
- 1
Numbers and vocabulary
Do the bandwidth, memory and failure-rate arithmetic every design here needs · Week 1 (8–10 h)
- 2025Jacob Austin, Sholto Douglas et al. (Google DeepMind) · Feb 2025 · Book · Advanced · ~15 hRoofline analysis, TPU/GPU networking, parallelism strategies and serving maths for LLMs, written by DeepMind researchers.L7Gives you the bandwidth and memory arithmetic (bytes/s per link, HBM, ICI/NVLink vs DCN) to put real numbers on any distribution or loader design.
- 2025Nouamane Tazi et al. (Hugging Face) · Feb 2025 · Tutorial · Core · ~10 hInteractive guide to data, tensor, pipeline, context and expert parallelism with measured throughput on up to 512 GPUs.L7Baseline for the parallelism vocabulary interviewers assume; skim if you already know FSDP/TP/PP trade-offs.
- 2026Amazon Web Services · 2026 · Reference · Core · ~0.5 hPer-prefix request rates (3,500 PUT / 5,500 GET per second), parallel ranged GETs, retries and scaling across prefixes.L7Gives the concrete object-storage limits that force you to add P2P or a cache tier when 5,000 hosts start at once.
- 2008Bram Cohen (BitTorrent.org) · 2008 · Reference · Core · ~1 hPiece hashing, metainfo files, tracker protocol and peer wire protocol; the base vocabulary for chunked P2P distribution.L7Baseline; read it so piece size, rarest-first and tit-for-tat are concrete when you adapt them to a trusted datacenter.
- 2026Hugging Face · 2026 · Reference · Intro · ~0.5 hWeight file format with a JSON header and zero-copy, memory-mapped loading.L7Baseline; know that headers allow parallel ranged reads per tensor or shard.
- 2026NVIDIA · 2026 · Reference · Core · ~1 hBroadcast, all-gather and other collectives with ring/tree algorithms; the in-cluster alternative to P2P for weight fan-out.L7Lets you compare ring/tree broadcast bandwidth with P2P swarm and pick per network tier.
- 2026Percy Liang, Tatsunori Hashimoto (Stanford) · Apr 2026 · Course · Core · ~40 h · videoBuilds an LM stack from tokenizer to distributed training; the systems, parallelism and data lectures show how training data and GPUs are kept busy.L7Watch the GPU, parallelism and data lectures; they let you reason about where loader or network bottlenecks sit in a real training step.
- 2025
- 2
Model and checkpoint distribution
Design chunked, content-addressed, P2P/tree distribution with coordinator-free recovery · Week 2 (8–10 h)
- 2019Uber Engineering · Mar 2019 · Tool · Advanced · ~3 hProduction P2P blob distribution (tens of TB, thousands of hosts) with content-addressed pieces, a tracker, origin cluster and bandwidth-aware peer selection.L7The closest open design to the interview prompt; lets you argue tracker vs DHT, origin rate limits and piece size with a real system.
- 2026Dragonfly maintainers (CNCF, originally Alibaba) · 2026 · Reference · Core · ~2 hP2P file and image distribution with a scheduler, seed peers and back-to-source; includes integrations for Hugging Face model downloads.L7Lets you compare a centrally scheduled P2P system with Kraken and BitTorrent and say which one recovers without a coordinator.
- 2026Spegel maintainers · 2026 · Tool · Core · ~1 hP2P image sharing inside Kubernetes with no central state: nodes advertise layers they hold via a DHT and peers pull from each other.L7A small, readable example of coordinator-free distribution you can describe end to end.
- 2024Hugging Face Xet team · Nov 2024 · Blog post · Core · ~0.5 hContent-defined chunking and chunk-level dedup for model files on the Hugging Face Hub.L7Lets you justify content-addressed chunks (dedup across versions, resumable, verifiable) as the unit of distribution.
- 2025NVIDIA (Dynamo project) · 2025 · Tool · Core · ~1 hRust service that caches model weights and moves them GPU-to-GPU from initialized replicas to new ones in a serving cluster.L7Shows how a current inference stack solves 'new replica needs 500 GB fast'; use it as a reference design.
- 2026Kubernetes project · 2026 · Reference · Intro · ~1 hRolling updates, maxSurge/maxUnavailable, rollbacks and readiness gates.L7Baseline; use readiness to guarantee traffic only reaches instances that loaded the right weights.
- 2016Google · 2016 · Book · Core · ~1 hHermetic builds, staged rollouts, canaries and configuration management.L7Lets you structure a staged weight rollout with canaries and automatic rollback.
- 2019
- 3
Data pipelines and checkpoints
Build a loader that saturates GPUs and checkpoints that restore elastically · Week 3 (8–10 h)
- 2026Thomas Breuel / WebDataset maintainers · 2026 · Tool · Core · ~1.5 hTar-shard datasets read sequentially from object storage, with shard-level shuffling and buffer shuffles.L7Lets you explain why sequential shard reads beat random file reads at 1,024 GPUs and what shuffling quality you trade.
- 2026MosaicML (Databricks) · 2026 · Reference · Core · ~2 hStreaming dataset format with deterministic, elastic shuffling and mid-epoch resumption across changing numbers of nodes.L7Lets you answer 'resume mid-epoch on a different node count with the same sample order' — a precise staff-level probe.
- 2026PyTorch · 2026 · Reference · Intro · ~1 hMap vs iterable datasets, workers, samplers, pinned memory and prefetching.L7Baseline; skim if known. Know worker sharding for iterable datasets and DistributedSampler semantics.
- 2026Meta / PyTorch · 2026 · Tool · Core · ~1 hCheckpointable data loader state and composable loading nodes for exact resumption.L7Lets you include loader state in the checkpoint so a restart neither repeats nor skips data.
- 2026Anyscale / Ray · 2026 · Reference · Core · ~2 hStreaming execution across heterogeneous CPU/GPU workers for preprocessing and feeding training, with backpressure.L7Lets you describe a disaggregated preprocessing tier with backpressure into trainers.
- 2026NVIDIA · 2026 · Reference · Intro · ~1 hGPU-accelerated decoding and augmentation to remove CPU bottlenecks in image/video pipelines.L7Baseline; mention when CPU decode is the bottleneck.
- 2026PyTorch · 2026 · Reference · Core · ~1 hSharded save/load with resharding across different world sizes, plus async saving.L7Lets you explain resharding on load (save on 1,024 GPUs, restore on 768) and async staging to CPU.
- 2024PyTorch team (IBM, Meta) · Jun 2024 · Blog post · Core · ~0.5 hMeasured checkpoint stall times for 7B–70B training before and after async DCP.L7Gives you numbers for the training stall each checkpoint costs, needed to argue checkpoint frequency.
- 2026
- 4
Cluster realities
Scheduling, preemption, stragglers and failure handling on real fleets · Week 4 (8–10 h)
- 2026Stas Bekman · 2026 · Book · Advanced · ~20 hPractitioner handbook from BLOOM/IDEFICS training: compute, storage, network benchmarking, fault tolerance, SLURM, debugging and checkpointing at scale.L7Lets you speak from the failure modes real large runs hit (hung NCCL, slow nodes, full disks) instead of textbook ones.
- 2026Kubernetes project · 2026 · Reference · Core · ~1 hPriority classes, preemption and graceful termination semantics in Kubernetes.L7Baseline for preemption signals; know grace periods and how a trainer learns it is being evicted.
- 2026Kubernetes SIG Scheduling · 2026 · Reference · Core · ~2 hJob queueing for Kubernetes: ClusterQueues with quotas, cohorts and borrowing, fair sharing, preemption and topology-aware scheduling.L7Lets you design quota/borrowing/credit semantics for a GPU job queue on a real API.
- 2026SchedMD · 2026 · Reference · Core · ~2 hThe scheduler most frontier training clusters use: partitions, multifactor priority, fair-share and preemption.L7Lets you compare HPC-style Slurm scheduling with Kubernetes for training and describe fair-share credits.
- 2024Engineering at Meta · Jun 2024 · Blog post · Core · ~0.5 hHardware reliability, fast recovery, efficient state preservation and network choices for Meta's GenAI training.L7Short frontier-scale framing of failure, recovery and checkpointing to open a cluster-management answer.
- 2026NVIDIA · 2026 · Tool · Core · ~1 hHang detection, in-job restart, straggler detection and local checkpointing for PyTorch/Megatron training.L7Shows concretely what 'detect a hung rank and restart in place' looks like in production tooling.
- 2025Thinking Machines Lab · Oct 2025 · Blog post · Core · ~0.5 hA managed fine-tuning API where many users' LoRA jobs share pooled GPUs; describes the abstraction and what the service handles.L7Gives a frontier-lab reference product for a multi-tenant fine-tuning service and its API boundary.
- 2022Chip Huyen (O'Reilly) · May 2022 · Book · Core · ~12 h · paidEnd-to-end ML system design: data pipelines, feature stores, deployment, monitoring and continual learning.L7Mid-level MLOps coverage; read the data and deployment chapters for an 'MLOps round', skip the rest.
- 2026
- 5
Senior-staff depth
Critique and cite frontier/hyperscale systems; run the 5,000-host and 1,024-GPU mocks · Weeks 4–6, interleaved (20–25 h)
- 2022Jason Flinn et al. (Meta), OSDI · Jul 2022 · Paper · Advanced · ~3 hMeta's system for distributing hot binaries, models and packages to millions of hosts: P2P data plane with a centralized control plane and per-use-case policies.L7Lets you defend 'smart control plane, dumb peers' against fully decentralized designs with production numbers from Meta.
- 2025Dingyan Zhang et al. (SJTU), OSDI · Jul 2025 · Paper · Advanced · ~2 hLoads weights from already-running instances over the compute network (RDMA/NVLink) instead of storage, multicast-style, during live scale-up.L7Lets you propose GPU-to-GPU weight transfer as the fast path and explain why storage bandwidth is the wrong bottleneck to fight.
- 2024Yao Fu et al. (Edinburgh), OSDI · Jul 2024 · Paper · Advanced · ~2 hFast multi-tier checkpoint loading (SSD/DRAM to GPU), a loading-optimized checkpoint format and locality-aware placement.L7Gives you numbers for the last hop (disk → GPU), which often dominates once the network is solved.
- 2020Huiba Li et al. (Alibaba), USENIX ATC · Jul 2020 · Paper · Advanced · ~2 hLazy, block-level image loading with P2P tree distribution across thousands of hosts at Alibaba.L7Lets you argue lazy loading vs full pre-pull and tree fan-out vs swarm for startup-latency-sensitive deploys.
- 2002Petar Maymounkov, David Mazières, IPTPS · 2002 · Paper · Advanced · ~1.5 hXOR-distance DHT with k-buckets; how peers find who holds a chunk without any central tracker.L7Lets you answer 'the coordinator is gone, how do peers still find chunks?' with a mechanism rather than a hope.
- 2023Andy Warfield (AWS), via Werner Vogels' blog · Jul 2023 · Blog post · Advanced · ~1 hHow S3 spreads load over millions of drives, heat management and why per-object throughput is bounded.L7Lets you explain where object-storage throughput limits come from, not just quote them.
- 2025DeepSeek · Feb 2025 · Tool · Advanced · ~2 hDisaggregated SSD + RDMA file system with CRAQ replication used for DeepSeek's training data, checkpoints and KV cache (multi-TB/s reads).L7A frontier-lab storage design you can cite when arguing a shared parallel FS vs object storage for training and checkpoints.
- 2021Derek Murray et al. (Google), VLDB · Jan 2021 · Paper · Advanced · ~2 hDesign and fleet-wide analysis of Google's input pipelines: parallelism autotuning, caching, prefetching and how often input is the bottleneck.L7Gives you data showing input-bound jobs are common and the levers that fix them, which grounds a 'keep GPUs fed' answer.
- 2021Mark Zhao et al. (Meta), ISCA · Aug 2021 · Paper · Advanced · ~2 hMeta's disaggregated data preprocessing service: preprocessing can need more power than training, so it runs on separate CPU fleets.L7Lets you argue disaggregated preprocessing workers vs on-host DataLoader workers with Meta's numbers.
- 2023Zhuang Wang et al. (Rice, AWS), SOSP · Oct 2023 · Paper · Advanced · ~2 hCheckpoints to peer CPU memory with placement and traffic scheduling so recovery is 13x faster than remote storage.L7Lets you propose in-memory peer checkpoints plus slower durable ones, and explain the correlated-failure risk.
- 2025Borui Wan et al. (ByteDance), NSDI · Apr 2025 · Paper · Advanced · ~2 hParallelism-agnostic checkpoint representation with automatic resharding and async I/O across ByteDance's training fleet.L7Gives production evidence for decoupling checkpoint layout from parallelism, a likely follow-up after 'how do you resume elastically?'.
- 2024Ziheng Jiang et al. (ByteDance), NSDI 2024 · Feb 2024 · Paper · Advanced · ~3 hByteDance's 12,288-GPU training system: diagnostics, straggler detection, fast checkpoint/recovery and network tuning, with incident data.L7Lets you discuss detection-to-recovery time at 10k+ GPUs with real numbers, which is the core of 100K-GPU cluster questions.
- 2024Llama Team (Meta AI) · Jul 2024 · Paper · Advanced · ~3 hSection 3.3 covers the 16K-H100 cluster, storage (Tectonic, 2–7 TB/s checkpoint bursts), network and 466 job interruptions in 54 days with root causes.L7Lets you quote failure rates and checkpoint bandwidth from a real frontier run when sizing recovery.
- 2024Apostolos Kokolis et al. (Meta) · Oct 2024 · Paper · Advanced · ~2 hFailure statistics from two Meta research clusters (24k+ GPUs), mean time to failure by job size and the effective training time ratio.L7Lets you predict how often a 4,096-GPU job will fail and set checkpoint cadence from that.
- 2024Engineering at Meta · Jun 2024 · Blog post · Advanced · ~0.5 hRolling maintenance of GPU clusters (firmware, kernels, drivers) in small slices while training continues, with consistency checks across hosts.L7Lets you describe landing infra changes on a live training fleet without stalling jobs.
- 2016Ramesh Govindan et al. (Google), SIGCOMM · Aug 2016 · Paper · Advanced · ~2 hAnalysis of 100+ Google network failures; most occur during management operations, leading to principles for safe change (canaries, rollback, drain).L7Gives you the evidence-based playbook for landing a fabric routing change safely.
- 2015Abhishek Verma et al. (Google), EuroSys · Apr 2015 · Paper · Advanced · ~2 hPriorities, quotas, preemption, admission control and cell sizing in Google's cluster manager.L7Lets you design priority/quota/preemption semantics for a GPU job queue from a system that has run them for a decade.
- 2021OpenAI · Jan 2021 · Blog post · Advanced · ~1 hOpenAI's large training clusters on Kubernetes: etcd, API server load, networking, gang scheduling and health checks for GPU nodes.L7Frontier-lab evidence for what breaks in Kubernetes at thousands of GPU nodes and how they patched it.
- 2025Anthropic · Sep 2025 · Blog post · Advanced · ~0.5 hThree overlapping production bugs (context-window routing error, TPU misconfiguration, compiler bug) across hardware platforms, and changes to evaluation and detection.L7A frontier-lab example of why routing must only send traffic to correctly loaded, validated instances, and how hard detection is.
- 2022
What the interviews ask
Problems that recur in this track across 21 reports (9 firsthand), with the resources that prepare you for each.
| Reports | Problem | Prepare with |
|---|---|---|
| 11 | Distribute model weights to thousands of machines4 firsthand · example report | Kraken: P2P Docker registryOwl: Scale and Flexibility in Distribution of Hot ContentHow To Scale Your ModelMachine Learning Engineering Open BookBest practices design patterns: optimizing Amazon S3 performance |
Question coverage
Every question group collected for this topic from candidate reports, mapped to the resources that prepare you for it. 17 of 17 groups are covered, which is 100% of all report mentions. Hover an example to see the full question.
Show the other 0 question groups
All resources
Everything collected for this topic, newest first. 2026 items are marked green. ★ marks a top pick.
Type
Level
Subtopic
- 2026Percy Liang, Tatsunori Hashimoto (Stanford) · Apr 2026 · Course · Core · ~40 h · videoBuilds an LM stack from tokenizer to distributed training; the systems, parallelism and data lectures show how training data and GPUs are kept busy.L7Watch the GPU, parallelism and data lectures; they let you reason about where loader or network bottlenecks sit in a real training step.Site shows Spring 2026; public videos are the 2025 offering.
- 2026Tianqi Chen, Zhihao Jia (CMU) · Jan 2026 · Course · Core · ~30 hML systems course covering frameworks, GPU programming, distributed training and LLM serving.L7Use for the distributed-training and serving lectures; recordings are not public.Spring 2026 offering; no public videos found.
- 2026Stas Bekman · 2026 · Book · Advanced · ~20 hPractitioner handbook from BLOOM/IDEFICS training: compute, storage, network benchmarking, fault tolerance, SLURM, debugging and checkpointing at scale.L7Lets you speak from the failure modes real large runs hit (hung NCCL, slow nodes, full disks) instead of textbook ones.Continuously updated.
- 2026Dragonfly maintainers (CNCF, originally Alibaba) · 2026 · Reference · Core · ~2 hP2P file and image distribution with a scheduler, seed peers and back-to-source; includes integrations for Hugging Face model downloads.L7Lets you compare a centrally scheduled P2P system with Kraken and BitTorrent and say which one recovers without a coordinator.
- 2026Spegel maintainers · 2026 · Tool · Core · ~1 hP2P image sharing inside Kubernetes with no central state: nodes advertise layers they hold via a DHT and peers pull from each other.L7A small, readable example of coordinator-free distribution you can describe end to end.
- 2026Hugging Face · 2026 · Reference · Intro · ~0.5 hWeight file format with a JSON header and zero-copy, memory-mapped loading.L7Baseline; know that headers allow parallel ranged reads per tensor or shard.
- 2026NVIDIA · 2026 · Reference · Core · ~1 hBroadcast, all-gather and other collectives with ring/tree algorithms; the in-cluster alternative to P2P for weight fan-out.L7Lets you compare ring/tree broadcast bandwidth with P2P swarm and pick per network tier.
- 2026Amazon Web Services · 2026 · Reference · Core · ~0.5 hPer-prefix request rates (3,500 PUT / 5,500 GET per second), parallel ranged GETs, retries and scaling across prefixes.L7Gives the concrete object-storage limits that force you to add P2P or a cache tier when 5,000 hosts start at once.
- 2026Thomas Breuel / WebDataset maintainers · 2026 · Tool · Core · ~1.5 hTar-shard datasets read sequentially from object storage, with shard-level shuffling and buffer shuffles.L7Lets you explain why sequential shard reads beat random file reads at 1,024 GPUs and what shuffling quality you trade.
- 2026MosaicML (Databricks) · 2026 · Reference · Core · ~2 hStreaming dataset format with deterministic, elastic shuffling and mid-epoch resumption across changing numbers of nodes.L7Lets you answer 'resume mid-epoch on a different node count with the same sample order' — a precise staff-level probe.
- 2026PyTorch · 2026 · Reference · Intro · ~1 hMap vs iterable datasets, workers, samplers, pinned memory and prefetching.L7Baseline; skim if known. Know worker sharding for iterable datasets and DistributedSampler semantics.
- 2026Meta / PyTorch · 2026 · Tool · Core · ~1 hCheckpointable data loader state and composable loading nodes for exact resumption.L7Lets you include loader state in the checkpoint so a restart neither repeats nor skips data.Repo now lives under meta-pytorch/data.
- 2026Anyscale / Ray · 2026 · Reference · Core · ~2 hStreaming execution across heterogeneous CPU/GPU workers for preprocessing and feeding training, with backpressure.L7Lets you describe a disaggregated preprocessing tier with backpressure into trainers.
- 2026NVIDIA · 2026 · Reference · Intro · ~1 hGPU-accelerated decoding and augmentation to remove CPU bottlenecks in image/video pipelines.L7Baseline; mention when CPU decode is the bottleneck.
- 2026PyTorch · 2026 · Reference · Core · ~1 hSharded save/load with resharding across different world sizes, plus async saving.L7Lets you explain resharding on load (save on 1,024 GPUs, restore on 768) and async staging to CPU.
- 2026NVIDIA · 2026 · Tool · Core · ~1 hHang detection, in-job restart, straggler detection and local checkpointing for PyTorch/Megatron training.L7Shows concretely what 'detect a hung rank and restart in place' looks like in production tooling.
- 2026Kubernetes project · 2026 · Reference · Core · ~1 hPriority classes, preemption and graceful termination semantics in Kubernetes.L7Baseline for preemption signals; know grace periods and how a trainer learns it is being evicted.
- 2026Kubernetes SIG Scheduling · 2026 · Reference · Core · ~2 hJob queueing for Kubernetes: ClusterQueues with quotas, cohorts and borrowing, fair sharing, preemption and topology-aware scheduling.L7Lets you design quota/borrowing/credit semantics for a GPU job queue on a real API.
- 2026SchedMD · 2026 · Reference · Core · ~2 hThe scheduler most frontier training clusters use: partitions, multifactor priority, fair-share and preemption.L7Lets you compare HPC-style Slurm scheduling with Kubernetes for training and describe fair-share credits.
- 2026Kubernetes project · 2026 · Reference · Intro · ~1 hRolling updates, maxSurge/maxUnavailable, rollbacks and readiness gates.L7Baseline; use readiness to guarantee traffic only reaches instances that loaded the right weights.
- 2026Anyscale / Ray · 2026 · Reference · Intro · ~1 hDistributed training orchestration with fault tolerance and checkpoint hooks on Ray.L7Baseline; an example orchestration layer for a fine-tuning service.
- 2025Thinking Machines Lab · Oct 2025 · Blog post · Core · ~0.5 hA managed fine-tuning API where many users' LoRA jobs share pooled GPUs; describes the abstraction and what the service handles.L7Gives a frontier-lab reference product for a multi-tenant fine-tuning service and its API boundary.
- 2025Anthropic · Sep 2025 · Blog post · Advanced · ~0.5 hThree overlapping production bugs (context-window routing error, TPU misconfiguration, compiler bug) across hardware platforms, and changes to evaluation and detection.L7A frontier-lab example of why routing must only send traffic to correctly loaded, validated instances, and how hard detection is.
- 2025Dingyan Zhang et al. (SJTU), OSDI · Jul 2025 · Paper · Advanced · ~2 hLoads weights from already-running instances over the compute network (RDMA/NVLink) instead of storage, multicast-style, during live scale-up.L7Lets you propose GPU-to-GPU weight transfer as the fast path and explain why storage bandwidth is the wrong bottleneck to fight.
- 2025Borui Wan et al. (ByteDance), NSDI · Apr 2025 · Paper · Advanced · ~2 hParallelism-agnostic checkpoint representation with automatic resharding and async I/O across ByteDance's training fleet.L7Gives production evidence for decoupling checkpoint layout from parallelism, a likely follow-up after 'how do you resume elastically?'.
- 2025Jacob Austin, Sholto Douglas et al. (Google DeepMind) · Feb 2025 · Book · Advanced · ~15 hRoofline analysis, TPU/GPU networking, parallelism strategies and serving maths for LLMs, written by DeepMind researchers.L7Gives you the bandwidth and memory arithmetic (bytes/s per link, HBM, ICI/NVLink vs DCN) to put real numbers on any distribution or loader design.Updated through 2026.
- 2025Nouamane Tazi et al. (Hugging Face) · Feb 2025 · Tutorial · Core · ~10 hInteractive guide to data, tensor, pipeline, context and expert parallelism with measured throughput on up to 512 GPUs.L7Baseline for the parallelism vocabulary interviewers assume; skim if you already know FSDP/TP/PP trade-offs.
- 2025DeepSeek · Feb 2025 · Tool · Advanced · ~2 hDisaggregated SSD + RDMA file system with CRAQ replication used for DeepSeek's training data, checkpoints and KV cache (multi-TB/s reads).L7A frontier-lab storage design you can cite when arguing a shared parallel FS vs object storage for training and checkpoints.
- 2025Hao Zhang (UC San Diego) · Jan 2025 · Course · Core · ~30 hLectures on GPUs, parallelization, collective communication, memory and LLM serving, with recordings linked from the site.L7The collective-communication and parallelization lectures support broadcast/all-gather arithmetic in distribution designs.Recordings hosted on podcast.ucsd.edu; access not verified.
- 2025NVIDIA (Dynamo project) · 2025 · Tool · Core · ~1 hRust service that caches model weights and moves them GPU-to-GPU from initialized replicas to new ones in a serving cluster.L7Shows how a current inference stack solves 'new replica needs 500 GB fast'; use it as a reference design.Date approximate.
- 2024Hugging Face Xet team · Nov 2024 · Blog post · Core · ~0.5 hContent-defined chunking and chunk-level dedup for model files on the Hugging Face Hub.L7Lets you justify content-addressed chunks (dedup across versions, resumable, verifiable) as the unit of distribution.
- 2024Wanchao Liang et al. (Meta), ICLR 2025 · Oct 2024 · Paper · Advanced · ~2 hPyTorch-native LLM training stack composing FSDP2, TP, PP, async DCP checkpointing and fault-tolerance hooks, with measured results.L7Lets you describe how checkpointing, parallelism and restarts fit in one modern training stack.
- 2024Apostolos Kokolis et al. (Meta) · Oct 2024 · Paper · Advanced · ~2 hFailure statistics from two Meta research clusters (24k+ GPUs), mean time to failure by job size and the effective training time ratio.L7Lets you predict how often a 4,096-GPU job will fail and set checkpoint cadence from that.
- 2024DeepSeek-AI · Aug 2024 · Paper · Advanced · ~2 hDeepSeek's 10,000-GPU cluster: network topology, HFReduce, 3FS storage and failure handling.L7Lets you discuss cost-driven cluster design trade-offs a frontier lab actually made.
- 2024Engineering at Meta (SIGCOMM 2024) · Aug 2024 · Blog post · Advanced · ~1 hMeta's RoCE training fabric: topology, routing, congestion control and operational lessons from 24k-GPU clusters.L7Lets you reason about routing and congestion changes on an AI fabric and their effect on collectives.
- 2024Yao Fu et al. (Edinburgh), OSDI · Jul 2024 · Paper · Advanced · ~2 hFast multi-tier checkpoint loading (SSD/DRAM to GPU), a loading-optimized checkpoint format and locality-aware placement.L7Gives you numbers for the last hop (disk → GPU), which often dominates once the network is solved.
- 2024Llama Team (Meta AI) · Jul 2024 · Paper · Advanced · ~3 hSection 3.3 covers the 16K-H100 cluster, storage (Tectonic, 2–7 TB/s checkpoint bursts), network and 466 job interruptions in 54 days with root causes.L7Lets you quote failure rates and checkpoint bandwidth from a real frontier run when sizing recovery.Read section 3.3 (Infrastructure, Scaling, and Efficiency).
- 2024PyTorch team (IBM, Meta) · Jun 2024 · Blog post · Core · ~0.5 hMeasured checkpoint stall times for 7B–70B training before and after async DCP.L7Gives you numbers for the training stall each checkpoint costs, needed to argue checkpoint frequency.
- 2024Engineering at Meta · Jun 2024 · Blog post · Core · ~0.5 hHardware reliability, fast recovery, efficient state preservation and network choices for Meta's GenAI training.L7Short frontier-scale framing of failure, recovery and checkpointing to open a cluster-management answer.
- 2024Engineering at Meta · Jun 2024 · Blog post · Advanced · ~0.5 hRolling maintenance of GPU clusters (firmware, kernels, drivers) in small slices while training continues, with consistency checks across hosts.L7Lets you describe landing infra changes on a live training fleet without stalling jobs.
- 2024Yazhou Zu et al. (Google), NSDI · Apr 2024 · Paper · Advanced · ~2 hHow Google's optical circuit switches and control plane route around failed TPU cubes and keep large slices available.L7Lets you discuss scheduling around failure domains and reconfigurable topology at Google scale.
- 2024Ziheng Jiang et al. (ByteDance), NSDI 2024 · Feb 2024 · Paper · Advanced · ~3 hByteDance's 12,288-GPU training system: diagnostics, straggler detection, fast checkpoint/recovery and network tuning, with incident data.L7Lets you discuss detection-to-recovery time at 10k+ GPUs with real numbers, which is the core of 100K-GPU cluster questions.
- 2023Ying Sheng et al. (Berkeley, Stanford), MLSys 2024 · Nov 2023 · Paper · Advanced · ~2 hUnified paging of adapters and KV cache plus custom kernels to serve thousands of LoRA adapters on few GPUs.L7Lets you argue adapter-based multi-tenancy for fine-tuned models and its memory accounting.
- 2023Zhuang Wang et al. (Rice, AWS), SOSP · Oct 2023 · Paper · Advanced · ~2 hCheckpoints to peer CPU memory with placement and traffic scheduling so recovery is 13x faster than remote storage.L7Lets you propose in-memory peer checkpoints plus slower durable ones, and explain the correlated-failure risk.ACM DOI 10.1145/3600006.3613145 is bot-blocked.
- 2023Andy Warfield (AWS), via Werner Vogels' blog · Jul 2023 · Blog post · Advanced · ~1 hHow S3 spreads load over millions of drives, heat management and why per-object throughput is bounded.L7Lets you explain where object-storage throughput limits come from, not just quote them.
- 2023Goku Mohandas (Anyscale) · 2023 · Course · Intro · ~15 hCode-first MLOps course: data, distributed training and tuning with Ray, serving, CI/CD for models and monitoring.L7Baseline; gives a concrete pipeline to talk through in an MLOps round, not staff depth.Date approximate.
- 2022Charles Frye, Sergey Karayev, Josh Tobin · Aug 2022 · Course · Intro · ~15 h · videoRecorded lectures on data management, deployment, monitoring and ML teams.L7Baseline; skim if known. Useful only for an MLOps-named round.
- 2022Leon Poutievski et al. (Google), SIGCOMM · Aug 2022 · Paper · Advanced · ~2 hHow Google incrementally changes topology and routing of a live datacenter fabric with SDN and optical switches.L7Lets you describe incremental, drain-and-shift fabric changes without disrupting long-running jobs.
- 2022Jason Flinn et al. (Meta), OSDI · Jul 2022 · Paper · Advanced · ~3 hMeta's system for distributing hot binaries, models and packages to millions of hosts: P2P data plane with a centralized control plane and per-use-case policies.L7Lets you defend 'smart control plane, dumb peers' against fully decentralized designs with production numbers from Meta.
- 2022Chip Huyen (O'Reilly) · May 2022 · Book · Core · ~12 h · paidEnd-to-end ML system design: data pipelines, feature stores, deployment, monitoring and continual learning.L7Mid-level MLOps coverage; read the data and deployment chapters for an 'MLOps round', skip the rest.Paid book.
- 2022Assaf Eisenman et al. (Meta), NSDI · Apr 2022 · Paper · Advanced · ~1.5 hIncremental and quantized checkpoints for TB-scale recommendation models at Meta.L7Lets you cut checkpoint bandwidth with differential checkpoints and discuss correctness of quantized restores.
- 2022Qizhen Weng et al. (Alibaba), NSDI · Apr 2022 · Paper · Advanced · ~1.5 hTwo-month trace of Alibaba's 6,000-GPU cluster: GPU sharing, fragmentation and scheduling policies.L7Lets you discuss fragmentation and fractional GPU sharing in multi-tenant fine-tuning or job queues.
- 2022Chip Huyen (Stanford) · Jan 2022 · Course · Core · ~20 hCourse on production ML systems: data engineering, training data, deployment, monitoring and infrastructure, with slides and notes.L7Baseline MLOps structure; lectures are not public, so use the notes for vocabulary only.Winter 2022 is the latest offering; lecture videos were not publicly released.
- 2021Mark Zhao et al. (Meta), ISCA · Aug 2021 · Paper · Advanced · ~2 hMeta's disaggregated data preprocessing service: preprocessing can need more power than training, so it runs on separate CPU fleets.L7Lets you argue disaggregated preprocessing workers vs on-host DataLoader workers with Meta's numbers.
- 2021Satadru Pan et al. (Meta), FAST · Feb 2021 · Paper · Advanced · ~2 hExabyte-scale multitenant filesystem at Meta, including how data warehouse and blob workloads share one cluster.L7Lets you discuss multitenant storage isolation for training data and checkpoints at hyperscale.
- 2021Jayashree Mohan et al. (UT Austin, Microsoft), FAST · Feb 2021 · Paper · Advanced · ~1.5 hTwo-phase snapshot/persist checkpointing with frequency tuned to keep overhead bounded.L7Lets you derive checkpoint interval from failure rate and stall cost, instead of picking 'every hour'.
- 2021Derek Murray et al. (Google), VLDB · Jan 2021 · Paper · Advanced · ~2 hDesign and fleet-wide analysis of Google's input pipelines: parallelism autotuning, caching, prefetching and how often input is the bottleneck.L7Gives you data showing input-bound jobs are common and the levers that fix them, which grounds a 'keep GPUs fed' answer.
- 2021OpenAI · Jan 2021 · Blog post · Advanced · ~1 hOpenAI's large training clusters on Kubernetes: etcd, API server load, networking, gang scheduling and health checks for GPU nodes.L7Frontier-lab evidence for what breaks in Kubernetes at thousands of GPU nodes and how they patched it.openai.com blocks curl.
- 2020Huiba Li et al. (Alibaba), USENIX ATC · Jul 2020 · Paper · Advanced · ~2 hLazy, block-level image loading with P2P tree distribution across thousands of hosts at Alibaba.L7Lets you argue lazy loading vs full pre-pull and tree fan-out vs swarm for startup-latency-sensitive deploys.
- 2019Myeongjae Jeon et al. (Microsoft), USENIX ATC · Jul 2019 · Paper · Advanced · ~1.5 hTrace study of Microsoft's Philly cluster: gang scheduling, locality vs queueing delay and failure causes.L7Gives evidence for the locality-vs-wait trade-off when you design GPU queue placement.
- 2019Uber Engineering · Mar 2019 · Tool · Advanced · ~3 hProduction P2P blob distribution (tens of TB, thousands of hosts) with content-addressed pieces, a tracker, origin cluster and bandwidth-aware peer selection.L7The closest open design to the interview prompt; lets you argue tracker vs DHT, origin rate limits and piece size with a real system.Uber's announcement blog blocks curl (406); the repo README and docs carry the architecture.
- 2016Ramesh Govindan et al. (Google), SIGCOMM · Aug 2016 · Paper · Advanced · ~2 hAnalysis of 100+ Google network failures; most occur during management operations, leading to principles for safe change (canaries, rollback, drain).L7Gives you the evidence-based playbook for landing a fabric routing change safely.
- 2016Google · 2016 · Book · Core · ~1 hHermetic builds, staged rollouts, canaries and configuration management.L7Lets you structure a staged weight rollout with canaries and automatic rollback.
- 2015Abhishek Verma et al. (Google), EuroSys · Apr 2015 · Paper · Advanced · ~2 hPriorities, quotas, preemption, admission control and cell sizing in Google's cluster manager.L7Lets you design priority/quota/preemption semantics for a GPU job queue from a system that has run them for a decade.Older but canonical.
- 2013Malte Schwarzkopf et al. (Google), EuroSys · Apr 2013 · Paper · Advanced · ~2 hShared-state optimistic concurrency scheduling vs monolithic and two-level schedulers.L7Lets you argue scheduler architecture for 100k-GPU fleets: one scheduler, many with conflicts, or two-level.Older but canonical.
- 2008Bram Cohen (BitTorrent.org) · 2008 · Reference · Core · ~1 hPiece hashing, metainfo files, tracker protocol and peer wire protocol; the base vocabulary for chunked P2P distribution.L7Baseline; read it so piece size, rarest-first and tit-for-tat are concrete when you adapt them to a trusted datacenter.Older but canonical.
- 2002Petar Maymounkov, David Mazières, IPTPS · 2002 · Paper · Advanced · ~1.5 hXOR-distance DHT with k-buckets; how peers find who holds a chunk without any central tracker.L7Lets you answer 'the coordinator is gone, how do peers still find chunks?' with a mechanism rather than a hope.Older but canonical.
Not found or not verified. No Anthropic or OpenAI engineering post on distributing model weights was found; Meta's Owl (OSDI'22) is the only frontier/hyperscale first-party paper on the problem. Uber's Kraken announcement blog blocks curl (406), so the GitHub repo stands in. No 2025–2026 MLOps course with public videos was found: Stanford CS329S last ran Winter 2022 without public recordings, CMU 15-442 (Spring 2026) has no public videos, and UCSD CSE 234 recordings are on podcast.ucsd.edu with access not checked. OpenAI pages (Kubernetes 7,500 nodes) return 403 to curl and are marked exists-per-search. Nothing public on xAI's 100K-GPU cluster management; MegaScale, Llama 3 and Google TPUv4 resiliency stand in. GPU 'credits' scheduling has no single write-up; covered by Kueue fair sharing/borrowing and Slurm fair-share. The ACM page for the Gemini (SOSP'23) paper is bot-blocked; Amazon Science page and PDF used instead. Links checked 27 Oct 2026.