LLM inference and serving design

The senior-staff bar

What L7-level depth looks like in this topic, the follow-up an interviewer would push on, and the resources that prepare you for it.

• Capacity math from roofline to GPU countCompute the theoretical max decode throughput of a 70B model on one 8×H100 node, then what you actually get under a 50 ms TPOT SLO, and how many nodes 1,000 rps needs.
  How to Scale Your Model — 'All About Transformer Inference' (with the Rooflines chapter)Transformer Inference ArithmeticEfficiently Scaling Transformer InferenceLLM Inference Performance Engineering: Best PracticesVidur: A Large-Scale Simulation Framework For LLM Inference
• KV-cache budget and the 'it doesn't fit' playbook70B at 128k context OOMs at batch 32. Size the cache per token, then rank GQA/MLA, FP8 KV, paging/preemption, offload and context parallelism by cost and quality risk.
  Efficient Memory Management for Large Language Model Serving with PagedAttention (vLLM)GQA: Training Generalized Multi-Query Transformer Models from Multi-Head CheckpointsDeepSeek-V2 (Multi-head Latent Attention)From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM ServingThe State of FP8 KV-Cache and Attention Quantization in vLLMTiered KV Cache Offloading in vLLM
• Scheduling for goodput under separate TTFT and TPOT SLOsTTFT is fine but inter-token latency grows with conversation length and p99 doubles at peak. Where does the time go, and when do you choose chunked prefill over disaggregation?
  Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (chunked prefill)DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM ServingTaking vLLM Apart: A Practical Guide to Disaggregated ServingFlash-Decoding for long-context inferencevLLM Production Metrics
• Router and cache-aware load balancingYou add a router in front of 200 replicas. What state does it hold, how do you trade prefix-cache affinity against load, and what happens with a hot system prompt or a replica failure mid-stream?
  KV-Cache Wins You Can See: from prefix caching in vLLM to distributed scheduling with llm-dvLLM Router: A High-Performance and Prefill/Decode Aware Load Balancer for Large-scale ServingSGLang: Efficient Execution of Structured Language Model Programs (RadixAttention)Optimizing AI Inference at Character.AILlumnix: Dynamic Scheduling for Large Language Model Serving
• Batch API economics, fairness and cancellationDesign a 24h-SLA batch API that shares GPUs with online traffic at a 50% discount. How do you preempt, fair-share across tenants, cancel in-flight work and prove the SLA holds during a capacity dip?
  Batch processing (Message Batches API)Fairness in Serving Large Language Models (Virtual Token Counter)Interview notes: Video Generation Pipeline (Sora-style)Ray Data: Working with LLMs (batch inference)Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving
• Cost per token and what not to buildWhere does the money go serving a 70B dense model versus a DeepSeek-class MoE to 100M users, and which parts would you buy (engine, router, KV store) rather than build?
  DeepSeek-V3/R1 Inference System OverviewInferenceMAX: Open Source Inference BenchmarkingDeploying DeepSeek with PD Disaggregation and Large-Scale Expert Parallelism on 96 H100 GPUsSplitwise: Efficient generative LLM inference using phase splittingTensorRT-LLM
• Correctness and reliability of the serving fleetA deploy changed model outputs but not latency, or an EP group lost a GPU. How do you detect it, contain the blast radius and keep evals trustworthy given batch nondeterminism?
  A postmortem of three recent issuesDefeating Nondeterminism in LLM InferenceElastic EP in SGLang: Achieving Partial Failure Tolerance for DeepSeek MoE Deployments"Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantization
• Critiquing someone else's inference designHere is a half-finished design doc for a batched inference service. Tear it apart: which three issues matter most, and which numbers are wrong?
  Interview notes: Design Doc Review: A Batched Inference ServiceDesign an LLM Inference API (techinterview.org) — use as a critique exercisePrinceton COS 597A: Efficient Systems for Foundation Models (Fall 2025)

Start here

The best resources for a time-poor senior engineer, newest first.

Learning path

Work through the stages in order. The last stage is the senior-staff layer: production case studies, papers to critique and large-scale trade-offs. Each resource's L7 line says what it adds at your level.

1. 1

   Foundations: inference arithmetic

   Derive prefill vs decode bounds, KV bytes per token and tokens/s per GPU from first principles · Week 1

   • 2026
     Stanford CS336 (Spring 2026), Lecture 10: Inference

     Percy Liang, Tatsunori Hashimoto (Stanford) · Apr 2026 · Lecture · Core · ~2.5 h · video

     An 85-minute lecture that derives why prefill is compute-bound and decode is memory-bound, sizes the KV cache, and walks through the main ways to make decoding cheaper: GQA/MLA, quantization, speculative decoding, and paging/batching. The executable lecture trace lets you rerun the numbers yourself.

     L7Baseline for an ML engineer new to LLMs: watch it once so the arithmetic is automatic, then spend your time on the systems papers.

     Course site (Spring 2026)Lecture 10 executable traceYouTube playlist (Spring 2026)
   • 2022
     ★Transformer Inference Arithmetic

     kipply (Carol Chen) · Mar 2022 · Blog post · Core · ~1.5 h

     The classic short derivation of KV-cache size (2·layers·d_model·bytes per token), memory-bandwidth-bound latency, the FLOPs-to-bytes ratio, and when batching becomes compute-bound, worked through for a 52B model.

     L7Older but canonical. Learn its back-of-envelope method until you can size a 70B GQA model at 32k context in under two minutes; the candidates who can't usually stall on this.
   • 2022
     Making Deep Learning Go Brrrr From First Principles

     Horace He · Mar 2022 · Blog post · Intro · ~1 h

     Explains compute-bound, memory-bandwidth-bound and overhead-bound regimes using clear GPU examples. It is the intuition the roofline model relies on.

     L7Baseline; read it if 'memory-bound' is not yet intuitive. The overhead-bound section explains why CUDA graphs matter for small-batch decode.
   • 2025
     ★How to Scale Your Model — 'All About Transformer Inference' (with the Rooflines chapter)

     Jacob Austin, Sholto Douglas, Roy Frostig, Anselm Levskaya et al. (Google DeepMind) · Feb 2025 · Book · Advanced · ~8 h

     The best single derivation of inference cost: the roofline for prefill and decode, the batch size at which decode turns compute-bound, KV-cache bytes per token, latency versus throughput Pareto curves, and sharding a model for serving. It includes worked problems with answers.

     L7Lets you compute 'theoretical max throughput, then actual under a latency SLO' on the whiteboard, which is the exact Anthropic phone-screen prompt, and defend your sharding choice with numbers.

     All About Rooflines chapterBook home
   • 2023
     LLM Inference Performance Engineering: Best Practices

     Databricks Mosaic AI · Oct 2023 · Blog post · Core · ~1 h

     Defines TTFT, TPOT, latency and throughput, introduces MBU (model bandwidth utilization) as the decode efficiency metric, and measures the effect of batch size and tensor-parallel degree on latency and cost.

     L7MBU is the number to quote when you say decode is memory-bound. This post shows how to measure it and what good values look like.
   • 2023
     GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints

     Joshua Ainslie et al. (Google Research) · May 2023 · Paper · Core · ~1 h

     Grouped-query attention: share K/V heads across groups of query heads, cutting KV-cache size by n_heads/n_kv_heads (8x for Llama-70B) at little quality cost, with uptraining from MHA checkpoints.

     L7Baseline, but you must put the GQA factor into every KV-cache estimate. Getting it wrong is a common error on '70B at 128k' questions.
   • 2025
     KV Cache from scratch in nanoVLM

     Hugging Face · Jun 2025 · Tutorial · Intro · ~1 h

     Implements a KV cache in a small PyTorch model, showing why past K and V are reusable while the query is only needed for the current token, and how prefill and decode differ in code.

     L7Baseline; answers 'why cache K and V but never Q' in code. Skip if obvious.
   • 2023
     Accelerating Generative AI with PyTorch II: GPT, Fast

     Horace He et al. (PyTorch team, Meta) · Nov 2023 · Tutorial · Core · ~3 h

     Takes Llama-7B decode from 25 to about 240 tokens/s in under 1,000 lines of PyTorch using torch.compile, static KV cache, INT8/INT4 weight-only quantization, speculative decoding and tensor parallelism, measuring against the bandwidth roofline at each step.

     L7Hands-on proof of the decode roofline: reproduce the numbers and you can explain each optimisation's speedup from first principles.

     gpt-fast repo
2. 2

   Serving toolbox

   Know how continuous batching, paging, prefix caching, chunked prefill, disaggregation, speculation, quantization and decode kernels work and what each costs · Weeks 2–3

   • 2023
     How continuous batching enables 23x throughput in LLM inference while reducing p50 latency

     Cade Daniel et al. (Anyscale) · Jun 2023 · Blog post · Intro · ~1 h

     The standard explainer of static, dynamic and continuous batching, with throughput and latency benchmarks across vLLM, TGI and Ray Serve.

     L7Baseline, but its diagrams are the clearest way to explain the idea in 60 seconds before going deeper.
   • 2022
     Orca: A Distributed Serving System for Transformer-Based Generative Models

     Gyeong-In Yu et al. (Seoul National Univ. / FriendliAI), OSDI 2022 · Jul 2022 · Paper · Advanced · ~2.5 h

     The origin of iteration-level scheduling (continuous batching) and selective batching: the scheduler re-forms the batch at every decode step instead of waiting for the slowest sequence.

     L7Lets you explain why static batching wastes GPU time and what continuous batching changes in the scheduler loop, which is what the Fellows OA and xAI coding rounds ask you to implement.

     Talk video and PDF (USENIX)
   • 2023
     ★Efficient Memory Management for Large Language Model Serving with PagedAttention (vLLM)

     Woosuk Kwon, Zhuohan Li, Siyuan Zhuang et al. (UC Berkeley), SOSP 2023 · Sep 2023 · Paper · Advanced · ~3 h

     Shows that contiguous KV allocation wastes 60–80% of memory through fragmentation and over-reservation, and fixes it with OS-style block tables, copy-on-write sharing for parallel sampling and beam search, and preemption by swapping or recomputation.

     L7Lets you answer 'hundreds of requests with different context lengths, how do you manage KV memory' with block size, preemption policy and their trade-offs, which is the core KV interview probe.

     vLLM launch blogTGI PagedAttention explainer
   • 2025
     ★Inside vLLM: Anatomy of a High-Throughput LLM Inference System

     Aleksa Gordić (vLLM blog) · Sep 2025 · Blog post · Advanced · ~4 h

     A code-level walkthrough of vLLM V1: engine core and scheduler loop, paged KV manager, prefix caching via block hashes, chunked prefill, guided decoding, speculative decoding, disaggregated P/D via KV connectors, multi-GPU/multi-node scaling, and the API server and DP load balancer.

     L7The best single source for drawing a real engine's request lifecycle on the whiteboard and saying where queueing, preemption and cache misses happen. It is also the map for the Fellows OA's 'vLLM-like codebase'.

     Author's mirrorvLLM V1 architecture post
   • 2023
     SGLang: Efficient Execution of Structured Language Model Programs (RadixAttention)

     Lianmin Zheng, Liangsheng Yin, Ying Sheng et al. (Stanford / Berkeley / LMSYS), NeurIPS 2024 · Dec 2023 · Paper · Advanced · ~3 h

     Introduces RadixAttention: the KV cache is kept in a radix tree with LRU eviction so any shared prefix (system prompts, few-shot examples, multi-turn history) is reused automatically, plus cache-aware scheduling that orders requests by longest prefix match.

     L7Lets you quantify prefix-cache value (hit rate times prefill FLOPs saved) and explain the conflict between cache-aware ordering and fairness.

     LMSYS blog: RadixAttention and SGLangSGLang v0.4: zero-overhead scheduler and cache-aware load balancer
   • 2024
     Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (chunked prefill)

     Amey Agrawal, Nitin Kedia, Ashish Panwar et al. (Microsoft Research India / Georgia Tech), OSDI 2024 · Mar 2024 · Paper · Advanced · ~2.5 h

     Shows that long prefills stall in-flight decodes (generation stalls), and fixes this with chunked prefill plus stall-free batching under a per-iteration token budget. This is now the default in vLLM and SGLang.

     L7Lets you diagnose 'tail latency is terrible when long prompts arrive' and choose between chunked prefill and full disaggregation with a clear criterion.

     SARATHI (original chunked-prefill paper)OSDI talk
   • 2024
     ★DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving

     Yinmin Zhong, Shengyu Liu et al. (PKU / UCSD Hao Zhang), OSDI 2024 · Jan 2024 · Paper · Advanced · ~3 h

     Argues for goodput (requests per second meeting both TTFT and TPOT SLOs) as the serving objective, shows that prefill/decode interference forces over-provisioning, and places prefill and decode instances with separate parallelism configurations.

     L7Gives you the vocabulary and the decision rule (separate SLOs, interference, KV transfer cost) for the disaggregation trade-off, a near-certain senior-staff follow-up.

     Hao AI Lab blog: goodput and disaggregationOSDI talk
   • 2022
     Fast Inference from Transformers via Speculative Decoding

     Yaniv Leviathan, Matan Kalman, Yossi Matias (Google), ICML 2023 · Nov 2022 · Paper · Core · ~1.5 h

     The original speculative decoding paper: a draft model proposes tokens and the target verifies them in one forward pass, with a rejection-sampling rule that keeps the output distribution exact. It gives an expected speedup as a function of acceptance rate.

     L7Lets you derive when speculation helps (memory-bound decode, high acceptance) and when it hurts (large batches, compute-bound), which is the senior-staff caveat.

     Chen et al., speculative sampling (DeepMind)Jay Mody: speculative sampling explained with code
   • 2025
     EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test

     Yuhui Li, Fangyun Wei, Chao Zhang, Hongyang Zhang (Peking Univ. / Microsoft / Waterloo) · Mar 2025 · Paper · Advanced · ~2 h

     The current state-of-the-art draft architecture used by vLLM and SGLang: it predicts tokens directly using fused multi-layer features and training-time test, reaching about 3–6.5x speedups.

     L7The method production engines actually ship in 2025–26; lets you discuss draft training and acceptance-rate drift under distribution shift.

     EAGLE (v1)EAGLE-2 dynamic draft treesSGLang speculative decoding docs
   • 2023
     AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

     Ji Lin, Jiaming Tang et al. (MIT HAN Lab), MLSys 2024 best paper · Jun 2023 · Paper · Advanced · ~1.5 h

     Protects the roughly 1% of salient weight channels (chosen by activation magnitude) through per-channel scaling, giving robust INT4 weight-only quantization without backprop.

     L7Lets you compare AWQ and GPTQ for a customer's hardware choice: calibration cost, generalisation and kernel availability.

     QServe: W4A8KV4 system co-design
   • 2022
     SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

     Guangxuan Xiao, Ji Lin et al. (MIT / NVIDIA), ICML 2023 · Nov 2022 · Paper · Advanced · ~1.5 h

     Moves activation outliers into the weights with a per-channel scaling factor so both weights and activations can be INT8 (W8A8), which speeds up compute-bound prefill.

     L7Lets you explain 'what breaks at each step down': activation outliers are why W8A8 and FP8 activations need special handling.

     LLM.int8()
   • 2023
     Flash-Decoding for long-context inference

     Tri Dao, Daniel Haziza, Francisco Massa, Grigory Sizov (Stanford CRFM / Meta) · Oct 2023 · Blog post · Core · ~0.5 h

     Explains why FlashAttention underuses the GPU during decode (batch × heads is small), and splits the KV sequence across thread blocks with a final reduction, up to 8x faster for long contexts.

     L7Answers 'why does decode slow down as context grows': per-step attention reads grow linearly with context, and Flash-Decoding plus paged KV kernels are the fix.

     PyTorch blog mirror
   • 2023
     Mistral 7B (sliding-window attention and rolling buffer cache)

     Albert Q. Jiang et al. (Mistral AI) · Oct 2023 · Paper · Core · ~1 h

     Describes sliding-window attention with a fixed-size rolling buffer KV cache (position i mod W) and prefill chunking, the design Mistral interviewers ask about directly.

     L7Baseline; read the short sections on SWA, the rolling buffer and pre-fill chunking, then be ready to discuss how a rolling buffer interacts with paging and prefix caching.
3. 3

   Hands-on: serve and measure

   Serve a model with vLLM and SGLang, plot TTFT/ITL/throughput against load, turn on prefix caching, and modify a small engine · Week 3 (8–12 h)

   • 2026
     Fast & Efficient LLM Inference with vLLM (DeepLearning.AI short course)

     DeepLearning.AI with the vLLM team · Jun 2026 · Course · Intro · ~2 h · video

     A short hands-on video course on serving with vLLM: KV cache and paging, continuous batching, prefix caching and quantized serving, with notebooks. It is the quickest route to having a running server to measure.

     L7Baseline; skim if you already run vLLM. Its value is getting step 3 of the learning path (serve and measure) done in an evening.

     vLLM blog announcement
   • 2026
     vLLM documentation and benchmark suite

     vLLM project · 2026 · Tool · Core · ~6 h

     Serve a model, enable automatic prefix caching, speculative decoding and FP8, then run 'vllm bench serve' to plot TTFT, ITL and throughput against request rate and concurrency. This is step 3 of the learning path.

     L7Measured curves from your own runs (the knee where TTFT climbs as concurrency rises) give you real numbers to cite in a design round.

     Quickstartvllm bench serveBenchmark suitesAutomatic prefix caching designGitHub
   • 2026
     SGLang documentation

     SGLang project (LMSYS) · 2026 · Tool · Core · ~4 h

     Install, send requests, then try RadixAttention prefix reuse, PD disaggregation, HiCache and speculative decoding on the same model to compare with vLLM.

     L7Running both engines lets you discuss scheduler and cache trade-offs from experience, not just from reading.

     PD disaggregationInstallGitHub
   • 2025
     ★nano-vllm

     Xingkai Yu (GeeeekExplorer) · Jun 2025 · Practice · Core · ~6 h

     A roughly 1,200-line reimplementation of vLLM's core (scheduler, block manager with prefix caching, CUDA graphs, tensor parallelism) that you can read in an afternoon and modify.

     L7The best rehearsal for the Anthropic Fellows inference-engine OA and xAI 'implement batching' rounds: add a prefill/decode-aware scheduler or load balancer to it yourself.
   • 2026
     GuideLLM

     vLLM project (Red Hat / Neural Magic) · 2026 · Tool · Intro · ~2 h

     A load generator for OpenAI-compatible endpoints that sweeps request rates and reports TTFT, ITL and throughput against SLO targets. Use it to check whether your deployment meets its SLO.

     L7Baseline tool; the value is turning SLO talk into a measured sweep.

     NVIDIA AIPerf (successor to GenAI-Perf)
   • 2026
     vLLM Production Metrics

     vLLM project · 2026 · Reference · Core · ~1 h

     The Prometheus metrics a vLLM server exports: TTFT, inter-token latency, e2e and queue-time histograms, running and waiting requests, KV-cache usage, prefix-cache hit rate, and preemptions.

     L7Your debugging checklist for 'p99 doubled after a deploy': split p99 into queue, prefill and decode time, then check KV usage and preemptions before blaming the model.

     Metrics design docOptimization and tuning guide
4. 4

   Design the reported prompts

   Rehearse batch API, batched online API, router, Sora fair queueing, chat backend and design-doc critique with numbers · Week 4

   • 2026
     Batch processing (Message Batches API)

     Anthropic · 2026 · Reference · Core · ~0.5 h

     The product contract of a real batch inference API: async submission of up to 100k requests, results within 24 hours, 50% price discount, per-request results, expiry and cancellation semantics.

     L7Lets you start the 'batch inference API over GPU clusters' design from a real SLA and API shape, then explain how off-peak capacity and preemptible scheduling make the discount possible.

     OpenAI Batch API guide
   • 2026
     ★Interview notes: LLM Inference API with Batched GPU Serving

     zoezou2015 (community notes) · 2026 · Practice · Advanced · ~2 h

     A worked reference solution (about 6,500 words with Python estimates) for the Anthropic batching prompt: requirements, API, a dispatcher in front of GPU replicas, real-time and offline pools, KV bytes per token, prefix-cache savings, and a discrete-event simulation of a size-or-time batcher.

     L7Practise against it after writing your own answer. Check its numbers rather than memorising them, and be ready to change assumptions when pushed.
   • 2026
     Interview notes: LLM Inference Serving (consumer chat backend)

     Adam1679 (community notes) · 2026 · Practice · Advanced · ~1.5 h

     A worked consumer-chat serving design with traffic math (multi-turn prompt growth), KV bytes per token, decode step time inflated by shared prefill, and replica accounting.

     L7Rehearses the ChatGPT-scale prompt with numbers. Use it to check your own estimates.
   • 2026
     Interview notes: Video Generation Pipeline (Sora-style)

     Adam1679 (community notes) · 2026 · Practice · Advanced · ~1.5 h

     A worked Sora-style design: arrival rates, GPU-seconds per job by resolution, Little's law for pool size, capacity-dip scenarios with requeueing, plus queueing and cancellation.

     L7Pair it with the VTC fairness paper to answer the OpenAI one-GPU-per-task fair-queueing and cancellation question at depth.
   • 2024
     Fairness in Serving Large Language Models (Virtual Token Counter)

     Ying Sheng, Shiyi Cao, Dacheng Li et al. (Stanford / Berkeley), OSDI 2024 · Jan 2024 · Paper · Advanced · ~2 h

     Defines fairness for LLM serving in terms of weighted input and output tokens, not requests, and gives the Virtual Token Counter scheduler, a work-conserving fair-queueing algorithm with proven bounds. It is the standard reference for multi-tenant fair queueing.

     L7Lets you answer the OpenAI Sora fair-queueing question and the token-cost rate-limiter question with a cost function and a scheduler that has guarantees, instead of only proposing per-tenant queues.

     Related: learning-to-rank scheduling (approximate SJF)
   • 2025
     KV-Cache Wins You Can See: from prefix caching in vLLM to distributed scheduling with llm-d

     llm-d maintainers (IBM, Red Hat, Google) · Sep 2025 · Blog post · Advanced · ~1.5 h

     Moves from approximate (traffic-estimated) to precise (KV-event-driven) prefix-cache-aware routing across vLLM pods, with measured TTFT and throughput effects, and explains why round-robin destroys cache reuse.

     L7Directly prepares 'add a router between the API and many backends': scorers for cache affinity versus load, what the router must know, and the hot-spotting it causes.

     Intelligent inference scheduling with llm-dllm-d GitHub
   • 2025
     vLLM Router: A High-Performance and Prefill/Decode Aware Load Balancer for Large-scale Serving

     vLLM team · Dec 2025 · Blog post · Advanced · ~1 h

     The vLLM project's Rust router: consistent hashing for sticky, KV-cache-friendly routing, power-of-two-choices, round robin and random policies, P/D-aware routing (prefill worker, then decode worker), health checks, retries with backoff and jitter, and circuit breakers.

     L7A concrete design for the 'inference router' question (policies, state the router needs, failure handling) to compare against your own.

     Preble: prefix-aware distributed prompt schedulingGateway API Inference Extension (Kubernetes)vLLM production-stack
   • 2026
     Ray Data: Working with LLMs (batch inference)

     Anyscale / Ray project · 2026 · Tool · Core · ~2 h

     Offline batch inference with vLLM inside Ray Data pipelines: streaming execution across CPU preprocessing stages and GPU model stages, with separate autoscaling of each stage's actor pool.

     L7Directly maps to the Mistral PDF-to-Markdown and batch-API questions: a heterogeneous CPU, GPU and memory pipeline with backpressure.

     Ray Data paper: heterogeneous streaming batch executionRay Data overview
   • 2026
     Rate limits

     Anthropic · 2026 · Reference · Intro · ~0.5 h

     Token-aware rate limiting in production: requests per minute, input and output tokens per minute, token-bucket refill, tiers, and how cached tokens count.

     L7Baseline; gives a real reference for 'cost scales with tokens, not requests' (estimate output tokens at admission, then reconcile).

     OpenAI rate limits guideHello Interview: Design a Distributed Rate Limiter
   • 2026
     Streaming messages

     Anthropic · 2026 · Reference · Intro · ~0.5 h

     The SSE event protocol of a production LLM API: message_start, content_block_delta, ping, message_stop, and error events mid-stream.

     L7Baseline; enough to design the streaming layer and mid-stream failure semantics of a chat or token-streaming server.

     MDN: Using server-sent eventsTGI: streaming explained
   • 2026
     Interview notes: Design Doc Review: A Batched Inference Service

     zoezou2015 (community notes) · 2026 · Practice · Advanced · ~1.5 h

     A practice 'tear apart this design doc' exercise with a ranked review: recomputing the doc's own capacity numbers, the cache-offset-is-not-a-hit-rate error, missing sections, and splitting a shared 8-GPU pool by traffic class.

     L7Directly rehearses the critique format used in Anthropic EM and staff loops, where you are judged on ranking issues, not listing them.
   • 2026
     Design an LLM Inference API (techinterview.org) — use as a critique exercise

     techinterview.org · Apr 2026 · Practice · Intro · ~1 h

     A primer-level design write-up (requirements, KV cache, continuous batching, routing, prompt caching) with flaws: its 70B KV-cache example is off by about 2x and ignores GQA, and it has no failure handling.

     L7Use it as a 'critique this design' drill: find the arithmetic error, the missing SLO decomposition and the absent failure modes in 10 minutes.
5. 5

   Senior-staff depth

   Argue frontier-scale trade-offs (cost per token, KV storage, disaggregation, MoE, reliability, correctness) from production case studies and papers you can critique · Weeks 5–6

   • 2022
     Efficiently Scaling Transformer Inference

     Reiner Pope, Sholto Douglas, Aakanksha Chowdhery et al. (Google) · Nov 2022 · Paper · Advanced · ~4 h

     Google's PaLM inference paper. It gives an analytical model for choosing partitioning (2D weight-stationary, batch-sharded attention) to trade latency against cost per token at 500B scale, with multi-query attention as the KV-cache lever.

     L7The template for answering 'how would you shard a 500B model for low-latency versus high-throughput serving' with a cost model rather than by guessing.
   • 2024
     ★Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving

     Ruoyu Qin et al. (Moonshot AI / Tsinghua), FAST 2025 best paper · Jul 2024 · Paper · Advanced · ~3 h

     Kimi's production design: separate prefill and decode clusters around a distributed KV-cache pool in CPU DRAM/SSD over RDMA, a KV-centric global scheduler (Conductor), and early rejection under overload, all driven by real traces.

     L7A frontier-lab production case: lets you design KV storage, P/D disaggregation and overload shedding together, with real hit-rate and SLO numbers.

     vLLM × Mooncake: serving agentic workloads at scale (2026-05)
   • 2025
     ★DeepSeek-V3/R1 Inference System Overview

     DeepSeek-AI · Mar 2025 · Reference · Advanced · ~1 h

     DeepSeek's production serving: prefill/decode disaggregation with cross-node expert parallelism (EP32 prefill, EP144 decode), dual-batch overlap to hide communication, load balancers for prefill, decode and experts, and a day of real costs, revenue and a claimed 545% cost-profit margin.

     L7The best public frontier-scale cost-per-token case: lets you reason from H800 counts and tokens per second to dollars per million tokens.

     DeepSeek-V3 technical report
   • 2025
     Deploying DeepSeek with PD Disaggregation and Large-Scale Expert Parallelism on 96 H100 GPUs

     SGLang team (LMSYS) · May 2025 · Blog post · Advanced · ~1.5 h

     An open reproduction of DeepSeek-style serving in SGLang: PD disaggregation, DeepEP all-to-all, two-batch overlap and expert load balancing (EPLB), with throughput per node and the cost per million tokens compared to DeepSeek's API.

     L7Lets you size an MoE serving deployment and name its failure modes (expert hot-spotting, all-to-all latency).

     GB200 NVL72 Part I (2025-06)NVIDIA: GB200 NVL72 + Dynamo for MoE
   • 2025
     ★A postmortem of three recent issues

     Anthropic · Sep 2025 · Blog post · Advanced · ~1 h

     Anthropic's account of three overlapping Claude quality bugs: a context-window routing error that sent requests to the wrong server pool, TPU output corruption from a misconfiguration, and an approximate top-k XLA miscompile. It covers why they were hard to detect across hardware platforms and the eval and monitoring changes made afterwards.

     L7Frontier-lab production evidence: lets you discuss sticky routing, cross-hardware consistency and quality monitoring in a latency/quality debugging round.
   • 2024
     Optimizing AI Inference at Character.AI

     Character.AI · Jun 2024 · Blog post · Advanced · ~0.5 h

     Serving over 20,000 queries per second: MQA plus hybrid local/global attention plus cross-layer KV sharing cut the KV cache by more than 20x, with int8 training and serving and a 95% inter-turn prefix-cache hit rate from sticky routing and an LRU tree cache.

     L7A compact consumer-chat case study: lets you argue model-architecture changes as serving-cost levers, and justify sticky sessions with a hit-rate number.
   • 2025
     Defeating Nondeterminism in LLM Inference

     Horace He et al. (Thinking Machines Lab) · Sep 2025 · Blog post · Advanced · ~1.5 h

     Shows that run-to-run nondeterminism comes mainly from a lack of batch invariance (kernel reduction order changes with batch size), not from floating-point concurrency alone, and builds batch-invariant matmul, RMSNorm and attention kernels to fix it.

     L7A senior-staff point to raise unprompted: dynamic batching changes outputs. Lets you discuss the cost of determinism for evals, RL on-policy training and debugging.

     SGLang deterministic inference follow-upvLLM bitwise-consistent train/inference
   • 2026
     Taking vLLM Apart: A Practical Guide to Disaggregated Serving

     Martin Hickey (IBM Research), vLLM blog · Sep 2026 · Blog post · Advanced · ~1.5 h

     A 2026 practitioner guide to what disaggregation buys and costs in vLLM: prefill/decode split, multi-turn KV reuse, a GPU-less frontend and render tier, streaming, and who runs it in production.

     L7Current view on when not to disaggregate (small models, short prompts, poor interconnect), which is what an interviewer means by 'and its costs'.

     vLLM disaggregated prefill docsBi-directional KV transfer for multi-turn agentic TTFT (2026-10)
   • 2025
     AIBrix: Towards Scalable, Cost-Effective Large Language Model Inference Infrastructure

     ByteDance AIBrix team · Apr 2025 · Paper · Advanced · ~2 h

     A production control plane around vLLM: LLM-specific autoscaling, prefix-aware routing, a distributed KV cache, high-density LoRA management, heterogeneous-GPU optimisation and failure diagnostics.

     L7Shows what the control plane around the engine has to do. Use it to discuss autoscaling signals (KV usage, queue depth) instead of CPU utilisation.

     vLLM blog: AIBrix release
   • 2024
     "Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantization

     Eldar Kurtic et al. (Neural Magic / Red Hat, IST Austria) · Nov 2024 · Paper · Advanced · ~2 h

     Over 500k evaluations of FP8, INT8 and INT4 schemes on Llama-3.1 across academic and real-world benchmarks, with deployment guidance by hardware and workload (latency- versus throughput-bound).

     L7Gives you the method for 'how do you prove quality hasn't regressed': which evals, which metrics, and what recovery to expect per scheme.

     Survey: low-bit LLMs, basics, systems and algorithms
   • 2026
     From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving

     Jie Li, Tongyang Wang, Yong Chen · Jun 2026 · Survey paper · Advanced · ~4 h

     The newest KV-cache systems survey. It classifies more than 30 systems by locality, lifetime, ownership and substrate into five archetypes (local-paged, disaggregated-pipeline, shared-store, memory-pool, hybrid-tier) and lists the KV-specific measurements that current evaluations leave out.

     L7Gives you a framework for 'the KV cache as a storage system', so you can design cross-worker prefix reuse and argue who owns the cache and when it gets evicted.

     Companion survey: system-aware KV cache optimization (2026-07)Older survey: KV-cache consumption review (2024)
   • 2026
     Elastic EP in SGLang: Achieving Partial Failure Tolerance for DeepSeek MoE Deployments

     SGLang team (LMSYS) · Mar 2026 · Blog post · Advanced · ~1 h

     How wide-EP deployments survive the loss of a GPU or node without a full restart: redistributing experts and keeping serving with degraded capacity.

     L7Failure handling at scale: a 96-GPU EP group is one failure domain unless you design it otherwise, which is the reliability point senior-staff answers should make.

     SGLang fast engine recovery (2026-08)vLLM elastic EP: near-zero-downtime scaling (2026-10)
   • 2025
     InferenceMAX: Open Source Inference Benchmarking

     SemiAnalysis · Oct 2025 · Blog post · Advanced · ~1 h

     Nightly open benchmarks of vLLM, SGLang and TensorRT-LLM across H100, H200, B200, GB200 and MI300X/MI355X, plotted as throughput per GPU against interactivity, and converted to cost per million tokens.

     L7Lets you quote current cost-per-token versus tokens-per-second-per-user curves by hardware, which helps when an interviewer asks 'where does the money go'.

     Live dashboardvLLM on Blackwell in InferenceMAX

What the interviews ask

Problems that recur in this track across 53 reports (17 firsthand), with the resources that prepare you for each.

Question coverage

Every question group collected for this topic from candidate reports, mapped to the resources that prepare you for it. 65 of 65 groups are covered, which is 100% of all report mentions. Hover an example to see the full question.

All resources

Everything collected for this topic, newest first. 2026 items are marked green. ★ marks a top pick.