Advanced LLM inference & serving — interview-grade map

Senior/staff inference-optimization interview topics — KV cache & paging, batching, speculative decoding, quantization, attention kernels/variants, long context, serving architecture — mapped to the best external source and to where this repo touches it.

Advanced LLM inference & serving — interview-grade map

Inference is the half a speech/latency background is strongest in. Format: what’s probed, the single best source, in the repo where real.

Fundamentals

1. Roofline: memory-bound decode vs compute-bound prefill. Decode reuses huge weight matrices once per token → bandwidth-bound; prefill is parallel → compute-bound. Derive arithmetic intensity. “Roofline for prefill vs decode on an H100 — where does each bottleneck?” Learn: LLM Inference Unveiled (roofline) · senior In repo: docs/research/mac_decode_baseline_m5pro.md — native models hit 293–767 tok/s; today’s bf16 4B via the HF path managed 7 tok/s, a textbook bandwidth-bound-plus-unoptimized-kernel case.

2. KV-cache memory math. Size it by hand: 2·n_layers·n_kv_heads·head_dim·seq·batch·bytes; why KV (not weights) caps batch/context. “KV for Llama-3-70B @ 8k, batch 32 — what limits concurrency?” Learn: PagedAttention/vLLM · senior In repo: docs/kv_cache_optimization.md; Sample.swift exposes --kv-quantize / --kv-preallocate.

3. TTFT vs ITL vs goodput. TTFT (prefill-bound), ITL/TPOT (decode-bound), goodput under SLOs; batch size trades TTFT against ITL. “p95 TTFT<300ms, ITL<40ms — tune the scheduler.” Learn: Inference Trilemma · mid In repo: Pace’s TTFW hunt (330→119ms) is a TTFT story; speech-and-systems-topics.md §1.

Memory & scheduling

4. PagedAttention. Naive contiguous KV wastes 60–80% (fragmentation); OS-style paging + block tables fix it and enable copy-on-write prefix sharing. Learn: PagedAttention · senior

5. Prefix / prompt caching (RadixAttention). Reuse shared system prompts / few-shot / chat history via a radix tree of KV blocks. “2k-token shared system prompt — avoid recomputing it per request?” Learn: SGLang/RadixAttention · senior In repo: Pace sends cache_prompt: true on every request — this exact win.

6. Continuous / in-flight batching. Iteration-level scheduling injects / evicts requests every decode step so the GPU never idles on stragglers. “Why does static batching tank throughput with heterogeneous lengths?” Learn: Orca (OSDI’22) · senior In repo: posttrainllm serve is single-stream — know this as the throughput lever you’d add for multi-tenant serving.

Decoding & quantization

7. Speculative decoding. Draft proposes k tokens, target verifies in one parallel pass; rejection sampling keeps the distribution exact. “Why doesn’t it change outputs, and when does it lose?” Learn: Speculative Decoding · senior In repo: SpeculativeDecode.swift (B14, T=0 byte-equality gate).

8. Self-speculation (Medusa, EAGLE). Bolt-on heads / feature-level autoregression beat a separate draft model on acceptance and avoid serving two models. Learn: EAGLE · staff

9. Weight quantization (GPTQ vs AWQ). GPTQ’s Hessian/second-order error compensation vs AWQ’s activation-aware salient-channel scaling; weight-only int4 helps memory-bound decode. “Pick a scheme for a latency-sensitive 70B.” Learn: AWQ · senior In repo: posttrainllm gptq / hqq (GPTQ.swift, HQQ.swift); quantized-HF checkpoint loading (commit ccf8937) is what let today’s A1 load a 4-bit base.

10. fp8 / activation quant / formats. fp8 on Hopper/Blackwell, int8 SmoothQuant for activation outliers, GGUF for CPU/edge. “weight-only int4 vs fp8 weight+act — which for throughput, which for accuracy?” Learn: TensorRT-LLM quantization · senior In repo: posttrainllm gguf-load / to-coreml (the edge/ANE path).

11. KV-cache quantization. The lever for long context + large batch; harder than weight quant (outlier keys, accuracy cliffs). Learn: roofline survey · staff In repo: Sample.swift --kv-quantize.

Kernels, attention & long context

12. Attention variants MHA/MQA/GQA/MLA. KV-head sharing shrinks the cache and raises arithmetic intensity; GQA is the dense standard, MLA (DeepSeek) compresses KV to a low-rank latent (~90%+ reduction). “Why MHA→GQA, and what does MLA add?” Learn: GQA · MLA/DeepSeek-V2 · senior In repo: Qwen3-4B (the A1 base) uses GQA — that’s why its KV cache is small.

13. FlashAttention v2/v3. IO-aware tiling avoids the N×N matrix; v3 adds Hopper async (warp-specialization, TMA, fp8); recompute-in-backward. “Why faster despite recomputing softmax stats?” Learn: FlashAttention-3 · staff In repo: posttrainllm rides MLX’s fused attention — the kernel you don’t hand-write.

14. Long context: RoPE scaling + sparse attention. Position interpolation vs NTK-aware vs YaRN; sliding-window (Mistral), ring/blockwise for sequence parallelism. “Extend 4k→128k — what changes and why does naive interpolation degrade?” Learn: Extending RoPE / YaRN · staff (RoPE itself: advanced-ml-systems-eval.md §2.)

Serving architecture

15. Disaggregated prefill/decode & chunked prefill. Split phases onto separate GPU pools (needs fast RDMA KV transfer) vs co-locate + interleave; goodput tradeoffs. “Prefill spikes blow your decode ITL SLO — disaggregate or chunk?” Learn: DistServe · staff

16. Inference parallelism: TP / EP / multi-LoRA. TP (split matmuls, all-reduce/layer, NVLink-bound); expert parallel for MoE (all-to-all, load imbalance); multi-LoRA serving (many adapters on one base, S-LoRA). “Serve a 671B MoE on 8 GPUs — TP vs EP vs hybrid?” Learn: Inference Handbook: parallelism · staff In repo: serve --lora is single-base adapter stacking — the seed of multi-LoRA serving.

Curated deep-reading path

Use this sequence after the numbered topic map. It moves from the universal bottleneck model to one-GPU kernels, fused attention, production serving, quantization quality, and finally multi-accelerator scaling. Papers and books are mixed deliberately with implementation worklogs: learn the governing idea, then see it survive contact with code and measurements.

1. Build the bottleneck model. Making Deep Learning Go Brrrr From First Principles separates compute, memory-bandwidth, and framework/launch-overhead regimes. Use that diagnosis before reaching for an optimization.
2. Optimize one foundational kernel. How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance progresses through coalescing, shared-memory tiling, block/warp tiling, vectorized access, occupancy, and autotuning.
3. Learn the attention algorithm. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness derives why reducing HBM traffic changes the algorithm, not just its implementation.
4. Study the advanced fused kernel. A Case Study in CUDA Kernel Fusion: FlashAttention-2 on Hopper applies CUTLASS layouts, TMA, WGMMA, asynchronous pipelines, and tile-size tradeoffs. Read this after steps 2–3.
5. Follow kernels through the compiler. Triton Kernel Compilation Stages follows a kernel from Python AST through Triton IR, GPU IR, LLVM IR, and device code.
6. Move from a model to a serving scheduler. Continuous Batching From First Principles derives ragged batching and dynamic request scheduling from attention and KV-cache behavior.
7. Quantify inference economics. All About Transformer Inference connects TTFT, per-token latency, batching, KV memory, arithmetic intensity, and model parallelism.
8. Read a real engine end to end. SGLang Deep Dive tours scheduling, RadixAttention, paged KV, CUDA graphs, speculative decoding, kernels, distributed execution, and observability.
9. Measure quantization as distribution drift. How Fireworks Evaluates Quantization explains prefill/generation KL divergence and token rejection rate alongside noisy task metrics and use-case gates.
10. Derive the single-chip-to-cluster boundary. How to Scale Your Model develops rooflines, Transformer FLOPs, communication collectives, accelerator topology, training parallelism, and inference scaling from first principles.
11. Apply the distributed-training toolbox. The Ultra-Scale Playbook combines educational implementations, production Nanotron anchors, and measured scaling experiments across data, tensor, pipeline, and context parallelism plus ZeRO and communication overlap. It is also indexed in advanced-llm-training.md.

Worked performance-engineering case studies

Ali Taha’s writing is a useful bridge from the concepts above to measured kernel and model work. Read the long, code-backed pieces as the durable path; use the shorter X threads as supplementary field notes.

Blackwell matmul, from naive kernel to cuBLAS-class performance. This four-part series builds the optimization stack incrementally: tiling and the memory hierarchy, Tensor Cores and TMA, 2-SM MMA and pipelining, then a persistent CLC scheduler. Keep the result’s scope in view: performance claims are tied to particular Blackwell shapes and configurations.

1. Introduction
2. Using the hardware
3. The optimizations behind 85% of SOTA
4. Breaking SOTA

Quantized inference, end to end. Four Bits: 4-bit quantization for FLUX.2 connects profiling, NVFP4 blockwise scaling, kernel overhead, calibration, output quality, and end-to-end latency rather than stopping at a microbenchmark.

Supplementary field notes. These are useful snapshots of current model and kernel work, but their short-form format makes them starting points for follow-up rather than standalone references:

• 22580: From GPT-2 to Kimi K3, explained
• Notes on writing the fastest video kernel in the world
• TurboQuant
• Quantization-aware distillation for Qwen 2512
• Optimizing FLUX.2 on B200

Suggested order

For targeted interview review, use topics 1–3 first, then 4–6 and 12–13; read 7 and 9 against the repo implementations. For durable systems learning, follow the curated deep-reading path in order, then read the Ali Taha case studies. That makes the Blackwell and FLUX results applications of an existing mental model rather than isolated performance tricks.