v0.29.0
Model Runner V2 (MRV2) is now the default execution engine for all models in vLLM. This release introduces CUDA graph memory profiling for KV cache auto-sizing and batch-sharded sampling to reduce per-step logits memory usage.
Verified State Diff
Impact & Verification Analysis
Developers and infrastructure engineers deploying LLMs using vLLM, particularly those utilizing Tensor Parallelism or speculative decoding.
This update optimizes memory overhead and improves inference throughput by standardizing the execution engine and enabling more granular control over KV cache and logits memory allocation.
Full Fact Overview
The transition to Model Runner V2 as the default architecture represents a significant shift in vLLM's execution pipeline, moving away from legacy runner implementations. Key technical additions include batch-sharded sampling, which scales memory efficiency by a factor of 1/TP (Tensor Parallelism), and the integration of CUDA graph memory profiling for dynamic KV cache sizing. Additional capabilities include support for prompt embeddings, hidden state extraction for speculative decoding, and padded full CUDA graph dispatch for uniform decoding scenarios.