🚀 vLLM 0.28.0 arrives with 584 commits and improvements focused on running language models faster and more efficiently.
⚡ The release adds major optimizations for Kimi-K3 and DeepSeek V4, along with advances in speculative decoding. This technique lets an assistant model propose tokens while the main model verifies them, reducing response time.
🧠 Model Runner V2 also matures with weight offloading, tiered KV caches, CUDA graphs for encoders, and support for attention-free models. KV storage can offload data to disk or CPU, helping serve large models when GPU memory is limited.
🌐 The Rust frontend adds a standalone renderer, multimodal inference over gRPC, and better integration with reinforcement-learning workflows. The release expands support for NVIDIA, AMD ROCm, Intel XPU, and CPU, together with quantization paths such as NVFP4 and MXFP4.
The project team also raised some defaults, including batched tokens, and enabled prefix caching for Mamba models.
💡 Explanation in a nutshell#
vLLM is an engine that helps deliver AI models to many people at once. This update optimizes memory, hardware, and text generation so systems can respond faster and handle larger workloads.
More information at the link 👇
