TL;DR: Transformers vLLM backends are now as fast (or faster) than custom vLLM implementations in many LLM architectures. Modelers can automatically leverage Transformer implementations to achieve ultra-fast vLLM inference for free. uv pip install –upgrade vllm –torch-backend auto
The Transformers library is now a reference modeling library for machine learning. It supports over 450 architectures through a consistent API and is designed with the primary goal of making model implementations self-contained and easy to understand. By reviewing the transformer code, contributors can easily learn how the architecture works and port it to other frameworks such as vLLM, SGLang, MLX, and llama.cpp.
We fully embrace this role in the ecosystem and go to great lengths to facilitate it. A big step in this direction was last year’s integration of Transformers as a modeling backend for vLLM. This allows modelers to run Transformer models (similar to LLM and VLM) within vLLM without porting anything. Transformers provides the modeling code, and vLLM provides highly optimized inference techniques such as continuous batch processing and custom attention kernels.
This integration is even better 🚀!
showcase
We connect vLLM’s Transformer modeling backend directly with vLLM’s hand-written native implementation across three very different Qwen3 models.
4B dense model on a single GPU 32B dense model on tensor parallelism Mix of experts on FP8 data with 235B parameters + expert parallelism on the same 8×H100 nodes
As a result, the Transformer modeling backend now meets or exceeds native throughput for all Transformers.
A single flag –model-impl transformer runs the * hug face model through the transformer modeling backend. Nothing changes about the service setup as it is configured with the usual parallelism options.
vllmserve Qwen/Qwen3-4B –model-impl transformer vllmserve Qwen/Qwen3-32B –model-impl transformer –tensor-Parallel-size 2 vllmserve Qwen/Qwen3-235B-A22B-FP8 –model-impl transformer –data-Parallel-size 8 –enable-expert-Parallel
*Models using linear attention are currently not supported, but will be supported soon. Custom models whose code resides in the Hub repository are not written compliantly and are unlikely to work.
Measurement method
Each model is compared under three conditions that are identical in all respects except the code path.
Native — –model-impl vllm, vLLM hand-drawn model (matching bars) After — –model-impl transformer (with PR) Before — –model-impl transformer (without PR)
A complete reproducible runner is available at Gist: benchmark.sh
So what’s new?
vLLM’s transformer modeling backend was noted as an inference bottleneck. By plugging in vLLM’s attention implementation at runtime, transformer models can be efficiently executed within the vLLM engine. However, there are many aspects to deployment, and only custom ports are relevant for maximum inference performance. Cross-GPU parallelization, compilation, fused kernels, etc. all help leverage the hardware to achieve ultra-fast inference.
The new model was integrated once for transformers and once for vLLM using custom optimizations.
When modelers wanted the absolute best performance, they were still creating custom vLLM implementations.
Once the new model is integrated into the transformer, it can be immediately used in vLLM at native vLLM implementation speeds.
The latest iteration of vLLM’s transformer modeling backend dynamically applies a fusion of inference-specific layers at runtime to match the speed of custom code implementation for compatible architectures.
How does it work?
vLLM’s Transformer modeling backend now uses torch.fx to perform static analysis on the model’s graph. This process searches for known patterns that can be optimized. Once a pattern is identified, we use an AST (abstract syntax tree) to manipulate the source code and rewrite some operations appropriately.
What can this accomplish?
Fusion operations that map many-to-one to (hyper)optimized vLLM kernels, such as those used for Expert Parallelization (EP) in Mixture-of-Experts (MoE) models. The other main fusion operations are vLLM’s MergedColumnParallelLinear and QKVParallelLinear. These blocks can be used to infer parallel plans for TP (tensor parallelism). If the decoder block list is easily identified, a PP (Pipeline Parallel) plan can also be inferred. The manipulated model is still fully (torch) compilable and passed through torch.compile and the CUDA graph, just like a dedicated vLLM model implementation. Unlike vLLM model implementations, Transformers model implementations can be used for training. Therefore, you can use the same model code for training/evaluation/RL rollout.
As shown above, this enables native vLLM inference speeds for compatible models without having to write a single line of code to optimize the model for inference.
We are in the process of writing a detailed blog post that dives deeper into these optimized inference techniques and details how to manipulate models to adapt to them.
resource
