Multi-LoRA Serving: Sharing One GPU Across Adapters

Multi-LoRA serving answers a capacity question that hits every team running fine-tuned variants: if each customer needs its own adapted model, do you deploy a full model copy per customer? With adapter-based serving you deploy one. The base model stays resident in GPU memory, and each request names a small LoRA adapter that is applied for that call. The vLLM and AWS engineers behind the current open-source implementation describe the consolidation case in direct terms: five customers that each use roughly 10% of a dedicated GPU can be consolidated onto a single GPU, with adapters swapped in and out per request instead of five mostly idle accelerators.

The economics matter most when fine-tuned variants multiply faster than traffic grows. A dedicated endpoint per variant leaves most of its GPU idle most of the time, and serverless GPU platforms bill that idle time through cold starts and minimum billing windows. Adapter sharing attacks the waste at its source: the expensive weights are loaded exactly once.

Why Adapters Change GPU Economics

Low-Rank Adaptation leaves the base weights untouched during fine-tuning and trains a pair of small matrices per adapted projection. At inference the adapter contributes a low-rank correction alongside the frozen base layer, so a fine-tuned variant costs megabytes where a full model copy costs gigabytes. Serving many customers from one process stops being a memory fantasy and becomes a budgeting exercise.

The budget still binds, though. Adapter weights, the KV cache and activation memory all draw from the same device capacity, and a pool of resident adapters shrinks what remains for sequences in flight. The same sizing discipline that governs KV cache memory planning for long contexts governs the adapter pool: measure what each rank class costs, cap what can be resident, and spill the rest to host memory.

The reward for that discipline is packing density. Idle GPU cycles from quiet tenants are absorbed by busy ones, so utilization rises without adding hardware or splitting traffic across underused replicas.

How S-LoRA Proved the Model

The architecture that made dense multi-adapter serving credible comes from the S-LoRA research system. It keeps every adapter in main memory and fetches only the ones needed by running queries into GPU memory, managed by a Unified Paging scheme: a single pool that holds adapter weights of different ranks together with KV cache tensors of varying sequence lengths, which keeps fragmentation from eating the budget. Heterogeneous batches, mixing requests for different adapters in one step, run on custom CUDA kernels with a tensor-parallel strategy designed for that mix.

The measured result set the reference point for the field. In the paper’s evaluation, S-LoRA served thousands of LoRA adapters on a single GPU and improved throughput by up to 4 times over HuggingFace PEFT and naive vLLM LoRA serving, with the number of served adapters rising by several orders of magnitude. Later production engines, including vLLM’s own implementation, descend from this design.

Configuring vLLM for Multi-LoRA

vLLM exposes the pattern through a small set of startup flags. You enable adapter serving, register the adapter catalogue, and size the pool; each request then picks its adapter through the ordinary model parameter of the OpenAI-compatible endpoint.

Hot-loading adapters without a restart is supported through dedicated load and unload endpoints, gated behind an environment variable, and remote resolver plugins can pull adapters from object storage or model hubs. Treat both as trusted-infrastructure features: vLLM itself warns that dynamic runtime LoRA loading should not be used in production unless the environment is isolated and fully trusted. For most teams the safer pattern is registering the adapter set at startup and rolling the deployment when the catalogue changes.

If your stack is Hugging Face TGI instead, the equivalent capability is native there too: TGI loads multiple LoRA adapters at startup through its LORA_ADAPTERS setting, a capability available since version 2.0.6, and routes each request with an adapter_id parameter.

1. Confirm every adapter targets the same base model and architecture; mixed base models cannot share a process.
2. Inventory adapter ranks and set the limit to the largest rank actually present; an oversized value wastes memory and can degrade performance.
3. Size the resident and host-cached adapter limits against your concurrency mix, keeping the resident set within the memory left after KV cache provisioning.
4. Register adapters at startup and verify they appear in the models listing before shifting traffic.
5. Promote requests gradually, watching time to first token and output throughput for regression against the single-model baseline.

MoE Models Raise the Bar

Mixture-of-experts architectures complicated the picture. A routed expert layer already batches tokens sparsely across experts, and adding per-request adapters stacks a second source of sparsity on top: the LoRA rank dimension is far smaller than the hidden dimensions it multiplies, so standard matrix kernels, tuned for roughly square shapes, perform badly on the skinny adapter operations at every adapted projection of every expert. vLLM answered with a fused MoE-LoRA kernel that folds the adapter shrink and expand steps into the existing expert routing kernel.

Getting there required profiling work the original implementation lacked: an early version suffered severe time-to-first-token regressions from kernel recompilation on new context lengths, which the team eliminated with a compiler hint, plus split-K decomposition and cache-aware scheduling for the skinny multiplies. The result shipped for the current MoE families: multi-LoRA serving works out of the box from vLLM 0.15.0 for GPT-OSS, Qwen3-MoE, DeepSeek and Llama MoE, and the vLLM and AWS team report 454% higher output tokens per second and 87% lower time to first token for GPT-OSS 20B in vLLM 0.15.0 compared with 0.11.1rc3, with the tuned configurations available on SageMaker AI and Bedrock.

Decision Checklist Before Consolidating

Multi-LoRA is the right default only under specific conditions. Run this checklist before committing:

• Every variant shares one frozen base model; divergent bases need separate deployments.
• Adapter ranks are similar, so a single rank limit does not waste memory for most tenants.
• Per-tenant traffic is spiky and low; a tenant that saturates a GPU alone may deserve isolation.
• Latency budgets tolerate the added adapter overhead, verified against your own baseline.
• Hot-loading, if used at all, runs only in an isolated, trusted environment with controlled adapter sources.
• Adapter versions are pinned and auditable, since a shared process couples every tenant to one rollout.

Sources

• vLLM blog: Efficiently serve dozens of fine-tuned models with vLLM on Amazon SageMaker AI and Amazon Bedrock
• vLLM documentation: LoRA Adapters
• S-LoRA: Serving Thousands of Concurrent LoRA Adapters (arXiv)
• Hugging Face TGI documentation: LoRA (Low-Rank Adaptation)