Glossary
What is vLLM?
In short
vLLM is an open-source library for running language models on a server; it serves many requests at the same time and uses video memory economically.
Models and products · vLLM
MMOOZZGG.com editorsChecked: 1 min read
For a child
Imagine a school canteen where one cook makes lunch for the whole class. If the cook makes one plate at a time, the line will last until evening. A smart cook puts the food into small boxes and cooks many portions at once without throwing anything away. There is a program that hands out answers to many people just as fast when they all ask a smart assistant at once.
In plain words
It is a program for servers that connects a language model to many users at once. A normal model launch quickly hits the limit of the graphics card memory: each conversation keeps its own intermediate calculations. The library splits them into small pages, the way an operating system splits RAM, so more conversations fit on the same graphics card. For example, a company puts the model on its own server, and the whole sales team uses the assistant at the same time without long waits.
For business
The code is open under Apache 2.0, there are no license fees, and requests and answers stay on your servers. The main cost here is graphics cards: the more fully the program uses them, the fewer servers you need for the same number of employees. An OpenAI-compatible interface lets you move existing applications from a cloud API to your own model with almost no changes. You will need engineers to set up the server, monitoring, and updates, or the savings on hardware will be lost to downtime. Before you buy hardware, measure throughput on your own requests: it decides how many graphics cards you need and what a million tokens will cost.
For a specialist
vLLM started at the Sky Computing Lab of the University of California, Berkeley. Its core is described in the paper by Kwon et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention” (SOSP 2023): the key-value cache (KV cache) is split into blocks, like paged virtual memory, so memory lost to fragmentation and duplication drops to almost zero. According to the authors, throughput grew 2–4 times compared with FasterTransformer and Orca at the same latency. Today the engine adds continuous batching, chunked prefill, prefix caching, FP8, INT4, GPTQ, AWQ, and GGUF quantization, and tensor, pipeline, and expert parallelism. The server is compatible with the OpenAI API and Anthropic Messages, and it runs on NVIDIA, AMD, and Intel GPUs and on CPUs.
Example
An insurance company runs an open model through vLLM on its own servers and connects a chat for its operators to it. When the load grows in the busy season, engineers watch the request queue and add graphics cards without changing the application.
Related terms
Questions and answers
Sources
Not sure whether your company needs this?
AI solutions map