MMOOZZGG.com

Glossary

What is vLLM?

In short

vLLM is an open-source library for running language models on a server; it serves many requests at the same time and uses video memory economically.

Models and products · vLLM

MMOOZZGG.com editorsChecked: 1 min read

For a child

Imagine a school canteen where one cook makes lunch for the whole class. If the cook makes one plate at a time, the line will last until evening. A smart cook puts the food into small boxes and cooks many portions at once without throwing anything away. There is a program that hands out answers to many people just as fast when they all ask a smart assistant at once.

In plain words

It is a program for servers that connects a language model to many users at once. A normal model launch quickly hits the limit of the graphics card memory: each conversation keeps its own intermediate calculations. The library splits them into small pages, the way an operating system splits RAM, so more conversations fit on the same graphics card. For example, a company puts the model on its own server, and the whole sales team uses the assistant at the same time without long waits.

For business

The code is open under Apache 2.0, there are no license fees, and requests and answers stay on your servers. The main cost here is graphics cards: the more fully the program uses them, the fewer servers you need for the same number of employees. An OpenAI-compatible interface lets you move existing applications from a cloud API to your own model with almost no changes. You will need engineers to set up the server, monitoring, and updates, or the savings on hardware will be lost to downtime. Before you buy hardware, measure throughput on your own requests: it decides how many graphics cards you need and what a million tokens will cost.

For a specialist

vLLM started at the Sky Computing Lab of the University of California, Berkeley. Its core is described in the paper by Kwon et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention” (SOSP 2023): the key-value cache (KV cache) is split into blocks, like paged virtual memory, so memory lost to fragmentation and duplication drops to almost zero. According to the authors, throughput grew 2–4 times compared with FasterTransformer and Orca at the same latency. Today the engine adds continuous batching, chunked prefill, prefix caching, FP8, INT4, GPTQ, AWQ, and GGUF quantization, and tensor, pipeline, and expert parallelism. The server is compatible with the OpenAI API and Anthropic Messages, and it runs on NVIDIA, AMD, and Intel GPUs and on CPUs.

Example

An insurance company runs an open model through vLLM on its own servers and connects a chat for its operators to it. When the load grows in the busy season, engineers watch the request queue and add graphics cards without changing the application.

Related terms

Questions and answers

What is vLLM? It is an open-source library that runs a language model on a server and serves many users at the same time. It uses video memory carefully, so one graphics card handles more requests.
What license does vLLM have? The project is distributed under the Apache 2.0 license, so you can use it for free in commercial systems. The costs are the hardware and the engineers’ work.

Sources

Not sure whether your company needs this?

AI solutions map
Contact us