MMOOZZGG.com

Glossary

What is a vision language model?

In short

A vision language model (VLM) is a neural network that takes an image and text and answers in words: it describes a photo, reads a document, or answers questions about a picture.

Language models · vision-language model

MMOOZZGG.com editorsChecked: 1 min read

For a child

Imagine a friend who can both look and talk. You show them a drawing and ask how many kittens are in it, and they count and answer. An ordinary talking program sees only letters. This one looks at the picture, understands what is drawn there, and tells you about it in words.

In plain words

An ordinary language model understands only text. This model got vision: a separate part of the network turns the image into a set of numbers that look like words and passes them to the language part. After that, everything works as in a chat, only you send a photo, a screenshot, or a scan with your question. The model can read a receipt, describe a product in a photo, find an error in a table, or explain a chart. It writes the answer as text, and such models usually cannot draw new pictures.

For business

Such models take on tasks that used to need a person with eyes: reading scans of delivery notes and contracts, checking photos from sites, matching price tags on a shelf, and answering customers who send a photo of a breakdown. The price depends on the number and size of images: a picture takes many tokens, and you see it in the bill. The risks: the model confuses small digits on poor photos, it can confidently describe something that is not in the image, and scans with personal data must not go to an outside service without a legal basis. Start with a hundred real documents and count the share of errors; it is easiest to do this on one process, such as request handling.

For a specialist

A vision language model (VLM) combines a vision encoder and a language model. The typical design: the image is cut into patches, a transformer-based encoder turns them into vectors, and a projection layer maps them into the embedding space of the language model, where they go along with the text tokens. Many encoders are based on CLIP (Radford et al., 2021): contrastive pretraining on 400 million image-caption pairs gave transfer to new tasks without fine-tuning (zero-shot). LLaVA (Liu et al., 2023) joined such an encoder with a language model and trained the pair on instructions generated by text-only GPT-4. Other designs use cross-attention to visual features or early fusion, where the network learns from a mix of tokens of different modalities from the start. Quality is tested on visual question answering (VQA) and on reading documents and charts; typical failures involve counting objects, small fonts, and object hallucinations.

Example

A service company gets photos of broken equipment from its customers. The model identifies the type of device and the visible damage, fills in the request, and asks a clarifying question, so the technician arrives with the right part.

Related terms

Questions and answers

How is a vision language model different from a multimodal one? Multimodality is broader: it covers sound, video, and other data types, and the model may also create them. A vision language model works with images and text and usually answers in text.

It is a special case of a multimodal model.

Can a VLM read documents and scans? Yes, reading documents, tables, and charts is one of its typical tasks. Accuracy depends on the scan quality and the font size, so important fields, such as amounts and company details, are checked by rules or by a person.

Sources

Where companies use it

Not sure whether your company needs this?

AI solutions map
Contact us