MMOOZZGG.com

Glossary

What is multimodal AI?

In short

Multimodality is the ability of an AI model to work with several types of data at once: text, images, sound and video. Multimodal AI is a model with this ability.

AI basics · multimodality

MMOOZZGG.com editorsChecked: 1 min read

For a child

To choose a watermelon, you look at it, knock on the rind and listen to the sound, and at home you taste a piece. You put together the hints from your eyes, ears and tongue. Computers used to do only one thing: either read or look at pictures. Now there are programs that look at a drawing and right away tell you about it in words, or answer a question about a photo.

In plain words

A modality is a type of data: text, a picture, sound, video. An ordinary language model understands only text. A multimodal one takes several types at once: you show a photo of a meter and ask what numbers it shows, or upload a meeting recording and ask for a list of decisions. Inside, the different data are turned into a shared set of numbers, so the model links the word “cat” with a picture of a cat. Some models only read pictures, and others draw, speak and make video themselves.

For business

For a business, this means one AI assistant reads document scans, site photos, screenshots of errors and voice messages, not only text. A courier sends a photo of a damaged box, and the system files the complaint by itself. An engineer takes a photo of the plate on a machine and gets the right manual. Pictures and audio cost more than text: one image takes many tokens in a model, and video takes even more. A model can be confidently wrong about small numbers in a photo, so key fields are checked against the accounting system. Which tasks pay back these features is shown with examples of generative AI at work.

For a specialist

Multimodal models process and match several modalities. The key technique is a joint embedding space. CLIP (Radford et al., 2021) was trained on 400 million image and text pairs from the internet to guess which caption belongs to which picture. It transfers to new tasks without fine-tuning (zero-shot), and the authors tested it on more than 30 datasets. Flamingo (Alayrac et al., 2022) joined pretrained vision and language models and solves new tasks from a few examples on interleaved sequences of pictures and text. According to the OpenAI technical report of 2023, GPT-4 takes images and text as input and returns text. Architectures differ: separate encoders projected into a language model (vision-language models), early fusion of tokens from different modalities, and models that also generate output of any type (any-to-any). They are evaluated on tasks such as visual question answering.

Example

An insurance company receives a photo of car damage and a short voice message from a client. The model describes the damage, checks it against the policy and prepares a draft claim for an employee.

Related terms

Questions and answers

What is a multimodal model? It is a model that takes in or produces several types of data: text, images, sound, video. For example, it answers a question about a photo or writes minutes from an audio recording.
How is multimodal AI useful for a company? Staff don’t have to turn everything into text: they can send a photo, a scan or a voice message. This speeds up document intake, complaint handling and the work of field staff.

Sources

Not sure whether your company needs this?

AI solutions map
Contact us