Glossary
What is modality in AI?
In short
Modality in AI is the type of data that a model works with: text, image, sound, video, a table or a sensor signal.
AI basics · modality
MMOOZZGG.com editorsChecked: 1 min read
For a child
You have eyes, ears, a nose and hands, and each of these helpers learns something different about the world. Eyes see color, ears hear music, hands feel whether something is hot or cold. A computer also gets information in different ways: it reads letters, looks at pictures, listens to sounds. Each of these ways is called a modality.
In plain words
Modality answers the question “in what form did the data arrive?” A client’s email is text, a photo of an invoice is an image, a support call is sound, and a camera recording is video. Each type needs its own way to turn data into numbers: text is cut into tokens, a picture into small squares, sound into short pieces. In the past, each type had its own program. Now models can take several types at once and turn one into another: describe a photo in words or turn text into speech.
For business
Modality decides which AI fits a task and what it will cost. Reading text emails is the cheapest. Document scans and photos need text recognition or a vision model and cost more per request. Calls are first turned into text, and the quality of this step strongly affects the result: accents, noise and a bad line cause errors in the transcript. Before a project, make a list: which data types enter the process, in what volume, where they are stored and who has access. Call recordings and photos of faces often contain personal data, and that narrows the choice of cloud services. How staff should give tasks to such models is shown with examples of generative AI.
For a specialist
In “Multimodal Machine Learning: A Survey and Taxonomy” (2017), T. Baltrusaitis, C. Ahuja and L.-P. Morency define a modality as the way something happens or is experienced. They call a task multimodal when it involves several such ways. They name five challenges: representation, translation between modalities, alignment of elements from different sources, fusion and co-learning. In practice, each data type has its own encoder and tokenization: subword tokens for text, image patches in a Vision Transformer, spectrogram frames or discrete audio tokens for sound. The representations are then projected into a shared space. Early fusion sends tokens of different types into one model; late fusion combines the outputs of separate models. Input and output can be of different types: text to image, speech to text, image to caption.
Do not confuse. Do not confuse it with modality in linguistics and logic, where the word means how a statement relates to reality: possibility, necessity, desirability.
Example
A service company takes requests in three ways: by email, as a photo of a fault in a messenger, and by phone. The model turns the call into text, describes the photo in words, and puts all three modalities into one request card for the engineer.
Related terms
Questions and answers
Sources
Not sure whether your company needs this?
AI solutions map