Glossary
What is text-to-image AI?
In short
Text-to-image AI is image generation from a text description: you write what you want to see, and a neural network draws the picture.
Images, video, audio · text-to-image
MMOOZZGG.com editorsChecked: 1 min read
For a child
Imagine an artist you can tell in words: “Draw an elephant in a hat who is ice skating.” The artist listens and draws exactly that elephant, even without ever seeing one. Some computer programs can do the same. The clearer you explain, the closer the drawing is to what you had in mind.
In plain words
You write a phrase, for example “a cup of coffee on a wooden table, morning light”. A few seconds later you get a picture. Inside there are two parts. The first turns the text into numbers that describe the meaning of the phrase. The second draws the image and checks against that meaning at every step. These systems learned from a huge number of captioned pictures, so they know how objects, styles and lighting look. The description decides a lot: name the style, the angle and what must not be in the picture. The same request gives new variants on every run.
For business
Image generation cuts the cost of illustrations for websites, ads, presentations and product cards. Marketing gets dozens of banner variants in the time of one meeting and tests them on the audience before ordering the final design. The price depends on the method: a subscription to a cloud service, payment per image through an API, or your own model on your own server, where product descriptions and photos never leave the company. The risks are legal and reputational: unclear rights to the result, likeness to another brand, distorted faces and lettering, errors in product details. A neural network can distort a real product, so a person checks every card. Who may generate what is set in the company’s AI use policy.
For a specialist
Text-to-image generation is conditional generation: the model produces an image from a conditional distribution set by the description. A language or multimodal encoder encodes the text. The CLIP model (Radford et al., 2021) was trained with a contrastive method to match pictures and captions on 400 million pairs. In the two-stage scheme of Ramesh et al. (2022), a prior turns the caption into a CLIP image embedding, and a diffusion decoder draws the picture from it. Latent diffusion (Rombach et al., 2021) works in a compressed space and feeds text through cross-attention; common open models are built on it. Classifier-free guidance strengthens prompt following, a negative prompt sets what is unwanted, extra conditions through ControlNet (Zhang et al., 2023) fix pose and contours, and adapters add a style or a character. Quality is measured with FID and CLIP score, as well as blind comparisons with human raters.
Example
A tableware shop prepares a New Year mailing. The designer describes the scene in words, gets twenty variants of a festive table, picks three and refines them instead of ordering a photo shoot.
Related terms
Questions and answers
How does image generation work in neural networks?
Sources
- Radford et al., Learning Transferable Visual Models From Natural Language Supervision (CLIP), 2021arxiv.org
- Ramesh, Dhariwal, Nichol, Chu, Chen, Hierarchical Text-Conditional Image Generation with CLIP Latents, 2022arxiv.org
- Rombach et al., High-Resolution Image Synthesis with Latent Diffusion Models, 2021arxiv.org
- Zhang, Rao, Agrawala, Adding Conditional Control to Text-to-Image Diffusion Models (ControlNet), 2023arxiv.org
Not sure whether your company needs this?
AI solutions map