Glossary
What is text-to-video AI?
In short
Text-to-video AI is video generation from a text description. A neural network takes a phrase or script and creates a short clip with motion, light and camera work.
Images, video, audio · text-to-video
MMOOZZGG.com editorsChecked: 1 min read
For a child
Remember a notebook with little people drawn in the corners of its pages? If you flip it fast, they start to run. Some programs let you describe a scene in words, and they draw many such pages in a row to make a cartoon. The hardest part for them is to keep the hero the same from frame to frame, so that he does not lose his hat.
In plain words
Video is many pictures in a row, usually dozens per second. The network does not draw them one by one. It draws them in a batch so that neighboring frames agree: a car moves smoothly, and the hero’s face does not change. You write, for example, “a drone flies over an autumn forest at dawn”, and a few minutes later you get a short clip. Many services can also bring a still photo to life or continue a video you started. The longer and more complex the scene, the more often odd things appear: extra fingers, objects that vanish, broken physics.
For business
Video generation makes draft and support clips cheaper: ad storyboards, backgrounds, short inserts for social media, a product shown in different interiors. A film set with a crew is replaced by hours of work from one person who runs the model, but final editing and sound stay with people. You pay per second of video or by subscription, and running your own model needs powerful graphics cards with a lot of video memory. The risks are rights to the result and to people’s likeness, accidental resemblance to real persons, errors in product details, and lost trust if viewers do not realize the video is generated. Decide before production starts where generation fits in the company’s video channel and where you need live filming.
For a specialist
Text-to-video generation extends the diffusion approach to the time dimension. In Video Diffusion Models (Ho et al., 2022), an image architecture was extended with temporal processing and trained jointly on images and video, and long clips were produced by conditional sampling on frames already created. Make-A-Video (Singer et al., 2022) showed that the link between text and image can be learned from text-image pairs, while motion can be taken from unlabeled video, with no text-video pairs at all. Modern systems usually compress a clip with a video autoencoder into a latent space, cut it into spacetime patches and process it with a diffusion transformer. The main difficulties are temporal consistency of objects, physical plausibility, clip length and the cost of compute. Modes of work include text input, image input (image-to-video), and continuing or editing an existing video.
Example
A travel agency makes short clips about destinations for social media. A neural network generates views of coastlines and streets from a description, and an editor adds real prices, the logo and a narrator’s voice.
Related terms
Questions and answers
Sources
Not sure whether your company needs this?
AI solutions map