MMOOZZGG.com

Glossary

What is text-to-video AI?

In short

Text-to-video AI is video generation from a text description. A neural network takes a phrase or script and creates a short clip with motion, light and camera work.

Images, video, audio · text-to-video

MMOOZZGG.com editorsChecked: 1 min read

For a child

Remember a notebook with little people drawn in the corners of its pages? If you flip it fast, they start to run. Some programs let you describe a scene in words, and they draw many such pages in a row to make a cartoon. The hardest part for them is to keep the hero the same from frame to frame, so that he does not lose his hat.

In plain words

Video is many pictures in a row, usually dozens per second. The network does not draw them one by one. It draws them in a batch so that neighboring frames agree: a car moves smoothly, and the hero’s face does not change. You write, for example, “a drone flies over an autumn forest at dawn”, and a few minutes later you get a short clip. Many services can also bring a still photo to life or continue a video you started. The longer and more complex the scene, the more often odd things appear: extra fingers, objects that vanish, broken physics.

For business

Video generation makes draft and support clips cheaper: ad storyboards, backgrounds, short inserts for social media, a product shown in different interiors. A film set with a crew is replaced by hours of work from one person who runs the model, but final editing and sound stay with people. You pay per second of video or by subscription, and running your own model needs powerful graphics cards with a lot of video memory. The risks are rights to the result and to people’s likeness, accidental resemblance to real persons, errors in product details, and lost trust if viewers do not realize the video is generated. Decide before production starts where generation fits in the company’s video channel and where you need live filming.

For a specialist

Text-to-video generation extends the diffusion approach to the time dimension. In Video Diffusion Models (Ho et al., 2022), an image architecture was extended with temporal processing and trained jointly on images and video, and long clips were produced by conditional sampling on frames already created. Make-A-Video (Singer et al., 2022) showed that the link between text and image can be learned from text-image pairs, while motion can be taken from unlabeled video, with no text-video pairs at all. Modern systems usually compress a clip with a video autoencoder into a latent space, cut it into spacetime patches and process it with a diffusion transformer. The main difficulties are temporal consistency of objects, physical plausibility, clip length and the cost of compute. Modes of work include text input, image input (image-to-video), and continuing or editing an existing video.

Example

A travel agency makes short clips about destinations for social media. A neural network generates views of coastlines and streets from a description, and an editor adds real prices, the logo and a narrator’s voice.

Related terms

Questions and answers

What does video generation mean? It is the creation of a video clip by a neural network from a text description, an image or another video. No camera or actors are needed: the scene is described in words.
Can you use a generated video in advertising? Yes, if the service’s terms allow commercial use and the clip has no other brands or recognizable people without their consent. Ads are also checked against advertising law.

Sources

Not sure whether your company needs this?

AI solutions map
Contact us