In competition with recent text-to-video models like OpenAI’s Sora and ByteDance’s Jimeng, Meta has introduced its beta version of Movie Gen, which can create 1080p HD video clips from text prompts and edit a user’s face into existing footage with synchronized sound.
According to a paper Meta put out about Movie Gen, the model is “pre-trained on internet scale image, and audio data,” with the ability to generate new AI-powered video, personalized videos conditioned on a person’s face, as well as edit existing footage or still images.
advertisement
advertisement
For example, in Meta’s announcement, the company shows a picture taken of a real woman’s face, which is then transformed into a video of her sitting in a pumpkin patch, sipping a generic-looking latte.
Another example shows a user instructing the model to alter a video of a young boy’s lantern into a bubble rising toward the sky, while another highlights the text prompt entered by a user who describes a fully imagined scene of a girl running down a beach in “blue jean shorts and a yellow t-shirt” and flying a kite in.
The high-definition videos can be up to 16 seconds long, and the audio is also AI-generated, designed to compliment the imagery with ambient noise, sound effects, and background music.
Meta’s personalized capability may be the most notable aspect of Meta’s Movie Gen release, as it sets it apart from other leading tech companies’ text-to-video launches, such as OpenAI’s Sora, which launched in February with the ability to create videos up to one minute long and replicate video-game environments like Minecraft while controlling the player character.
ByteDance’s recent release of Jimeng AI in China contains many of Sora’s basic functions, but stoked interest in the market because of its potential integration opportunities with the company’s massively popular social media app TikTok.
x wrote on Threads that the company isn’t ready to release Movie Gen as a product any time soon due to high expenses and the generation time being too long.
However, were Meta to integrate its Movie Gen technology into its family of social apps, the tech giant could have a profound effect on the future of social media, creator culture, and
advertising opportunities.