Alibaba has announced Wan2.1-Vace, an open source AI model designed to shake up how videos are created.
Vace does not appear from thin air. This is part of the wider Wan2.1 family of Alibaba’s video AI models. And they argue that it is “the first open source model in the industry to provide a unified solution for a variety of video generation and editing tasks.”
If Alibaba can successfully avert users from not having to juggle multiple individual tools towards a single streamlined hub, it could be a true game changer.
So, what can this actually do? Well, first of all, you can use all sorts of prompts to liven up your video, including text commands, stills, and even snippets of other video clips.
But it’s not just about making videos from scratch. The editing toolkit adjusts only selected bits of images or specific frames to guide AI, advanced video “repainting” (more details), and even stretches the video. Alibaba believes these features are “enabled to flexible combinations of different tasks and increase creativity.”
Imagine creating a video in which a particular character interacts. Vace claims that it can. Have you got a still image you want to be dynamic? Alibaba’s open source AI model allows you to achieve that by adding movement that looks natural.
For those who like to fine-tune, there is the advanced “video repaint” feature mentioned earlier. This includes transferring poses from one subject to another, precise control of movement, adjusting depth perception, and even changing colour.
One of my eye-catching features is its “ability to support the addition, modification, or removal of selective areas of a video without affecting its surroundings.” It’s a huge plus for detailed editing. You no longer accidentally ruin the background when you’re trying to fine-tune one small element. Plus, you can make the video canvas larger and enter new spaces with related content to make everything look richer and more vast.
You can take flat photos and turn them into videos to tell you exactly how to pull out the path and move them. Should I exchange a character or object with something else that I provide as a reference? no problem. Do you want to animate the referenced characters? end. Do you control the pose accurately? You got it.
Alibaba cleverly extends an example of an open source AI model to widescreen video by taking tall, thin vertical images and automatically adding new bits and pieces by referencing other images and prompts. It’s pretty beautiful.
Of course, Vace is more than just magic. It involves clever techniques designed to handle the frequently wise reality of video editing. The key part is what Alibaba calls the Video Condition Unit (VCU), which “supports the unified processing of multimodal inputs such as text, images, videos, masks, etc.”
Next is what is called the “context adapter structure.” This clever engineering “injects the concepts of different tasks using formal representations of temporal and spatial dimensions.” Essentially, think about getting a really good understanding of time and space in the video into AI.
With all this clever technique, Alibaba believes Vace will be a hit in quite a few areas. Think about quick social media clips, eye-catching ads and marketing content, rugged post-production special effects for film and television, and even custom education and training videos.
Alibaba makes WAN2.1-VACE open source to spread the love of AI
This powerful model that builds AI models usually costs a lot of money and requires a lot of computing power and a lot of data. So, is Alibaba making WAN2.1-VACE open source? That’s a big deal.
“Open access helps more businesses to quickly and cost-effectively create high-quality visual content tailored to their needs by lowering barriers to using AI,” explains Alibaba.
Essentially, Alibaba wants more people, especially small businesses and individual creators, to get top tier AI without breaking the bank. The democratization of this powerful tool is always a welcome sight.
And they don’t just drop one version. For those with serious horsepower, there are 1.4 billion parameter models and a more agile 1.3 billion parameter models for lighter setups. You can hug your face and Github or get it for free now via Modelscope, the unique open source community of Alibaba Cloud.
(Image source: www.alibabagroup.com)
Reference: US Slam Brakes of AI Diffusion Rules, Harden Chip Export Curb
Want to learn more about AI and big data from industry leaders? Check out the AI & Big Data Expo in Amsterdam, California and London. The comprehensive event will be held in collaboration with other major events, including the Intelligent Automation Conference, Blockx, Digital Transformation Week, and Cyber Security & Cloud Expo.
Check out other upcoming Enterprise Technology events and webinars with TechForge here.
