Have you ever enjoyed the dubbed movie without paying much attention to its story, but love to watch mismatched lip movements and facial expressions. The AI software won’t let you enjoy this anymore as it can manipulate an actor’s lips and facial expressions to make them convincingly match the speech of someone speaking the same lines in a different language. The AI-based tech could reshape the movie industry and as we know every coin has two sides.
This technology is related to deepfakes, which uses AI to paste one person’s face onto someone else. It allows directors to effectively reshoot movies in different languages, making foreign versions less irritating and ridicule for audiences and more faithful to the original. As it can automatically alter an actor’s face so easily might be controversial if not used carefully.
As reported by Wired, the AI dubbing technology was developed by Flawless, a UK company cofounded by the director Scott Mann, who claims that he became tired of seeing poor foreign dubbing in his films.
How does AI-based dubbing technology works?
AI-based dubbing technology involves capturing the facial expressions and movements of an actor in a scene as well as someone speaking the same lines in another language. Then it is combined to create a 3D model that merges the actor’s face and head with the lip movements of the dubber and finally, it is digitally stitched onto the actor in a scene. Flawless drew inspiration from this project for its product.
Scott Mann and his association with Flawless and Deepfakes
After realizing that how the dubbing ruined carefully crafted scenes in the foreign version of his most recent movie, Heist which stars De Niro. Mann claims that dubbing sometimes involves changing dialog significantly and, to make it more closely match an actor’s lip movements.
“I remember just being devastated,” Mann told Wired in an interview. “You make a small change in a word or a performance, it can have a large change on a character in the story beat, and in turn on the film.”
When Mann started to find the solution for his problem, he found deepfakes and met Christian Theobolt who was heading the AI-based technology dubbing project. Christian Theobolt is a professor at the Max Planck Institute for Informatics in Germany.
Mann says the company is in discussions with studios for creating foreign versions of many movies. Other demo clips Flawless showcased were Jack Nicholson and Tom Cruise delivering famous lines from A Few Good Men in French and Tom Hanks speaking Spanish and Japanese in scenes from Forrest Gump.
As per Mann’s claim, Flawless is also looking at how its technology could help studios avoid costly reshoots by having an actor appear to say new lines. But he says some actors are a little unsecured when they see themselves manipulated using AI.
“There’s a fear and wow—they’re the two reactions I keep getting. It’s going to be invisible pretty soon,” said Mann. “People will be watching something and they won’t realize it was originally shot in French or whatever.”
What Virginia Gardener’s said about AI-based dubbing technology, Deepfakes?
Virginia Gardener, an actress who stars in Mann’s latest film said this when she saw herself speaking in Spanish, “I think this is the best way to be able as an actor to preserve your performance” in another language. She added “If you’re trusting your director and you’re trusting that this process is only going to make a film better, then I really don’t see a downside to it.” Virgnia Gardener doesn’t seem to be unsecured rather suggested that films modified with AI should include a disclaimer.
Advantages of Deepfakes
AI video manipulation can be controversial for good reason. Free deepfake programs that can unnoticeably swap one person’s face onto someone else in a video scene have proliferated with advances in AI. This software identifies key points of a person’s face and uses machine learning to learn how that person’s face moves.
Disadvantages of Deepfakes
This AI-based technology has been used to create fake celebrity porn and damaging revenge-porn clips targeting women. Experts worry that deepfakes can show a famous person in a compromising situation might spread misinformation. This technology could also be used for propaganda and fear-mongering.
Different opinions on Deepfakes
In 2019, a report warned that AI might not be able to protect the potentially disruptive effects of deepfakes on society. Study authors Britt Paris and Joan Donovan said that deepfakes are part of a long history of media manipulation and would require a social as well as a technical fix to be resolved.
Darryl Marks, the founder of Adapt Entertainment, a company in Tel Aviv that is working on another AI dubbing tool, says he isn’t sure how some actors react to seeing their performance altered, especially if it isn’t clear that it’s been done by a computer. “If there’s a very famous actor, they might block it,” said Marks.
“There are legitimate and ethical uses of this technology,” says Duncan Crabtree-Ireland, general counsel of the Screen Actors Guild. “But any use of such technology must be done only with the consent of the performers involved, and with proper and appropriate compensation.”
Hao Li, a visual effects artist who specializes in using AI for facial manipulation, claims that movie directors and producers are showing a growing interest in deepfakes and related AI technology. Li is working on a movie in which more advanced deepfake Softwares are used to let a director see an actor transformed in real-time.
Li says the situation feels similar to when the use of photo-realistic computer graphics became available in the 2000s. Now, thanks to AI, “suddenly everyone wants to do something,” he added.
Now, it would be interesting to see that how deepfakes can be beneficial or they end up in the hands of the wrong people.
You can visit Flawless’ website to view scenes from different movies being translated almost to perfection. While watching it you almost think that the actors are speaking French, German for real.