Hollywood studios are now experimenting with a cutting-edge technology called lip-sync dubbing. This innovative technique uses artificial intelligence (AI) to realistically alter the mouth and facial movements of actors on screen, synchronizing them with the dubbed audio.
Unlike traditional AI dubbing, which focuses on creating speech tracks, lip-sync dubbing concentrates solely on the visual aspect of localization.
Studios have been collaborating with localization networks to use human actor voice performances for the dubbed audio. The technology can even translate text shown in scenes, such as street signs or TV news tickers.
The potential benefits of lip-sync dubbing are significant. It promises a more immersive experience for audiences and the possibility of creating global hits. Although it represents a new cost for studios, they believe it will pay off by expanding and engaging audiences more deeply.
"The difference did shock us all how much better it was," said Scott Mann, co-CEO at generative AI firm Flawless. "The more exciting [uses of this tech] are doing things that VFX could never do."
Several major Hollywood studios are currently testing lip-sync dubbed versions of films and TV shows to determine if it improves audience reach and engagement enough to justify the investment.
Flawless is focusing on the five biggest U.S. studios, providing each with a capacity of 1,000 hours over the next 12 months.
Studios are devising strategies for these market tests, deciding on the type of content and specific languages to prioritize. They are also considering whether to localize English-language content for international markets or vice versa.
While lip-sync dubbing will need to prove itself among audiences, some may be more receptive to it than others. Studios may not always want to disclose when the technology has been used.
An exception in the 2023 SAG-AFTRA agreement allows studios to use lip-sync dubbing without securing consent from actors.
As this technology continues to evolve, it will be fascinating to see how it shapes the future of content localization and global entertainment.
CONCLUSION
Lip-sync dubbing represents an exciting new frontier in content localization. As studios experiment with this AI-powered technology, it has the potential to revolutionize how audiences experience films and TV shows worldwide.
