How one company’s controversial cloning technique is changing Star Wars forever
Respeecher is the world’s leading company for “voice cloning.” But they still need human input.
Luke Skywalker has been cloned.
It’s not the plot of a Star Wars novel from 1990 or an outlandish fan theory for Mandalorian Season 3. This is real life, and to get a little more specific, it is Luke Skywalker’s voice that has been cloned. Using artificial intelligence, Ukrainian tech company Respeecher makes a business out of on-the-nose oral mimicry.
Respeecher’s ground-breaking process is driving the future of de-aging technology and digital creations in film and other media, including Star Wars. Essentially, they take what the character (portrayed by an actor, of course) sounded like years ago and manipulate the vocal tones and characteristics to form new lines to use in the present day, years after the actor themselves retired.
HORIZONS is a newsletter on the innovations and ideas of today that will shape the world of tomorrow. This is an adapted version of the June 16 edition. Forecast the future by signing up for free.
“In general, there are two approaches to synthesize a voice,” Respeecher founder Alex Serdiuk explains to Inverse. The critical difference comes down to the performance, he says.
“The one the most well known and would be text to speech like an Alexa speaker or Siri. What we do obviously isn’t like that.”
To Respeecher, the work they do is art. But accurate replication of someone’s voice and the ability to make the oral clone “speak” also sounds like a handy tool for fake news actors and misinformation peddlers. Amid the war in Ukraine and Russian disinformation campaigns, Respeecher is all too aware of the dangers of this technology.
Respeecher has no desire to rewrite history, however. Inverse sat down with Serdiuk to learn more about how his company developed and uses this divisive technology to bring past Star Wars characters back to life on screen, and he reveals to Inverse the one limitation to the tech that could act as a guardrail against its use for disinformation.
The Respeecher team in Kyiv, Ukraine.
Act natural
Founded in 2018, Respeecher’s mandate is simple: Clone human speech convincingly.
To do this, Respeecher teaches their artificial intelligence to simulate how a specific person’s voice sounds — but not how a person talks. Founder Alex Serdiuk breaks down this distinction by drawing a line between cadence and voice.
The point of their technology is to try and figure out how to “change the vocal timbre” of a voice. Respeecher calls this “voice cloning,” an interesting analogy, since a physical clone also doesn’t do the exact thing the original did, but in theory, looks exactly the same.
But just like a physical clone needs the original body’s DNA, Respeecher needs a voice input to make its process work. Ultimately, it depends on human performance.
“We can’t synthesize a performance.”
As first revealed in a behind-the-scenes special, Disney Gallery: Star Wars: The Mandalorian, the voice of Luke Skywalker — in both The Mandalorian and The Book of Boba Fett — was “de-aged” using Respeecher’s technology. Magazines like Esquire and other media outlets suggested that young Luke’s voice was “completely synthesized.” But that interpretation isn’t entirely accurate. Respeecher didn’t just input a bunch of audio files into an A.I. and then have it spit out a de-aged Luke voice. It needed Mark Hamill.
“Professional voice input is extremely important,” Serdiuk explains.
“We don’t actually have any control over performance. That’s something that our technology doesn’t do. It doesn’t create a performance.”
Serdiuk uses a hypothetical dream project — having an A.I. Freddie Mercury perform live — as another example.
“Training the technology to make another singer sing in a particular voice is something our technology can already do. We can recreate the vocal apparatus of a particular person in a particular moment of their life,” Serdiuk explains.
“But the second part is still on humans. The whole performance. Speaking style. Inflections. All those tiny things that make our speech,” he adds. And it is not foolproof.
“[The technology] has problems with whispers. We don’t currently have control of that with our system.”
Creating an artificial Luke Skywalker
Respeecher is nothing without a baseline human performance, Serdiuk says. So, how did that work for Mark Hamill in the Season 2 finale of The Mandalorian?
Essentially Old Luke and Young Luke coalesced within Respeecher’s A.I.
Luke Skywalker's appearance in 'The Mandalorian' Season 2 was a big deal. But what about how he sounded?
“In general, there are two approaches to synthesize a voice,” Serduik explains.
“The one the most well known and would be text to speech like an Alexa speaker or Siri. What we do obviously isn’t like that. We can’t synthesize a performance. That means that someone should be saying the lines that are needed to be converted into a target voice.”
Here’s how it works: Mark Hamill (or another actor) delivers a basic vocal performance, encouraged by directors and all actors involved. Then, Respeecher gets to work, using the performance as its baseline.
For Luke Skywalker, countless sources were used to recreate a younger Mark Hamill voice. In this case, Respeecher aimed for Mark Hamill circa 1983. Even though Luke only says a few lines in both Mando and The Book of Boba Fett, Serdiuk explains that to teach their A.I. how to talk like Young Luke, they needed hours and hours of audio from that period to train the machine.
Mark Hamill in 1984, a year after Return of the Jedi hit theaters.
“Some of the recordings were quite old,” Serdiuk says.
“Some of them were from tape. There was an audiobook. There were some ADRs from the movies, as well as some video game recordings.”
“When you work with data that was created almost 40 years ago, at the end of the day, you need to mix it in the modern production and make it sound like it was recorded yesterday,” he explains.
Intriguingly, Respeecher is listed in the credits of Obi-Wan Kenobi Episode 3 (the first appearance of Darth Vader in that series) Serdiuk can’t confirm how much work he and his team did on de-aging the voice of James Earl Jones.
“I can’t say yes or no on James Earl Jones,” Serdiuk says.
There are, he says, “lots of secrets still with Obi-Wan Kenobi. But we were uploading data packets [for Obi-Wan Kenobi] to Skywalker Sound when the invasion began.”
Ben versus Darth Vader in Obi-Wan Kenobi.
Ironically, Serdiuk admits he wasn’t a huge Star Wars fan before getting Lucasfilm as a client. But now, he very much is.
“I became a way bigger Star Wars fan,” he says. “Interfacing with the story made me into a big fan.”
A Base Under Siege
The invasion Serdiuk refers to is, of course, the Russian invasion of Ukraine. Respeecher is based in Kyiv, which has seen significant shelling as a result of Russia’s military offensive.
When Inverse asked how Respeecher keeps working during the war, Serdiuk shrugs defiantly.
“The war is not new to us,” Serdiuk says, sounding almost like someone who might fight for the Rebel Alliance... a Ukrainian Cassian Andor, perhaps.
“Once we know the limitations [of a technology], that can protect us.”
“Russia started the war more than eight years ago,” he says, referring to the 2014 annexation of Crimea.
“We’ve been preparing as a company. We've had some contingency plans in place and we managed to relocate part of the team before this full-blown invasion in the end of February started.”
Alex Serdiuk, CEO of Respeecher.
On the horizon...
While Respeecher keeps tinkering with Star Wars’ most classic characters, it is worth looking at the company’s earlier projects to understand where this technology is going.
In 2020, Respeecher released a short film called In the Event of Moon Disaster. The idea was to demonstrate just how good their technology was at cloning a voice. In the film, you can see and hear Richard Nixon making an announcement regarding a massive disaster on the Moon in 1969. The video is convincing, but it is also fake.
Was this the end of history? Did Respeecher prove that news and history could be manipulated at will?
Serdiuk says the goal of the film was just the opposite.
“Obviously, there are some malicious purposes of using voice cloning technology,” he says.
“But we used to be scared of Photoshop, we thought that most of the images on the internet could be manipulated. Our grandparents believed everything was written down with someone with a typewriter,” he muses.
“But once we know the limitations [of a technology], that can protect us. Our goal is about bringing awareness of the existence of such technologies,” he says, adding that doing so may democratize the technology.
“That’s how you save history,” he adds.
HORIZONS is a newsletter on the innovations and ideas of today that will shape the world of tomorrow. This is an adapted version of the June 16 edition. Forecast the future by signing up for free.
Check out more about Respeecher here. Obi-Wan Kenobi is streaming now on Disney+
