If you click the little gear for the settings on this video, you'll notice a feature that you may not have seen before, and not many videos have.
It actually lets you switch the audio track to one of several languages, and hear a dubbed
spoken version, instead of just having to read translated subtitles like you'd usually do.
Now this feature is limited access, not many channels have it, and I had to request access to it.
But once it rolls out, it could change a lot about how YouTube works for international audiences.
But having this feature poses a problem, because those dubbed translations were not made automatically.
So how do you make them, and how did I?
Well the story starts a couple weeks ago, when I made a tweet that basically
said how all the requirements already exist to do this kind of thing with AI.
There are tools to transcribe audio, translate it, and then use AI voices to speak it, and then you could sync it up with subtitles timing.
And it's even possible today to train a custom voice on yourself, and have you be able to speak in other languages.
Unfortunately that is very expensive right now.
So I was really surprised that there was seemingly no service that could tie all of these separate features together into one thing.
And I decided, just like I made the YouTube Spammer Purge app last year, that I would just do it myself.
And so I did, I made this open source Python program, it's on GitHub, and it's called Auto Synced and Translated Dubs,
very creative name.
And as you can guess, yes, that's what I used to create all the translated dubs for this video.
So that's what I'm going to talk about today, what the program does, how it works,
and even if you're not a creator yourself, hopefully you'll still find it interesting.
And maybe once this feature rolls out to more people, then you can tell your
favorite creators about it, or this might even have uses outside of YouTube I'm sure.
I should point out that YouTube actually does have an experimental project called
"Aloud" which does something similar to this, but it's not exactly what I had in mind.
First, the project is invite only, although I do have access to it.
Another thing is it only supports Spanish and Portuguese right now, which I wanted to be able to do a lot more than that.
Third, a key thing that I had in mind was the use of the subtitle timings to exactly sync up the speech of the dubs with the original content.
Whereas with Aloud right now, it requires a little bit of manual synchronization, otherwise there'll
be spaces and gaps, since the AI isn't speaking at the exact same speed as the original person.
And finally, while Google's AI voices are good, they're not the best, and I wanted to be able to use Microsoft Azure's latest AI
voices, which in my opinion are a lot higher quality, both in terms of how realistic they sound and literally the sample rate is higher.
And not to brag, but my program does address all of these shortcomings I just mentioned.
Now before I show it off, I will admit it's not nearly as easy to set up as Google's project, so I wouldn't consider it a replacement for it.
And I do hope that Google implements some of these features into their own project, because it's not really
that hard to implement if I was able to do it, and that way everyone can benefit in a much easier way.
So now I can get into showing you how the program works and what it does.
First and most importantly, you're going to need to get a subtitle SRT file for the video, and they really need to be human edited.
I know that saying this is kind of like the meme for how to draw an owl, but if you at least make a proper transcript, then YouTube does have a
feature where you can put in the transcript and it will automatically create the timings, and then you can just download the SRT file from there.
If you were to use the automatic transcription that Google gives you, which first of all
doesn't have any punctuation, you probably know it can mess up quite a bit sometimes.
So if you were to then translate it, it's just going to be extra bad.
And the reason you need a SRT subtitle file is because not only do they have the text, but they
also have the timings for each spoken group of words, which is the key to this whole program.
Anyway, once you have the SRT subtitle file, the program will use Google API to translate that text
into whatever language you set, and it will even generate a new translated subtitle file for you.
Then once it's all translated, for every one of the languages you choose, which by the way, all the configuration is done in some config files,
and basically once you set them up, you don't have to change them again, and there's a file where you put in the settings for all the languages,
and then you can basically have them as presets, and then choose which languages you want to enable to translate for any given run you're doing.
So going back, for every single one of these languages, it's going to feed the list of text lines to the text-to-speech
service from the subtitles, which will then synthesize a bunch of audio clips in an AI voice for that particular language.
But here's the problem, we want all of these audio clips to be as synchronized with the original video
as possible, both in terms of where they're located, but also the speed of how these are being spoken.
Now we do know, based on the subtitle timings, how long each group of text should take to
speak, because we simply subtract the end time to the beginning time for that group of text.
But at the moment, none of these text-to-speech services let you tell it, "I want the
voice to speak at this speed such that it ends up this exact duration in milliseconds."
Or at least there's no direct way of getting it to do that.
That's foreshadowing.
So we now have all the audio clips, and we know how long each of them should be.
So the program actually offers two options for how to get it the right length to put in.
One option is to simply stretch or shrink every single audio clip to be exactly how long you need each one to be.
Then to assemble the final audio track, the program will basically look at the length of the original video and
make a silent audio track exactly as long as that, and then overlays each audio clip exactly where it should be
based on the subtitle file onto that silent clip, and then once it's all together you have the translated track.
The downside is, "time-stretching" is what it's called, shrinking or extending the audio clip, degrades the audio significantly.
And this is despite me searching out and using what I believe is the best freely
available time stretcher algorithm program thing, which this program uses.
In a little bit I'll play the audio clip comparing the stretched audio verse another technique and you'll be able to hear what I mean.
Now that being said, I will say that whatever YouTube uses to stretch and shrink the sound for how you set the play speed, theirs is fantastic.
It has like no degradation in quality, but unfortunately whatever YouTube uses, I'm sure it's proprietary, so I can't use that.
So to solve this and get both a maximum quality audio clip that is also the
exact correct length, I added a feature that is what I call a two-pass synthesis.
With this, once the program gets the audio clip back from the text to speech service, it will then again look at how long the
audio clip is, and then compare that to how long it should be from the subtitle file, and just gets the ratio between those.
And we can actually use that ratio with the text to speech service, because they do allow you
to input a variable for how fast you want the voice to speak compared to its default speed.
And the key here is it's actually synthesized at that changed speed.
It's not like they just synthesize it and stretch it like we did.
So we send a second speech request, this time with the adjusted speed, and it will
end up returning a file that is effectively exactly the correct duration we put in.
And it's not quite exact, but it's like an imperceptible amount of milliseconds different.
And the reason we couldn't do this before is because we didn't know how fast the default speed was going to speak that particular set of words.
We needed to do it first, look at how long it took to do it, and then tell it,
"okay, well you need to do it this much faster," and then it'll be exactly right.
And fortunately, the variable they do take in for that allows you to have enough decimal points where it is exact enough.
If you're wondering, there is an option in the program to also stretch the
second pass, so it literally is exactly down to the millisecond correct length.
But that is off by default because no matter how little you stretch the audio clip, it degrades it significantly.
So now I can give you an example of the two different techniques.
The first one being if you just do one pass and then stretch it, and the second one if you do two passes.
So with the original release of Windows Vista, you had a 30-day grace period where pretty much nothing happened.
So you probably heard the difference, the two pass sounded better.
The reason it's optional is because it's going to cost you twice the amount of API calls, because there
is a free tier of these text-to-speech services usually, but if you go above that, you're paying for it.
It might not be a big difference, but something to consider, so I gave it an option.
And then from there, it's the same as before, you simply take each audio clip,
put it where it needs to be in this audio file canvas, and then it's done.
And then it just does that same loop for all the languages you set.
And once you let it run through everything automatically, you'll be left with a bunch of audio dub files and also the translated subtitle files.
But wait, there's more, because now that you have all these audio files, you need to actually
attach them to the video file as additional audio tracks when you go to upload the file to YouTube.
So I made a separate script like that and included that also.
Although at some point I'll probably just integrate it into the main program.
This script just takes each audio file and gets the language from the file name, and then uses a very popular program
called FFmpeg to add the audio track with the proper language tagging to the video without having to even convert the video.
And if you go and look at the audio tracks in a media player, you'll see that they're all there.
Oh, and it even has an option to merge a sound effects track into each dub before adding it to the video.
So all the languages will have music or effects that are used throughout the original video.
But wait, there's still more, because YouTube lets you add translated titles and descriptions
so that those will show up in people's native language depending on their locality.
So yes, I included another script that translates that too.
You simply put in the title and description, and it will again use the Google Translate API for all the languages
you had set, and then put it into a text file from which you can easily just copy and paste them all from.
So this way, everything is covered.
Someone who speaks another language will see the title and description in their own language, they'll
see the subtitles there, they'll even hear the video in their own language from start to finish.
So I'm definitely going to start doing this on most of my videos going forward.
Like I said, I would like to be able to make a custom voice model so that all the languages were also spoken in my own voice.
But at the moment, that is too expensive.
Microsoft Azure does offer this service, and they even have in preview the
ability to do this cross-lingual thing where it'll speak in multiple languages.
However, you do have to pay for the compute time for training that model,
which is $52 an hour, and apparently it takes 20 to 40 hours to train a model,
so that's $1,000 to $2,000.
And you have no way of knowing how good it's actually going to sound until after it's done.
From there, using the model is a similar cost, it's $24 versus $16 for however many characters.
Although you do have to pay for endpoint hosting, which is basically where the voice model is available for use, and that's $4 an hour.
Although you can just suspend it or pause it when you're not using it.
Just make sure you do remember to suspend it.
Google Cloud also allows custom voices, but I don't think they let you do it in multiple languages yet like Microsoft Azure does.
I couldn't find any pricing on how much it costs to train the model,
I think it might actually be free with Google.
However, the catch is that you need to pay apparently $2,900 a month to host the model and actually use it.
And it's not like Microsoft Azure where you can pause it when you're not.
Another thing is Google says it'll take several weeks to train the custom model where with Azure, what, 40 hours?
So big difference there.
I would like to make a prediction though.
I have no doubt that eventually AI will become so good and so cheap that YouTube will just do
this whole thing with the transcription and dubbing for basically all videos automatically.
You won't even have to choose to do it.
Right now, I think the main limiting factor is the transcription ability, the speech to text.
Because while it might be good for ideal speeches where you're like this close to the microphone speaking clearly like
an audio book, in the real world for cases like me where I tend to talk kind of fast and use a lot of tech jargon, as you
probably know from watching YouTube and using those automatic subtitles, they're not exactly that great most of the time.
Now, if you're wondering about my whole workflow, like if you're looking to do this yourself, to transcribe the videos, I
actually use OpenAI's "Whisper" model, which is free and you can run it locally on your own GPU if you have a powerful enough one.
And I have found this model to be way more accurate than anything else I've seen,
even Google's transcription API.
The Whisper thing even does punctuation pretty decently.
I mean the first time I tried it, I put in a 4,500 word video that was 22 minutes long and it missed three words.
I was astounded.
It's not always that good, but it's a really good starting point from which you can
just play the video and then correct any minor changes you see as you go through it.
Also, another program in my workflow is called Descript.
It does a whole bunch of stuff, but I literally just use it for the transcription editing abilities.
Descript will generate its own transcript, but the OpenAI Whisper one is way more accurate I found.
Though fortunately Descript does let you just replace the transcript.
You paste it in and it'll sync that new one you put in.
And then what I like is there's hotkeys for really quickly changing the punctuation and capitalization of
words on the fly as it's playing, which most of the time is the issue with transcription services, even OpenAI.
Like the main thing is there's like a period in a wrong place,
so this is super easy to fix that.
Also importantly, I found the way that Descript exports the subtitles files, which it can do, is more suited for making dubs.
Because unlike YouTube's subtitles, where the timings sit literally up against each other
with no breaks, Descript's subtitles will actually start and stop exactly with the sentences,
so the dubs are even more accurate in timings.
Now I did add a feature in the program to account for subtitles like on YouTube, where it'll add a buffer time between the timings so that
when they get generated, there'll be a little bit of space between the sentences, so they're not just all strung together with no breaks.
But of course it's still going to be better if the subtitles are generated from the start to be exactly lined up with the speech.
Anyway, the program does have some more configuration options in the config files,
I'm not going to go through a lot of that, it's mostly just formatting stuff.
You can even customize like how much space is between sentences when the voice speaks, that sort of thing.
But if you're really interested, you can just look at the three config files, they're all explained in there.
So hopefully you found this video cool and interesting.
If you did, be sure to give it a big thumbs up so I know you liked it.
And if you want to keep watching, the next video I'd recommend is where I
talked about a speech enhancer AI tool made by Adobe, which I was blown away by.
So you can click that right there, I'll put the link here.
So thanks so much for watching and I'll see you in the next one.