what's going on everybody in welcome to
a video about neural networks that speak
every voice in this video is generated
from a neural network I know people will
claim they could told me you I could
hear the difference there's absolutely
no way I would be fooled by this I have
some of the best years not too big not
too small just right give it long enough
and you might realize something is wrong
but I think this voice is pretty
convincing it's another question of it
they're all kind of time where we can
trust audio or video it is right now we
live at a time we're almost full of our
information comes from text audio and
that you only know all that this can be
faked it's entirely possible that main
things we've already seen our heart were
faked like this
I wonder how people will use this
technology it seems like it would mostly
be used for bad things
what can we even do about this the
reaction of many will be to just bend
things like this and this is not
necessarily the answer
the truth is impersonating others for
nefarious purposes is already illegal
the bad actors from other countries do
not care about our laws they will add
this technology to their arsenal of
cyberattack whooping or Eve citizens
must remain vigilant wherever they get
their information and must remain aware
that the threat of fate audio and that
it was here now we live in an era where
most people get their information online
from many sources including social media
while this has allowed for more freedom
of the press which underpins our
democracy it has also along for foreign
regions to penetrate your flow of
information that is why tonight I am
proposing that we build an Internet wall
it is a horrible if we look at young or
Russia they do not have this crowd don't
think you understand how this wall would
stop the flow of then information might
be the dumbest idea I have ever this
wall you've ever seen okay but no this
would be a beautiful people would come
to visit just to see my internet wall no
Iestyn ate that the money made from
cheers to see the internet wall would
pay for the internet will continue SUIM
by the biscaia the issue here is just
like any fake news often the truth takes
a long time to get out and some people
will just never see the correction with
the last year's presidential election we
saw just how much fake information could
impact us by clogging up information and
that was just with fake text I wonder
how much more destructive fake audio and
video will be all right and back to
reality or is it just kidding or am I
anyway what I'd like to do now is show
you guys how you get started with this
what's involved and hopefully simplify
it a decent amount I had no idea how
complicated sound could actually be with
spectrograms and things like Fourier
transforms all this stuff so I'm not
gonna be getting too deep into the weeds
just kind of show you how you could do
what I've done and kind of what I found
and by no means done with doing fake
audio and the next thing I'd like to do
is fake video and then combine the two
so there's still a lot I'm trying to do
and I just I don't really feel like
doing a line-by-line thing but I thought
I would share with you guys what I found
so far so with that the model that I
ended up going with is this DC TTS model
which is I used this code here but it's
based on the following paper which take
note was written in late 2017 so almost
two years ago actually but you know the
history of doing text-to-speech on
computers is and has always been pretty
much someone goes into like a studio and
records hundreds of hours of audio and
then we use things like digital signal
processing or DSPs I'll probably refer
to it from now on to do things like
working on the pauses between words and
sentences and then do do things to do
like the speech pattern and stuff like
that it's all these like tools and and
hard coded rule-based
things for text-to-speech in most cases
even when doing Texas speech with neural
networks historically people have
applied more rule-based things to the
output now obviously the dream would be
to do text-to-speech without any of that
just simply throw in some examples get
output that is just awesome
so that was the goal and also side goal
is to figure out how can we do this and
how quickly can we do this and could we
just do this with like anything on like
YouTube like could we just take just
about any YouTube video a mimic a voice
on that video so with that in mind this
is the model I ended up with but I did
try other models including taco Tron and
Taco Tron v2 and I found this one to be
the best which is kind of curious and
one of the things that you
might notice if you've been reading and
not listening we're both is this as deep
convolutional that's right it's a
convolutional neural network there is
not a recurrent cell to be seen which is
also interesting because traditionally
you would think of a task like this to
go from text to speech text is a
sequence speech is also it's not like a
single scalar value right it's a
sequence so most of the time you would
think I would use a sequence to sequence
model which is a recurrent neural
network nope
this is using deep convolutional neural
networks with attention now this is
guided attention which I'll explain in a
moment but interestingly enough it's
turning out over the last couple years
to be the case that deep convolutional
neural networks are actually
outperforming recurrent neural networks
at these sequence to sequence types of
tasks they're doing it in their training
faster and they're getting better
results and that's awesome
I'm not sure that I'd be ready to say
recurrent neural networks are dead we
just need a better way to do recurrent
neural networks because the biggest
thing is the amount of data that's
required and how slow they are to Train
so if we could speed them up we might
see good things so for example I recall
using a recurrent neural network on like
Emma Stata so it's not like recurrent
neural networks aren't like they're
they're just as trying to think of a
topic wide as a convolutional neural
network can be so I wouldn't necessarily
write them off completely
don't forget normal networks themselves
got written off completely not long ago
so anyway yeah cool so what's guided
attention guided attention is this it's
basically the assumption that everything
is going to be linear so the input to
the output will always be linear as
opposed to consider a machine
translation task where let's say you're
going from English to Spanish and you've
got big house needs to translate to Casa
Grande okay well the adjectives to
describe that noun in English comes
before the noun but in Spanish comes
after the noun so it's not a linear
relationship in all cases in other words
might be but
are gonna be flipped around and so
attention is there to actually help with
this sequence the sequence which was
previously mostly used for recurrent
neural networks but now we're using them
in convolutional neural networks as well
and it turns out with convolutional
neural networks plus attention we can do
the exact same thing with the added
benefit of doing it way faster the other
nice benefit that I found is the end
result is just better so really cool so
for example I just want to play a quick
example of let me see where it is yeah
so of a Texas speech that is pretty
close to the typical ones I would find
by training a mediocre sample size of
say 600 samples so I'm just gonna play
one real quick I that is one okay um you
probably might not even understand what
is being said there but I'll play it one
more time it's saying Python is the one
and only true programming language so
yeah that doesn't work well so that's
Taco Tron on 500 samples one thing to
note Taco Tron requires hundreds more
like a hundred plus hours of sample
audio and weeks to months to train so I
did not do Taco Tron justice also
there's now Taco Tron v2 so um just
understand that that's just an example
but I wanted to show you because every
time I've ever played with
text-to-speech that's about what it
sounded like like in terms of Texas
Beach with neural networks so there's
something there you can tell the the
neural network is kind of grasping on to
things but it's not it's just not there
it's just simply not there so then I
trained this deep convolutional neural
network with this this guy's code and
I'll try to put a link in the
description if I forget someone remind
me you can also just google like DC TTS
github and I'm confident you'll find
this page but anyways I trained this
model and this one is on the LJ speech
data set which is here which is a female
voice I think it I don't even know I
don't I don't know if it's Kate
Winslet's boys because there's he's got
these other
sample audios you've got kate winslet
and nick offerman and then i'm not sure
whose voice it is but it's a singular
voice you need to train with just one
voice otherwise you'll get a meshing of
voices which maybe that's what you want
but probably not
anyway I use the LJ speech data set and
trained for I want to say about like
three days like the first one the so
this this model has two networks I'll
explain that more in a moment
the first one took about one day the
second one took about two days and this
is the result that I got now this voice
is quite good in my opinion and it was
trained with many hours of sample audio
from an audio book okay so that is still
a little robotic you can if you really
pay close attention you can kind of hear
these weird little fluctuations with
somebody who knows more than I do I
really don't know much about digital
signal processing and all that stuff
someone who's really good at it could
fix that voice and use like something
really specific probably for that boys
and fix it but this is straight from the
from the DC TTS model output from here
on github and that's pretty good I mean
that's clearly a person we would accept
listening to that audio I think I was
just I was pretty staggered by those
results because I've never seen in a a
Texas speech model produce such results
besides Taco Tron which is a big pain to
train now that data set consists of
13,000 samples and it's about 24 hours
total of audio it's a lot of audio so my
initial thought was what can I do this
with YouTube and how little how few
samples can I use so that I started I
try to use my own videos because I felt
like if I'm gonna fake someone's voice I
probably should do my my own first and
then do other people's it's kind of like
if you're a police officer and you want
to have a Taser
they make you tased yourself first but
it's kind of the same thing so so so so
my hope was that I could take my youtube
videos and then also take the caption
file for the YouTube videos so I tried
that but the problem is the caption file
doesn't quite like the timestamps on the
caption file aren't quite right and then
the caption file also eliminates often
things like um and
and other like stutters I'm stuffing so
like it's just because of the guided
attention this is such an integral part
here because the assumption is
everything will be linear everything
will line up we are saying with 100%
certainty so we need it to actually line
up so it has to be perfect
unfortunately caption file no go so that
I'm thinking well maybe captions are
that way because because it's better for
the person to read it so you could read
it faster if we don't do include the
stutters and stuff so then it's like
okay let's try using like Google's
speech-to-text try that similar problems
it would chop things it would leave
things out it would fix stutters it
would get rid of ohms and us so then I'm
like well I guess I'll manually
transcribe this stuff and I don't think
probably anyone watching this video has
manually transcribed audio but let me
just say it is the worst thing it is the
most painful thing I can't think of a
thing more painful than transcribing
audio at least unless it's like really
clean audio so in the in my case there's
a lot of stutters there's the this and
lots of us and lots of times I like
repeat a word twice or three times and
some of its really hard to hear and you
have to go back and play in like at
least what I found personally is my like
I just naturally will Nix
when people say and um like a lot of
times I just like don't even hear it so
I'm sitting here typing it out and then
usually I would play it back one more
time or two more times after it was
fully typed out just to make sure and it
was just staggering how many times I
would miss things even like two twice I
would miss things when I'm like really
trying to pay attention to it so really
interesting and also just painful just
painful to uh to transcribe things so
then I'm like oh my gosh because they're
you know the the LJ speech dataset is
thirteen thousand samples I was finding
I could do about a hundred samples an
hour transcribing um so and then I could
do it max about like 200 a day beyond
that it was like just killer just too
too brutal
and so I do the math that's two months
to get a new voice and cool truck
outside my house I love it I love one
you're recording like every time I
record big trucks have to come by yeah
okay so after manually transcribing
about 600 samples the result I got was
Python is the one true programming
language so that one's not too bad
there's a couple of small issues one is
it sounds like I'm underwater the other
one is you can hear these like little
flickers in the background and I think
that's the model actually trying to
recreate the sound of the keyboard which
is cool but also not necessary don't
really want that
in the audio unless I want to add it
after the fact so it sounds better but
clearly we need more work and it's just
not great it's not as good as that
original voice so then I'm thinking to
myself man do I really like am I really
gonna have to do thousands of samples
because generally you know your initial
results are going to be pretty good and
then you know it's this law of
diminishing returns so the more samples
you get the less you know improvement
you're gonna see per new sample so so
now I'm just like man what do I do and
then I think well what about transfer
learning so I decide well this model
sounds good now this voice is quite good
in my opinion and it was trained with
many hours of sample audio from an audio
book so could I take that that model
right but the two networks do a transfer
learn could I get there so so my next
question was okay well like time to look
back at the model and figure out how
everything's working so let me explain
briefly how this model actually works so
this DC TTS model basically the way it
works is it's got two models the first
model is a text - Mel is what they're
calling it and this is basically the
conversion of text to a Mel spectrogram
then the second model and that's also
where your attention is then the second
model this is your SSRN or
spectrograms super-resolution Network
and that one just basically kind of
clarifies everything else so my question
was what if I take that voice that
there's this female voice feed in say a
few hundred examples of my own voice and
then do a few more thousand steps of
learning effectively just doing a quick
transfer there could that work and what
I found was first I had to figure out
how many steps to go and at least what I
found was it takes about somewhere
between two and fifteen thousand steps
on the on model one the text to mel
takes about that many steps and then on
the ssrn it really takes like one to
four thousand steps it really no more is
required I was kind of baffled by how
quickly that one learns I'm not really
sure why that's learning why the super I
guess because all that model really is
doing is learning how to take a mel and
convert to this like super resolution so
maybe that's it but I will say you have
to train it you can't not train it
otherwise the voice just will sound
wrong so what I found was to train so
for example are before I show you some
pictures to train let's say zero to you
know 15,000 steps that's it that's our
week one to 15,000 steps that's about
like 3 to 45 minutes it was like a step
one thousand steps per 3 minutes so
that's for model one and then to do
model 2 it would take you know to do two
to four it would take about 30 minutes
okay let's say so all in all about an
hour and 15 minutes to train a new model
now the question was of course how few
samples could we do this with so first
let me play the voice I got I think this
one's an early transfer I'm just gonna
play it I'm not Iying a human like you
yeah this is an early train it sounds
like I don't actually have this properly
labeled I a human like you and it sounds
to me like it it's a transfer of one so
model one the text too
but not the ssrn but then after
significant training on both sides this
we get a voice like this which i think
is pretty good so that sounds pretty
darn close to me we get a voice like
this which i think is pretty good
there's a few pronunciation issues but
again the pronunciation issues will go
away the more samples that you have so
in this case that voice is off about 200
samples so not many at all and to to
transcribe that many samples it would
take maybe two hours so what I did was I
basically we went from taking like over
two months to create a new voice brought
that down to about three and a half
hours for anyone's voice given that you
have approximately 200 decent samples so
and again each one is anywhere between
two and ten seconds I think the average
is usually around like six or seven or
something like that so about fifteen
minutes of audio so you just need 15
minutes of audio you can try with less
but then you get weird things so for
example let me play let me play this one
for you hello my name isn't Stan I'm not
the machine so I call that one crisis of
identity because you can even hear it's
like flipping back and forth between the
female and male voice which is kind of
funny so okay so as I train let me just
show you kind of what I'm looking for so
I'm just gonna grab sent x3 just as an
example different voices seem to require
different amounts of time and if you
were sitting here physically you can
kind of more manually like if you see
the attention going in the wrong
direction you can kind of fix for it so
one of the really cool things I don't
know sir some of you guys have seen
attention graphs so this is at a
thousand steps two thousand steps you
can see it starts at least to like form
that line there and then about four
thousand steps there's some serious
attention here and just for the record
like without this guided attention this
takes a lot longer to happen so the fact
that this is already happening by like
four or five calyx these are already
looked
these are already pretty darn
good-looking attention graphs like dang
so now going backwards I'm just gonna
start us to this the female voice model
stopped here this is where I stopped it
now you could train this this one first
a little more and then do the transfer
like there's so many more things for me
to trial and error here and I've already
trial neared a lot of things but this is
where I stopped so then one thousand
steps with a new voice you can see the
attentions kind of it gets a little
confused the resolution gets quite poor
and then kind of loses attention towards
the end there but then by 187 it's also
not really that great 188 at least it's
sort of improving 189 is improving and
then we keep going I forget where I
stopped on this one if it's 199 that
one's pretty good not very good right
losing attention towards the end there
and basically I just keep training until
I see a pretty good one like so for
example this one's pretty good like look
how yellow that is man like yellow is
good right that's more closely aligned
basically that one's bad resolution and
199 I'm pretty sure it's the one I
stopped on yeah so I can keep going on
this voice but and you can see it kind
of loses attention towards the end but
that's still pretty good and that's the
voice you've been hearing as my fake
voice throughout this video so anyway
that's just like an example so you're
just you're just looking for these
graphs to kind of tighten up and turn
yellow and make it to the end without
this like a long a long line there
basically to simplify it so cool so
besides that I think that's really
everything I really wanted to show you
guys now moving forward there might come
a time where I do more of a line by line
tutorial but like I said I'm just not
done and so if you guys want to take
part in doing it it's really simple
download this person's code okay and
then basically you go into the data set
so this LJ speech data set if I click on
this measure was copy but metadata
contains all your transcriptions so if I
click on that that's not what I wanted
to open it with do I already have that I
don't have it open dang it
let's just open it in text editor you
can see here
separated by bar here but also take note
there's two translations I'm looking for
the bar I am not seeing where's that bar
okay I couldn't find it on the first one
but there's the bar here is the bar for
the number-6 basically and so you could
do you could make your own so for
example like when I was making my own
audio what I would do is I decided a new
one so I made my own by simply I created
a program it would start with a label
like this bar show me the texts it
wanted me to say I would say the text it
recorded it with some PI audio and then
output this file and then it output the
waves in this directory and these are
all your waves that match so the file
name here matches up with the name here
so if you wanted to make your own that's
how you do it now with the YouTube
videos like for Elon Musk boys and
Donald Trump and Obama and whoever else
I did for those unfortunate I wish the
captions worked I almost kind of want to
like throw away a guided attention and
just try regular attention because it
was it would be so much more cool to
just automate the entire process and I
still think I can't automate it but the
way that we did that was download the
audio split by silence try to make the
clips between 2 and 10 seconds long
split by silence of course and then I
would play it and then transcribe it
that way so it's still very very manual
and that's like the most that's the
longest part of the task and that's the
only part of the task that really
requires human input so it'd be cool to
automate that process for sure but
anyways I did find out this you could
definitely fake someone's voice with 15
minutes of audio which is honkers I mean
that's just crazy it's so it's a very
small amount of audio that you need and
the fact that you can do it in about 3
hours is also pretty crazy so moving
forward what I'd like to do is continue
looking to see if I can make these
voices any better so the fewer the fewer
of the samples that you add it starts to
pronounce certain things weird like for
example if you try to make some of these
other voices as I transcribe them the
term like neural network never came up
so when you would have them say neural
times a new role or something weird it
just didn't sound right
so wouldn't mind trying to figure out
how how can we get these models to be
just a little better and then maybe
maybe even use a bass voice or like have
different bass voices so if you start
with like a a Bridgette have a like a
British bass voice male and female a US
base voice male and female and then
start from there I think you'd probably
have better results because keep in mind
all I did was create a bunch of male
voices from a female voice a female
British voice and I didn't do any
British people so so yeah a lot of
things that we could definitely test
moving forward but anyway I thought this
stuff was really cool obviously there's
all kinds of ramifications here we we
you know I think most people are
completely unaware that this technology
exists now nowadays if you get a phone
call you can spoof the phone number
first of all but also we can now spoof
voices so you had a phone call from a
family member you have no idea right we
all know that strange emails with
suspicious links in them from family
members we probably shouldn't trust but
if you get a phone call from your family
member you're you probably believe it's
actually them but it might not be so
that's pretty crazy
also with like fake news and stuff like
that like in the last like 2016
presidential election I mean it was so
hard to keep up just with the amount of
fake information in text form that was
coming out now imagine like all the fake
information that can come out with fake
audio and fake video you know yes we can
validate this stuff but it takes time
just like it does with the text so you
know it's almost like a social denial of
service that they I feel like went on
during the you know the last election
and it's only get this kind of stuff is
only gonna get worse I mean just imagine
floods of fake information and text
video and audio form it's crazy this is
crazy I think this is a really important
field that people are paying attention
to so the other thing I'd like to work
on besides doing fake video in the forum
like I'd really like to create a text to
video so besides doing that I'd also
like to start working on adversarial
networks to this so just detecting fake
audio and video I think pretty much like
all social me
websites and news agencies are gonna
need this to start validating the stuff
that's coming through because I think
it's gonna be a big deal I mean it's
already a big deal and we saw how big of
a deal it was in 2016 I think it's only
gonna get worse
so anyways on that super light note
shout out to my most recent channel
members thank you guys very much for
your support
I'm gonna attempt to say all of these
names I'm sorry if I mispronounced feel
free to correct me in the comments
section uh Lukas son ready Casey Meehan
Peter Freeman black hot pranic Kothari
Jesse Jones Payne max Phillip Wagner
Dylan died and up dumb mein Anders
Nielsen I Adele Ajay
Jason Roberge not Brian Bradley in
Carter babban thank you all very much
all brand-new members welcome welcome
hope you enjoy the content okay so
questions comments whatever feel free to
leave them below if you've got any
suggestions on where to look I obviously
I know about deep fakes but I'm not
actually trying to do deep fakes for the
like text to video stuff what I'm quite
honestly trying to completely generate
things so there are there is some
research and talking heads but I've
never seen any code for it and the
papers are still kind of vague about the
model so I don't know if I'm gonna have
to just try to figure that one out all
on my own and by on my own I mean me and
Daniel so yeah it'll be it'll be really
interesting but any suggestions there as
well feel free to leave them below or
come hang out with us in discord GG /
centex
otherwise that's it guys see in another
video real or fake who knows