What I've been wondering is you know
If you've got a lot of pictures of horses and you really want to turn them into a lot of pictures of zebras
How are you gonna do that, right?
Marker pen. Yeah, marker pen, just draw some lines there. Yeah, that is quicker
Today we're going to talk about cycle-GAN, which is a really interesting
innovation in generative adversarial networks(GANs)
Now rob has already covered this in one of his videos what a GAN is and what it's used for typically
Generating images of interest. Pictures of trees, pictures of people, pictures of buildings, you know, you can train GANs to do these kind of things
What's interesting about GANs? Is that you actually have two networks
you have one their work that's learning to generate images and you have a second network that's learning to try and tell the difference between
The generated ones and some example real images and by training the both together
You force the generator to produce more realistic looking images
There's another domain that GANs are useful, right which is not generating an image from scratch
It's turning an image from one domain into another
so for example
you might take a photo to try and make you look like a painting or you might take a
Black-and-white photo and try and make it a colored photo or something like this
There are lots of ways to do this kind of style transfer but gans are one of the ways you could do it
It's gonna be slightly different to what what was talking about because in this case we have an image going in, right?
and not just noise and we have an image coming out
So one way you could train this is just with a standard kind of encoder decoder
Network like we've talked about before so you might have an image let's say here so this is a photograph
And you have a kind of encoder decoder
Networks like this that we're gonna call our generator and then it outputs another image
These aren't typically too big but they usually do filter the network down and back up again
And then we have I know an artist impression or something like that of the picture now
One way we could train this is to get a bunch of paintings and a bunch of photos of where those paintings were
sort of captured font, right or
Commission a bunch of artists to go out and paint the load of stuff that we have the photograph for and then we've got this
Supervised training data that we could use we could just put in pairs of photo
what it's meant to look like in painting form and then
even train a net weather
Hopefully we'll map one to the other and the idea I guess is that you then put in a new photo?
But it's never seen and it does a nice job of painting it right? That's the hope
So this is like a kind of basic level of training up a network to produce some kind of style transfer
But the problem is that it's a bit of a pain to have to get these pairs of data, right?
So a lot of the time the data isn't paired a lot of time. We have a lot of pictures of paintings
We have a lot of pictures of photos, but it all look completely different places
So producing a mapping between one and the other is a little bit more difficult
What a gam does is has a discriminator here?
So this is going to be a discriminator and this is is it a real or fake image the bet of a discriminator gets at?
Detecting the fake paintings the better mix generators will have to get converting photos to paintings, right?
That would be the idea and this works pretty well
Why and it works well for generating photos and third foot changing photos into other domains
But gans have a lot of problems with training that make them quite difficult to use in practice
One of them is something called mode collapse imagine that we are trying to produce again
Now that takes a picture horse and converts it into a picture of a zebra
So what's happening is we put in a random picture of a horse and then the discriminators?
Is it a real zebra or fakes everywhere?
And if it's determined to be a fake this generators gonna have to get better at changing that horse into a zebra
Horses into zebras is a kind of arbitrary
problem that you might wonder what why we do in that well
for fun mostly but also because actually it's just a textual check text will change the shape of a zebra is
Approximately the same as a shape of a horse which makes this problem slightly easier, right? We're talking about adding stripes
We're not talking about totally changing the shape of the object
the problem is that let's imagine that the generators have realized for discriminator how in terminator worked and
Just said okay i'm going to produce this exact same picture a zebra
This is gonna look bad by the way. So this is my picture of a zebra
How did I am good is it
Right again again again would have been better an untrained can work. I've been better now
So let's suppose the generator
Generate this same picture every time any picture of a horse cause it just generates that picture
It's a simple you asked was ever. Here's who's ever it's the only one you're getting man
This is something called mode collapse. The idea is that the generator has now completely failed to produce interesting pictures
Just producing the same picture every time
imagine we wanted a
Generator that produce digits like six five four three, and this was determining whether they were real did you or not?
well
if you just produce six every time it's problem is so much easier and we discriminate it can't
Complain about that because they are technically digits. This is the exact same problem
So cycle Gann is essentially a two gun system that deals with this very problem
So how does it do it? Well, what we have is we have two guns. Alright, because if one is good then two better
and what we try to do is we're trying to make sure that not only are we taking our
Image and turning into something that looks good. And it's in it's indistinguishable from a real one
But also that we can then go back again
To prove that we haven't just generated the same license image. It's got nothing to do with the input, right?
So, this is our input we have a generator a generator Network G
So this is G which is going to generate a picture of a let's say as of a zebra then we have another network F
Alright
Which is taking pictures of zebras and turning them back into pictures of horses
So there's two guns and there's going to be two discriminators, right?
This is a scream later here saying is this a real picture of a zebra?
Question mark to a discriminator and when as a discriminator here saying is this a real picture of a horse?
So there's a lot of loss functions being applied here
what's a loss function a loss function is a function we use to calculate how wrong or right a
Particular network is so in this case
The loss function is you were supposed to say this was a real image. And you said it was a fake image
So your arrow, is that is that much right?
Without going to any numbers. That's basically what it does right, but this is how we train these things, right?
You calculate a loss just say look you were supposed to say this was a fake zebra
But actually you the wheels Emperor so next time change your weight so that you say it was a feint ever
That's what the loss function is for. So a horse comes in the G generator turns it into a zebra, theoretically
Right. This goes into the F function
It turns it back into a horse and then we have a final loss which is what is the difference?
Between this horse here and this horse here because if we can't recreate the exact same image
Then we just produce a different picture whereas everywhere but it's that's cheating my for this to work
We have to convert this horse into a zebra not just produce a picture of his ever
You see what I mean? Right the same with something like star transfer to a Monet painting, right? We take a photograph
We turn it into a Monet painting. It looks great
But if we can't turn it back then what you've done is just draw a male painting for me
It's got nothing to do with the input
so in some sense this G and F function are inverses of one another one does of a function and the other one undoes the
function what's interesting about cycle gain apart from the fact this kind of a loop here is they have this loss that also measures a
Distance between this image and this image. So not only ensure that these are both
Producing realistic-looking inputs and outputs, but that these two at the end of the same and you can do it both ways
You can go horse
To zebra to horse and measure this distance here and to make sure they're same and you can go zebra to horse
As ever and make sure this distance is the same right so you can work in both ways
You can put F first or G first
So this is exactly how you train it
you take a number of images of horses and a number of images of zippers and
Then you train it just like again you say well, here's a horse image go in. Was it a real image or not?
Right, and sometimes you give the discriminative real images. Sometimes you give it fake images and it tries to get better with that
You don't put it through after to toe it back into a horse
back
Discrimination is also working to make sure this F function is better and then you make sure these two are the same as well one
Way of looking at this is that you're trying to separate out what the content of images and what a style of images
I'm the content should be the same
Alright the content of the horse image and the zebra image should be the same
Because if it's not you won't be able retrieve one from the other
The style is you know aspect how it looks my is it a painting? Is it a zebras at horse?
These are things that we can apply later kind of to the base image. That makes sense
And this can be applied to a whole array of different problems like this kids in medical. Imaging right?
We're looking at it in plant. Imaging
It's used for style transfer
So in the paper alone, there was about 10 different examples of things you can do
You can Kate you can take pencil drawings and turn them into into men doings of an object
so like, you know a boy of a shoe - an actual picture of a shoe you can go from
photos to Monet paintings to Van Gogh paintings
You can go from horses to zebras apples to oranges and the list goes on right and if you have a look online
You'll see all kinds of weird wonderful things that people are using this cycle can and variants of it
Too to do is a resolution. Good. No
It's okay, right the the resolutions are get so this kind of size of training data
We're talking about 30 centimeters is in the hundreds of pixels squared
So 2 to 500 pixels squared usually training them takes all the memory once they're trained you can put through
So if what we say inference time, you can put through bigger images and there were ways you could train it
so that
It will work on bigger images when it actually gets to applying it
Right, but the images I've been testing on a smaller 512 by 512 something like that. We're not converting for Herod is yet?
There are there is work that maybe we'll talk about overtime lifestyle gang which does try and produce higher resolution output what you'll find
I think if you watch a video of a cycle gun being applied to every frame of an image
It's gonna be little bit noisy is you're gonna have to take steps it tries to move that out. There's a few videos of like
Parts of 2001 a Space Odyssey as a Picasso painting where every frame is a Picasso painting
it's quite something to look at right but there's a bit of noise because you know
The input changes even slightly some is going to change on the output right and so it sort of flickers
To get it to get it to be smooth
You're gonna have to do some kind of extra processing
So I've come on this right you can you can download the code online. We'll put a link in the description
You can download it as long as you've got all graphics card
You can start putting
Random pictures of horses in turn them into random pictures of zebras and feel bit pleased of yourself
These kind of Gans are being used for all kinds of things that you see like a super resolution
For example way taking a very low resolution
Image and you want to sort of blow it up so we can see a lot of fine detail
You know, that's sort of CSI. It harms in arts it zoom in on that. I
have some concerns about this because
One thing to remember is that these are trained to look nice. They're not trained to be an accurate
representation of
The ground truth specifically right? There's no if you take a blurry image and you know,
I'm saying this to cover my own back more anything else if you take a blurry image and you zoom it up and it looks
Like me, that's probably a coincidence
You can't trust that it was actually me maybe more but it makes you know
But my point is like you're not gonna be a loser in a court of law
Because it could be argued that it just created an interesting looking face. It's not created the actual face from these few pixels
This it's gonna be a while before we're looking at
Networks that can actually work out what truly someone look like
If you look at the input and output of this cycle can for example
You'll see that it'll take a horse and turn into a zebra and when it turns it back the images are close
But they're not the same often. The grant is a little bit muted
I think probably because there's ever tend to be on dryer grass. I'm sort of the plains and stuff, right?
So it changes the rest of the image as well to deal with this
So these interesting things there does so super-resolution. It's one thing that's used for
Medical imaging converting from one domain of medical image like an MRI to a CT scan right? I'm quite worried
I wouldn't use case for that is but you know
There's a lot these sort of things can be done the recent Samson video where they had people's faces and things like deep fakes
Right, which are I would just say morally questionable, but these are all things generated using these Joseph's adversarial networks
You're taking inputs which are you know noise or other images or pose?
Estimations if you're trying to make a face do and when it's generating an image what looks like that thing
So there's lots of ways of controlling what different bits of this do to get an output that you want
Encryption and specifically kind of modern encryption and how it works
Now Before we jump straight into something like the Advanced Encryption standard. I wanted to talk about SP networks
or
substitution permutation networks because they are the basis for a lot of modern cryptography not all of it, but a lot of symmetrical