I wanted to talk a little bit more about deep learning and some of a kind of slightly more,
Large and interesting architectures that have been coming along in the last couple of years, last few years.
So just a very brief recap, right? We've got videos on this
I'm going to draw my network from the top down this time. So rather than there being a square input image
I'm just going to draw a line which is the image from the top
So you can work with your animation magic and sort this all out for me. Brilliant.
So I'm going to be talking about deep learning and convolutional neural networks.
So a convolutional neural network is one where you have some input like an image.
You filter it using a convolution operation.
And then you repeat that process a number of times.
To learn something interesting about that image.
Some interesting features.
And then you make a classification decision based on it.
That is usually what you do, right?
So you might decide, well this have got a cat in it or this one's got a dog in it.
Or this one's got a cat and a dog in it and that's very exciting.
So from the top down right because I've always
My pens gonna run out of ink if I start trying to draw too many boxes.
You've got an input image, but it's quite large usually.
So here's an input image and I'm gonna draw it like this.
This is from the top.
So if this is my image, I'm gonna go to the top and look at it straight down.
Which I realized sort of like that. Does that work?
Now there's three input channels because of course we had usually red green and blue, right?
So in some sense, this is multi-dimensional. We're gonna have our little filtering so I'm going to draw a couple of kernels.
Let's maybe draw four. We're gonna do a convolution operation using this one on here
So it's going to look over all of these three channels
it's going to scan along and it's going to calculate some kind of features like an edge or something like this and that's going to
Produce another feature right and now there's four kernels of each gonna do this. So we're gonna have four outputs. Don't worry
I'm not going to do an 800 layer deep network this way
So each of these gets to look at all of the three something that's a bit a bit of a sort of quirk of deep
Learning but maybe isn't explained
Often enough, but actually these I'll have an extra dimension that lets them. Look at these
so the next layer along will look at all four of these ones and so on what we also then do and I'm going to
Sort of get why not? Why not use multiple colors?
We then sometimes also spatially down sample. So we take the maximum of a region of pixels.
So that we can make the whole thing smaller and fit it better on our graphics card.
We're gonna downsample this so it's gonna look like this and then okay, I'll just do a yellow one. Why not?
Can we see yellow on this? We'll soon find out. Yeah. Yeah
So let's say there's two kernels here and you can kind of see it.
I think we need to go pink here. Pink? Pink! Alright pink, forget yellow.
No yellow on white. That was what I was told when I first started using PowerPoint.
I like pink. Yeah, that kinda, that can work.
It kinda looks a bit like the red.
So that's going to look at all these four so and there's two of them. So there's going to be two outputs, right?
Just think of in terms of four inputs two outputs. So that's going to be sort of like this
I'm just going to go back to my blue and forget the colors now and you just repeat this process for quite a while
Right depending on the network. There are more advanced architectures like resinates, but let this become very very deep you
Know hundreds of layers sometimes but for the sake of argument
Let's just say it's into the dozens
usually so we're gonna down sample a bit more and so on and then we'll get some kind of
final feature vector
Hopefully a summary of everything that's in all these images sort of summarized for us
And that's where we do our classification
so we attach a little neural network to this here and that all connects to all of these and then this is our reading of
Whether it's a cat or not, that's the idea the problem with this is that these number of connections here are fixed
This is the big drawback of this kind of network
You're using this to do this very interesting feature calculation and then you've got this fixed number of it's always three here
There's always one here
So this always has to be the same size which means that this input also has to always be the same size. Let's say
256 pixels by 256 pixels, which is not actually very big
So what tends to happen is that?
We take our image that we were interested in and we shrink it to 256 by 256 and put that in you know
and so when we train our network
We make a decision early on as to what kind of appropriate size we should use now, of course, it doesn't really make any sense
Currently because we have lots of different kinds of sizes image, obviously
They can't be too big because we're run out of RAM
But it would be nice if we if it was a little bit flexible
The other issue is but this is actually taking our entire image and summarizing it in one value
So all spatial information is lost right?
you can see that the spatial information is getting lower and lower as we go through this network to the point where all we
Care about is if it's a cat not where is the cat? What if we wanted to find out where the cat was or?
Segment the cat tutor or somet in a person or count a number of people right to do that
This isn't gonna work because it always goes down to one. So that's kind of a yes or no is yeah
Yeah, yes or no. You could have multiple outputs
If it was yes, dog, no cat, you know different outputs
Sometimes instead of a classification you output an actual value like the amount of something
But in this case, that's not that's not worry about it now
You've told me that this is an amazing market so I'm gonna have a go at this
I said anyone ever raised your marker in your videos. I mean, this is a first that
Okay, it's work he's just gonna take quite a while because it stuck this rubber is tiny you know what I qualities Marcus
All right. There we go
All right. So the same input still produces this little feature vector
But now instead of a fixed size neural network on the end
We're just going to put another convolution of one pixel by one pixel
So it's just a tiny little filter
but it's just one by one and that's going to scan over here and produce an image of
Exactly the same size but this of course we'll be looking for all of these and working out in detail what the object is
So it will have much more information than these ones back here
So, you know
this could be outputting a heat map of where the cats are or where the dogs are or
You know the areas of disease in sort of a medical image or something like this
And so this is called a fully convolutional network because there are no longer any
Fully connected or fixed size layers in this network. So normal deep learning in some sense or at least up until so 2014-2015
Predominantly just put a little new network on the end of this. That was a fixed size now
We don't do that
And the nice thing is if we double the size of this input image, I mean we're using more RAM
But this is going to double little double and in the end
This will also double and we'll just get the exact same result which is bigger so we can now put in different size images
the way this actually works in practice is that when one your deep learning library like
Cafe 2 or pi stalks or tensorflow will allocate memory as required
So you put in an input image and it goes well, ok with that input image
We're going to need to allocate this much RAM to do all this and so the nice thing is that this can now have information
On where the objects are as well as what they are picks output. So
We'll show a few examples of semantic segmentation on the screen so you can see the kind of thing
We're talking about the obvious downside here, which is what I'm going to leave for
Another video is that this is very very small, you know
maybe this is only a few pixels by a few pixels or something like this or
You haven't done that much down sampling and so it's not a very deep network and you haven't learned a whole lot if you are
Looking for where is the carrier's image? You have kind of it's down in the bottom left. It would be very very general
So it would be you know bit sort of area. Maybe there's something else going on over here
It depends on the resolution of this image looks great with different colors in line. But what are you actually using this stuff?
Alright, so, I mean we have to extend this slightly, which I'm you know
Normally going to postpone for another video because this is too small for us to be practical, right?
What we could do is just up up sample this we could use linear or bilinear interpolation
to just make this way bigger like this and have a bigger output image and
It would still be very low resolution you'd get the rough idea of where something was but it wouldn't be great
Right, so you could use this to find
Objects that you're looking for. So for example in our lab, we're using this for things like analysis of plants
So where are the wheat is how many are there that can be useful in a field to try and work out
What the yield or disease problems are going to be you can do it for medical images where the tumors in this image
Segmenting x-ray images we're also doing it on human pose estimation and face
Estimation so you know, where is the face in this image? Where are the eyes?
What shape is the face this kind of thing so you can use this for a huge amount of things?
But we're going to need to extend it a little bit more to get the best out of it
And the extension we'll call an encoder decoder Network
Are you tying it up now? What are you doing? It's not neat enough this there's little bits of unwrapped out bits
Bear with me I start on the next video in a minute. Yeah
That's as good as it's getting it