All right, so we talked about separable
Filters and the nice thing about separable filters is that they're incredibly quick
And this means that for example in computer games you could actually do some of this on the graphics card
At 60 frames a second, right? These are the kind of things that people who are writing computer games
Not me people who know what they're doing
well, I think if you the games are thinking about right, how can we do post processing like motion blur and
Bloom and other kinds of filters that look good in a game, but don't drive the whole thing to a halt
Separable filters are a great way of doing this
The problem is that Gaussian blur which is the thing
We kind of focused on
Doesn't look that good as a blur right if you say well I want to do some depth of field
I want to do some optical depth of field
All right
So I want the foreground person in my game to be nice and sharp and I want the background to be lovely blur, right?
It will look a bit odd. If you use a Gaussian because that's not what a lens does
You need to use a proper bokeh blur. I don't know how you pronounce that where bouquet is right bouquet right bokeh bouquet
Oh, yeah, I lost interest. It's because of the way that the optics of a camera work, right? So a
Gaussian your point is spread over a sort of Gaussian function. So it decreases like this, right?
That's not what happens in an optical blur my lens blur
So if we remember back a while ago, we did a light-filled video
You've got a point light source here, which is something in your scene a person a light. They've tree not important, right?
It's gonna be quite a few of these if they if your image has more than one pixel in it
You have a lens right which looks like this
it might have other glass optical elements and you have an imaging plane here like this and
What happens is point light don't just travel directly into your camera. Everything just emits light in lots of directions depending on its properties
so it comes in like this and when it's in focus
It will come and it will hit a point on here and that pixel will obtain that color
the color will be the sum of all of these light rays and then it sort of simplified to be usually between nought and
255 if you're out of focus
What will happen is these light rays will come pass the camera like this and focus on a point behind it
And actually you'll have this light will now be spread over this whole area
From the front if this is the front of our square sensor every point will now be a circle like this
and each other point will also be a circle and another circle in another circle and this gets you that kind of
Characteristic Lens Blur that you see when something like light sources christmas tree lights or something are out of focus
Cut to a nice shot of this so people spend a lot of money on lenses
Primarily, I think because of the quality of this kind of blur. They're not perfect circles all the time
Sometimes there's slightly odd all their hexagons because you've got an aperture and things like this
They give this blurry sort of unique characteristic even down to a lens which you can't really get on a computer
But we can have a go because we've got to do it at 60 frames a second
You haven't got the option of thinking about Zeiss and like although this is most obvious when you're looking at point light actually this applies
To any blur right? If you do a Gaussian blow of a scene, that doesn't have very bright bits
It will still look a bit different than if you have a normal bouquet blur
Right, and and actually you look much better. If you do it properly. Can we recreate our
Separable filters, but now perform a lens blur or at least an approximation to a lens blur
And the answer is not it's not so easy, but you can kind of have a go
You have to start looking at imaginary numbers a little bit
But if we just sort of not worry about it too much you essentially replace your separable
Convolutions with two complex separable convolutions and we can start to start to get something going
So one particular game engine that does this is EA's Frostbite engine which drives a lot of its sports games
So things like Madden will have a nice blur on certain shots that's created by an artificial lens blur
Not a Gaussian blur other games might kind of fake it a bit of a Gaussian
I don't know Primavera folks. It's not technically correct. There's a kind of short paper on this
There's a few blogs online that talk about stepper lens blur. There's some implementations out there
I've got my own implementation that I wrote which I'll release at the end of this video
There was a nice presentation at the Game Developers Conference in 2017 showing office technology, right? So, this is real-time
Bouquet blur not Gaussian blur right? So how does it work?
well
the idea is that what we want to do is instead of having our kernel which is sort of like this and a
Gaussian we need to have it be a circle right in an ideal world that right
Which from the top looks like a nice cut clear-cut circle now you can do that
But you can't separate it
you can have a
kernel
Which is jazz a load of zeros and then a circle of ones in the middle and that will produce you a very nice filter
But it will take ages because you have to do all this point wise and it's not separable
So what we need to do is find a function that we can split into a vertical pass and a horizontal pass
All right, let's look at example, right one of the nice things about Gaussian is but it's separable right? It's a naturally separable function. So
How can we convert a Gaussian to something that kind of approximates this circular shape?
And the answer is that we can sort of add in an imaginary component and create sort of complex Gaussian
Now obviously we're not gonna go into huge amount of detail on complex numbers
But they have an imaginary and a real component. The imaginary component is multiplied by I which is a square root of minus 1
the good news for us is that most of the time when we actually need to use it the eyes get multiplied out and they'd
a sort disappear or become a minus number
All right
That's all buddy time gonna go into actually it's a bit like when we talked about our discrete cosine
Transform and when maybe some time we'll talk about Fourier transforms
We have waves that add together to create something the shape we want in this case
The wave is going to be a Gaussian, right? But it's going to go up and down like this
So instead of our Gaussian going like this, which is not quite centered what we're going to roll on it
We have one that goes up and down right in the complex the complex
Component of this Gaussian is the bit that makes it do that
So we have a Gaussian that sort of goes down like this and sort of centers around here and it comes down like this
All right. It's symmetrical
And then we have another one but perhaps come sort of like this
and when you add them together you
get the thing we want so these kind of cancel out and you sort of get some of looks a bit like this with a
Bit of wobbling around and then it comes up about here and then kind of goes like that and then down like that
Right and that is our circle and that is also separable because it's essentially got a Gaussian as it's sort of primary function
So I've coded this up and I've been using it to mess around with different blurs
Most of the code is to do with creating the kernels
I do some normalization and things like this then I pass the image from each component and the very end
I combine them Mayon. I'll draw a little diagram to show how that works
the more components you use a bit like in
JPEG or any kind of sort of adding of waves the more ways you use the better approximation. You're going to get right?
So if you're not watching it for a game engine, you can use maybe two each time you do this
You've got quite a few convolutions you can have to do these are complex multiplication operations in addition
It takes time, even on a graphics card, right and bear in mind. There's other game things to do, right? The game isn't static
with just this nice blur
So you've got other stuff to be worrying about fact that there's an imaginary number involved in some sense to me
Anyway, personally, I like to look at it. It's basically not making any difference at all
The only difference is but the addition and multiplication that you would normally do in a convolution is now a complex addition and a complex
Multiplication which are sort of well defined structures that you can just apply and this is a mathematical term not just as saying it's different
It's absolutely a mathematical term right and it's actually not very complicated. You have your image, right?
We have our first component C naught here and we have another component so you want we could have lots of errs
this has a
Complex kernel so that we'll have a real and imaginary right but they're just going to look a bit like gaussians like I showed you
This is going to be the same over here. These are different ones
And the idea is that when we add these two together, we get a very nice disk
so what we're going to do is we're going to
converge with these and we're going to converse in which with these and we're going to add them together at the end because I see
A lot of addition goes on. All right
so we take our image we go through here and we get another image out, which is our
Real and another image out which is our imaginary we go through here we get an image out. That's real and image out
that's an imaginary and
Then we add these two together
in a weighted sum and then we take our fine or weighted images for each component and we add them together in our
Final output like that. The reason we have to do these separately is so that we can make it separable, right?
This is this can be done separately
This can be done in the same way very very quickly, right if you're doing it for just two components
There's a few convolutions you have to do add it all together and you have your result the SEPA bility means that instead of doing
all the kernel for every location
we do a vertical pass of a strip and we take that output and put it through a horizontal pass and
Then we combine the outputs at the end and that's that's it mathematically equivalent to doing the 2d version but much much faster
So let's have a look at some outputs right now
I've written some code here very straightforward that essentially produces these complex kernels in one dimension and kind of higher than 10 image
You know in the correct order add them up at the end so we can blur smooches, you know
I'll release the code so you can have a look the original post that led me to this work
Outlined a few parameters for these kernels that butt looks good, right they heat optimized so I put them in as well
So let's have a look a couple of these kernels and see what they are. So if I just look at the zero
The one component pass, right?
So this is if you want to try and create a disc with just one of these complex kernels, like what would it look like?
Well, let's have a look. These are the real and imaginary parts of this particular component
And this is going to look roughly like a disc when we add these two together. Let's see. So if I take the two together
We can run it and it's kind of a doughnut he disc right?
And the reason is because you can't get a perfect disc with just a real and imaginary part added together
It's not going to work. So what we can do is we can take another kernel add that to it
and now we've got kind of two wheels and two imaginaries in some loose sense add these all together and maybe we can start to
Approximate this disc so we'll get a little bit better if I change this
And run that together
All right. So now it's got a bit of a ripple to it
But this is starting to look quite a lot better. Like this looks to me quite a lot like the disc
I'm hoping for and this is still separable so I can take a strip from the middle with a strip here and
I can run it in two passes and save a huge amount of time
because this is a sort of a this is I think a
64 or 65 by 65 pixel kernel, this will be a big deal if you wanted to run this over an image
Especially, you know a sort of resolution of modern games run out so we can get even better we can add more and more components
So if I've ramped up the number of components like we get somebody looks really really quite good. There we go
so this now looks I think quite a lot like a disc so essentially
This is equivalent to running a circle of ones in a convolution over an image
It's just that now we can do it in these separable components and save ourselves a huge amount of time. It's not perfect, right?
The size is difficult to manage. The edge is not as sharp as I'd like this should be a sharp edge
But you can't get an absolutely sharp edge, but for a sort of game engine something in the background, it's gonna look great
And it's going to run really really quick in the EA and in the Frostbite engine
They'll only use two components so it won't be a perfect
Colonel but it will look pretty good stars are a classic example of some where we can use this blur
Because of course their point lights have a nice spread out nicely
So this is what happens if we Gaussian blur this image, it looks dull and light just like not high-resolution and doesn't look great
right, if we if we look at the lens version, I mean
I think it's aesthetically better, right?
When it looks like this, right each of these points has spread out into a nice disc
I've had to do some exposure here because it's that's difficult to do but I think it looks pretty good
So that's the Gaussian and that is the lens blur. I think that looks a lot better
so this is our fake lens blur by using our
Our circular disc kernel with lots of components because I wasn't worried about speed right?
Even if you did it with two components, we still look good
So if you were running this in a game
What you would do is you would start to separate out the foreground and the background of a scene you'd blur
Anything you wanted to not be in focus and keep anything
You did want to be in focus sharp, and then you'd start to bring them all together at the end by you know
Alpha blending or overlaying
so you sent me a picture of
Some Lego so we've got a lego man here on a table with some lights in a jar behind, right?
so this is quite a good example of something we could try this out on so you also
Thankfully gave me some foreground and background
Masks for this so I blurred the background and I blended in the foreground as you might do in a game
But love and encode it. I kind of hacked it together in paint, but I think it looks really good
So so this is our Gaussian attempt. So here I've essentially taken the man away
I've blurred at this using a Gaussian and I put the man back in basically and he looks okay
But if we flip to the lens blur, I think it looks a lot better, right?
It looks a lot more like a true sort of bouquet blur. You might expect from a lens
and so this is exactly what you'll see when you play FIFA and when you play
PGA Tour and when you play Madden and any of these sports titles when it cuts to a kind of nice depth of field shot
Let let's say means instant replay. This will be at work behind the scenes. There's loads of different tricks
It's just one of them. I think this is really interesting because apart from the fact that I think this is quite mathematically elegant, right?
Not everyone will agree. I
Just think it's cooler in these games and in you know
They're doing all these little tips and tricks to make it run at 60 frames
but still look really good and you don't notice like you know what no one really knows until you read up on it that
Madden has a different depth of field approach to some other games, but maybe you just liked that a little bit better
You didn't know and that's one of the things you liked about that game
And I just think that's really cool that you know
These things are going on behind the scenes so we don't know they exist but they just make it a little bit better
What you think right? So in this case, it's going through glass. The glass is moving things around that
Is going to be lost a little bit in our in our blur
Another thing is about our disk is not absolutely sharp. Right and we can't approximate hexagons with it
So you can't do that kind of blur
But I think the main problem is that the exposure is very hard to get