so today we're going to be looking at
different ways of doing autofocus with a
camera so when you're taking a photo
with any kind of camera really so a lot
of phones or definitely things like slrs
like this you might see an auto focus
mechanism so here we've got mf for
manual focus and af for autofocus so if
we got it set on auto focus
essentially the camera and these days
the computer inside the camera decides
whether a picture is is blurry or not
and corrects that by moving the lens so
we're going to look at a few of the
the methods you can do that
okay so let's start uh back in the day
so the first kinds of
focus assist i suppose you could call it
so back in the sort of round about the
1930s there was something called
rangefinder cameras came out these are
cameras that have a sort of extra
separate mechanism on the side of them
which was the rangefinder mechanism now
this was all done with optics and the
way it works is you basically see a
viewpoint of what you're looking at and
then you get a kind of like a ghost
image that's shifted to one side
normally what you have to do is change a
dial on the rangefinder mechanism to
bring the two images kind of lying on
top of each other so these days uh it's
all kind of joined into the imaging
system so generally speaking there's two
different ways you can do autofocus
there's active methods and there's
passive methods so an active method will
fire something out from the camera
and it will
use that to work out how far away
something is so
some of the most
some of the earliest mechanisms for
doing active
autofocus use something a bit like sonar
so it would send out a ping from the
camera
and it would basically time how long it
took to hit something and bounce back
just like the way sonar works
and it would use that to work out a
distance that you could then use to
focus it
so you can use sound you can also use
light so some active systems will fire
out
light from the camera what we're
actually going to concentrate on today
is the passive approaches how do we
actually
focus a camera
when we're not sending out light or
sound so we're just using
the light that's coming into the lens to
do the focusing one of the most popular
passive mechanisms of autofocus is
something called phase detection so you
might be aware of phase detection
systems on your camera you might have
different autofocus points different
areas of an image that you can choose to
focus on let's have a look at how phase
detection works in each one of those
regions with a phase detection system if
this is our lens coming into the camera
what we do is we essentially measure how
light behaves at different points on the
lens so if we have a ray coming from
here and we sort of follow it through so
i'm not going to draw all the
complicated optics here your light rays
that go through the lens fall upon an
autofocus sensor so this is really
essentially a sort of 1d strip of
photodiodes so set of pixels essentially
it's like a little image sensor that the
light hits here now the trick behind
phase detection
is you measure two paths of light
so you would actually have a second
sensor so this second light ray here
actually hits us a different sensor
imagine behind the scenes that this is
separated via optics but just to kind of
simplify things let's draw what's
happening here so what we get hitting
this set of pixels if you like is we get
to see one of the image features in the
image so if we've got a very simple
image what we might have here
is a little
peak so perhaps the edge of something at
one of those detection points
and because in this case the image is
well focused these two curves will
overlay if you imagine our two images
hitting our autofocus sensor and they're
perfectly aligned in the case where the
image isn't in focus what happens is
these beams go through
the pixels like this and they actually
focus just behind the pixels so what
that means is if we draw our curves we
have one peak that's kind of up here
like this and we have another peak
that's a little bit below like this
because we're not in focus so the nice
thing about
phase detection
is that what you do is you measure the
offset of these two peaks and the
distance between them tells you how out
of focus they are so let's just draw the
last case where so this one is kind of
focused past the sensor and the other
way that you can be focused is in front
of the sensor so we have light coming up
here
and perhaps doing something like that so
our focus point is here
so when it hits the autofocus sensors
they're going to be offset again so the
two sensors will give a reading of one
kind of curve up here
and one curve down here not drawn very
well
okay and then we get another distance
out what these sensors do so remember in
reality there's probably two of these
inside the system that the light's
hitting this is a very simple image that
we're making here in practice the two
curves might be quite complicated you
know they might be different features
that we see they're not just going to be
a straight peak most of the time it'll
be some kind of pattern of light that's
hitting these sensors and so the job of
the
phase detection mechanism is to work out
how to move one of these curves so that
it lines up with the other one so
mathematically you can do something
called cross correlation there which is
a way of essentially looking at how to
best match two signals that are offset
from each other and what that gives you
is a distance and it's that distance
that phase detection uses to drive the
lens so the nice thing about phase
detection is that once it's calculated
this distance is very fast to focus
because not only does it know it's out
of focus but it knows by how much so
once it's calculated this difference it
can say to the to the mechanism driving
the focus lens okay move this much in
this direction
so we've got a distance but we also know
whether we're focused too far away or
focus too close because if you notice
here the red peak here is above the
green peak when we're focused behind and
here the green peak is above the red
beak when we're focused in front by
knowing which way to shift
these patterns it knows how far to move
the lens and which direction so phase
detection tends to be one of the
quickest ways to focus a camera do most
systems use this kind of thing or do
most is so a lot of systems will use
both so a lot of slrs will use both
and the reason is when you're focusing
through the viewfinder it tends to use
phase detection because it's using the
optics of the
the lens system to steal a bit of the
light and pass it to these sort of pairs
of auto focus sensors so you get to one
pair for each autofocus
region
but you can only do that when you're
looking through the viewfinder if you
open the live view
so that changes how the optics in the
camera works
and so then it will tend to use a
process called contrast detection which
we'll look at now
now contrast detection does work on
light that's hitting the imaging sensor
so we're not using the optics in here to
divert light around to the autofocus
sensors this is just using the sensor
that is essentially used to capture the
final image what we're going to be doing
is
reading off some values of those pixels
that make up your image
and one of the things one of the
properties about focus
is that the contrast of the image so
sort of the differences between the the
bright bits and the dark bits get more
extreme
the more in focus you are so when you
have nice crisp focus you get nice clear
differences between black areas and
white areas
so what that means is if we have a way
of calculating those differences
so how kind of sharp our edges are and
our corners are and how
different our regions of light and dark
are
so we can measure our contrast we can
kind of work out how in focus we are
so if we just work through how we would
do that using a really simple example
we can look at some other kind of
gotchas that happen on the way and think
about why it's quite slow to do this as
well so we've got a photograph here
which i've just turned black and white
because it just makes the processing a
bit simpler so we're just using a tool
here called image j which allows us to
do some pretty simple scripting just to
get our pixel values and to blur the
images as well you can just download
this and try stuff out what's going on
here
is we can get the values of the pixels
and in order to work out how much
contrast we have in the image probably
the very simplest thing we can do is
just look at pairs of pixels and
calculate the difference between them in
terms of brightness so if we just go
through an image a pair of pixels at a
time and calculate the difference
between them
when we kind of maximize that the total
of all those differences
then we're in pretty good focus because
we've got the most contrast we can have
so that's what this simple example here
will do and this line here is just
calculating the difference between them
so i'm just calculating differences in
the x direction so in the row along here
sometimes and this is true with phase
detection as well your calculations of
contrast or phase can either be sort of
in the x direction along the rows or
along the columns or you can get sort of
cross sensors that do both in phase
detection in the contrast here we're
just going to do
neighboring x pixels okay
so we could calculate all the
neighboring y pixels as well but just to
keep it simple i'm doing this we could
use probably a better measure of
contrast so something like a sabell
operator
or something else that's good for
detecting edges
but just as a very simple example let's
just measure the difference between
neighboring pixels and see how that
changes as we go out of focus which i'll
simulate by blurring the image so if i
run this it will move over the image
and it takes a little while because i'm
moving across all the pixels
we get a number here which is
essentially the total of all those
differences between the pairs of pixels
so it doesn't really matter absolutely
what the number is
what we're going to do is try and find
the peak okay of these these values so
actually we're starting off in focus
here and we've got a value of about five
million let's make it a little bit out
of focus so if we apply a gaussian blur
so i don't know whether you can see
there but it's gone about our focus
there you can see we've lost our crisp
edges so if i run this again being a
little bit out of focus
it's taking a little while to go over it
so our first value was 5.1 million we've
now got a value of 1.2 million so we've
gone from 5 million down to 1 million
as the total of our differences so we've
gone about our focus and we've got a
lower
um contrast value if you like
so let's take it to the extreme case
terribly out of focus image
looks like my camera worked right no
comment so run it
and there we are so now we've gone from
1.2 million
down to 145 000. and if we take it to
the extreme the real extreme case we're
going to get very low values coming out
here
so what's happening
if we have an algorithm that does this
is that we can plot these values on a
curve so if this is our focus motor
driving the lens and this is our measure
of contrast which in this case is just
differences in pairs of pixels
what's going to happen is we're going to
get some kind of curve like this so when
we're out of focus either way the
value's going to drop down like it does
there when we're in focus
we're kind of at this peak point here so
the trick with contrast detection is
finding this peak with my camera
operator head on and looking at the shot
i'm looking at now i've got the laptop
quite close to me yeah i've got you in
the middle and i've got the blinds at
the back
and experience tells me that autofocus
will look at those blinds and go hey
they're nice they're going to look great
if they're sharp and you're going to be
blurry in your face yeah so that can be
a problem if you're running this over a
whole image you're going to get issues
so quite often
for example on on your phone you can you
can essentially select a region to focus
on so you can press a region on your
phone or you can select a a focus point
on live view or something like that what
that will do is it will only calculate
this difference across a particular
region of an image so if we just want to
focus on the library here we can just
calculate this over there otherwise
you're right it's going to end up kind
of optimizing this curve for
something in the image that you might
not care about the shiny stuff you just
shiny stuff
and of course the other thing to say i
suppose is we've gone through every
pixel in the image here but actually you
would probably only sort of sub sample
the image in order to make it quicker
you might have noticed it took a little
while to work these methods tend to be
quite slow they're not as slow as this
because they're not calculating every
single pixel you don't generally need to
but the catch with this method
is that you can tell
you're out of focus so when we were out
of focus at sort of 1.2 million on our
account sort of down here if this is 5
million where we were to start with and
then the first time we blurred it we
went down to 1.2 million we know we're
here but we don't know whether we're
there or there
so we don't know which way to move the
lens you'll notice when cameras use this
as a hunting mechanism so it has to move
the lens a little bit and work out
whether it's got better or worse
so it will tend to move it in quite big
jumps like this and as soon as it starts
getting worse
it will kind of hunt back
so you get these steps moving up the
curve to try and find the sort of
optimal focus point up here so you need
to search so that's one of the reasons
why
contrast detection is pretty slow
unlike phase detection where it says
move this much in this direction move
the lens this much in this direction to
focus
with contrast detection you don't get
that you just say i'm out of focus but i
don't know whether i'm too far away or
too close the other reason why this
method can fail
is if it doesn't have anything to
measure contrast on to start with so you
need some kind of texture so if you're
trying to focus on the sky region for
example up here you can imagine that
even you know the more and more i blur
the sky
it's not having that much effect on the
focus
that's pretty true of the phase
detection as well so if you've got no
edges visible it's very hard to do that
pattern matching to work out where you
need to move sometimes you you'll see
things like these charts which provide a
nice contrast between black and white
edges that use to assist
the camera focus focusing mechanism and
the nice thing about the calibration
charts like this so they have very good
contrast um bright areas and dark areas
that make focusing it nice and easy so
as an example let's try perhaps focusing
on this and taking some out focus images
and we can see how the contrast focusing
mechanism performs as we go in and out
of focus nice i'll put it on autofocus
let's see okay
so you might notice with your cameras
that both of these mechanisms will
fail to work very well when you haven't
got much texture so if you're pointing
just at the white wall there it's going
to struggle to find focus
and low light low light is a problem so
you often get an assist beam so some
cameras will use a flash to light up the
scene so they can see what it's doing
some will send out um
uh
they'll have a little extra bulb that
they light up the scene with um which
could be done in infrared so you can't
see it and then it could focus in
infrared
but yes you get that problem as well
some systems will even project out some
kind of structured light so like a grid
of light or a texture of light just to
help these algorithms focus a bit better
so one advantage of the of the active
methods is of course they'll focus in
complete darkness because if you're
using sonar it'll bounce back off the
wall whether it's lit up or not
um the disadvantage being it will bounce
back from a window as well so you can't
take photos through glass and things
like that so it's kind of swings and
roundabouts with all these different
mechanisms to kind of summarize the last
two methods that we've talked about the
phase detection is nice and quick but it
needs its own optics to work
the contrast detection works without
fancy optics and it works on a live view
where you can just see the image
but it's a bit slower because you have
to do this hunting approach
so
wouldn't it be nice if we could do some
kind of phase detection but on the
actual image sensor and so there's some
technologies coming along now so things
like dual pixel focusing
where what they've tried to do is
essentially bury these
autofocus points throughout the sensor
so i think it's canon that do this
approach i don't know if there's other
approaches available and the way it
works essentially each pixel is
comprised of two photodiodes so they
kind of work in pairs and each one of
the photodiodes
has some kind of micro lens attached to
it you've got optics going on but it's
spread out across the sensor and each
one of these pairs of photodiodes is
used to do
phase detection focusing so it works on
the on the back main image sensor the
same one that's used to capture the
image so you use the pair of
photodiodes hence dual pixel
to do the essentially it's like phase
different as phase detection focusing
but when you want to take the picture
both of the photodiodes will work
together
to act as a pixel to take the picture
so the nice thing about that is it's
still working on the back plane so when
you're looking at the lcd panel it's
still doing um phase detection so it's
it's nice and fast
this is called the aperture problem or
the barbershop pole illusion
because it's got stripes moving up and
down and the idea being that there's not
enough information here to
to accurate plus your row times 100 plus
your x times one and that will give you
the exact point in memory linear memory