the idea is to take a single photograph
of a face and as outputs we get a 3d
model in volumetric form where do you
start doing something like that this
kind of thing is becoming more common
now with cnns you can give them a
problem and just let them try and solve
it this is convolutional neural networks
right yeah that's right so
the idea is that
uh so as the image comes in we apply
some filters
um the values of these filters are
learnt through gradient descent and mike
has a video
talking about like that this is a
research project then yeah yeah so
of about eight months um and we decided
it would be quite fun if we release
uh kind of an online demonstration
mainly so people didn't have to run
uh the code on their own computer
because you have to install all sorts of
dependencies and you need a gpu and
yeah this is a much easier way you can
just upload a photograph
and as an output you get a 3d mesh which
you can rotate and play with
this is the object which you can
download uh and view in mesh lab so if i
also disable the colors this is the
actual the object without
without any texturing applies the actual
mesh which comes out of the network is
um slightly higher resolution than this
but because of demand we have to kind of
compress it a bit in general it seems to
capture things like the mouth and the
nose
yeah you can try and compare
yeah so it's running on one of the gpu
machines that we have in the in the
group and uh
i think at the moment it's processing
maybe 80 photographs per
well at a time maybe 3 000 per hour
this is
a photograph of your face
and we have
i'm gonna draw a magic box for now and
i'll try and fill it in in a bit and so
everything is coming this direction and
as output we produce a cube and this is
a 3d volume made up of much smaller
cubes and in each one of these cubes the
network progresses a value of zero or
one
and then we use an algorithm called
marching cubes which takes the surface
of all of these ones
and
we simply
send that back to the website and we use
the framework 3gs
to to render it so the magic box turns
the simple image somehow into this 3d
shape and then you basically color it in
with the pic with the photograph yeah so
the texturing at the moment is is
ridiculously uh simple so if you upload
a side profile it will actually take the
uh the texture from from the front of
your face on the back
and you might also notice some
distortion around the sides where it's
actually included the background of the
behind the face but the the the problem
was not to try and uh improve texturing
it was to try and increase improve the
quality of the 3d reconstruction it has
good good performance and have you taken
some of these 3d volumes and compared
them to actual 3d scans just to see how
close they are
uh or is it all been by eye we did a lot
of testing
and calculating of error which is
probably actually one of the most
challenging problems uh in in doing this
work
um
the ground tree that we have doesn't
have any details you know it doesn't
show wrinkles or spots
um but the fit is very good to the face
so it should you know try and match the
shape of the face i've got to ask the
magic question which is what's going on
in the box so inside uh we have an
architecture which actually looks like
an hourglass and it's called the stacked
hourglass so here we have the first part
of the hourglass and we have it up
sampling again and then we have a second
hourglass in here the image comes in the
spatial resolution as it passes through
the convolutional neural network is
getting smaller
um until we have something maybe
10 pixels by 10 pixels so not the
original resolution which is closest to
200 by 200. we then up sample it and
while we're up sampling it we actually
have smaller hourglass networks
which
are working at larger resolutions so
these come from an earlier point in the
network
so as this is being up sampled we're
combining the result from these smaller
cnns
this increases the
resolution around the side of the face
for example
so that you have more detail
otherwise you just get a blurry blurry
orb basically yeah so as you have these
larger cnn's inside these increase the
global context so you know that the eyes
are in the right place and that the
mouth is below the eyes for example this
then goes through another stacked
hourglass network which is identical so
it also has these smaller hourglass
networks and as output we just take the
3d volume so the hourglass there is
taking
a 2d image so these smaller but higher
resolution hourglasses at the sides are
they making the size how does it
extrapolate it
yes so it can actually be thought of as
a segmentation problem so there's a lot
of work on using convolutional networks
for segmentation so if you take a single
image it will it'll give you a mask of
pixels
and you give each pixel a number um
with it being like a person or a dog for
example instead of regressing a single
class
uh per channel as output so if you if
you're segmenting dogs and humans you'd
have two channels
in this case we regressed 200 channels
but they're all the same class they're
all the human face so anytime that you
have a set of zeros you know that it's
part of the face if you imagine a cube
drawn around my face because you know i
have to animate this now your face is
now inside a cube yeah
the slices inside a cube and if you chop
my face up into smaller slices you would
see
a set of wands any place that it is so
on the front you would just see a few
ones on the tip of my nose as you get
towards my ears you'll see
kind of you know just ears is around
an oval shape like i suppose it's
working out what features the 2d image
is of and then it's putting those in the
right place in 3d space is that right
exactly yeah so before we we process the
image we first move the face so it's in
the same sort of spatial
dimension as the the cube that we output
so the output of the 3d cube should be
spatially aligned perfectly with with
the face which is why when you see it on
the website
the the actual volume is perfectly
aligned and if you have a slight
rotation to your face
the the
the mesh will also be aligned with with
that one thing i noticed was hair was
hitting this what's going on there yeah
so the training site doesn't contain any
any hair it's it's just the 3d shape of
the face and when we were producing the
data set that we trained from we
actually didn't bother with the back of
the head it would be quite nice to have
have the back of the head as well the
problem is that since this is volumetric
problem we actually have to produce
these volumes which means checking the
voxels are inside the mesh or not
and that can be quite time consuming we
have some statistics which i've been
watching continuously in case something
crashes so in the back end we have six
queues and at the moment there are 67
images
waiting across those queues you can see
that we've had 181 000 photographs
uploaded
most of those have been since the 12th
of september and today i think is the
19th yeah
so just in the past minute we've had 70
which have been uploaded so that's why i
feel sorry or bad for the
technical services
at the moment there's so much traffic i
don't want to just switch it off and i
would like it to be available for other
researchers who want to try and look at
our work
we'll see what happens but i hope that
we can keep it running so
the reason why it doesn't work on the
online demonstration when you use a side
pose is because the face detector we're
using doesn't doesn't recognize these
faces
so instead of producing
horrible results we would rather just
make it look good we end up with a much
smaller image and lots of features going
all the way back so these are my
different convolutions of convolutions
of convolutions of convolution