- Hello.
Let me start with how we started this project.
So, as many of you already know,
gaze tracking is a big enabling technology for VR and AR.
And we have been hearing many interesting thoughts
about using gaze as a new input modality.
And by reading gaze, we can understand
what people think or feel,
and I don't think I have to mention
everything that, more applications
will be possible by having gaze tracking
working inside VR and AR.
It's also a very friendly environment
for doing gaze tracking.
So you just insert a small camera inside the headset,
and you can get this image.
And what gaze estimation does
is to associate this image with one gaze direction.
But this example here is very friendly example.
When you actually encounter in real application,
is more difficult.
So for example, the person can have long eyelashes drooping,
and you cannot occlude the view of pupil and iris,
which are important features for doing gaze tracking.
And, some people have small eye openings.
Just to be clear, this person is not being sleepy.
And, many people wear glasses,
and these creates very strong reflections
on glasses' surfaces,
and I often wear my glasses downward,
and then that can, the glasses frame
can actually occlude pupil and iris.
There's also in-built complexity in our eye.
For example, as your pupil constricts,
it moves to up and toward the nose.
And many gaze tracking algorithms base
on pupil location.
So, this pupil center shift is known to cause about,
up to five degrees of error in worst cases.
So if, maybe this is something, you're in that part,
can do well because we know that
it's accurate and robust for some kind
of image processing problems.
But we need data that's large
and that covers all the variations.
So we know that there are many existing datasets,
but we notice that we are very interested
in specific one configuration that's near-eye configuration,
using infrared light.
So, looks like we need to create datasets ourselves.
So these are what we did.
The first two are about datasets.
So, the first thing is,
rendered synthetic dataset,
that's anatomically accurate,
and we borrowed one model that's already existing
and modified it to suit for our use cases
that's near-eye configuration.
The second was capturing real dataset,
specifically for near-eye configuration.
And then the third is training on your own network,
to do gaze-related tasks.
So let me start with they synthetic dataset.
So our dataset is based on one excellent prior arts.
That's SynthesEyes, created by Wood and colleagues.
But we modified the,
geometry and material property of that model
to more accurately reflect properties
that are important for near-eye gazes dimension.
We rendered two million images in total,
and was in high resolution.
And each image comes with labels
of gaze direction and eye location with respect
to the camera, and pupil location in the image,
and pixel-accurate region labels.
So, let's take a close look at the changes.
so the top image is one sample image from SynthesEyes.
As you can see, it's rendered for visible light.
So, it has material properties
for red, green, and blue light.
But what's more commonly used
in near-eye gaze estimation is infrared light.
So we apply the changes for infrared light.
The second thing we noticed was,
the shape of the eyeball, which in pitch,
becomes important for the eye gaze estimation.
So here you might notice size of the iris,
with respect to the eye opening is slightly larger
than the image in the bottom.
So we adjusted eyeball size,
and also applied small changes to more closely
reflect the average shape of the eye, human eye.
Then the third thing was the animation
that those pupils enter shift,
as the pupil size changes.
So we know that as pupils change more,
it moves toward the nose and up,
and we permatized that move motion
so that you can modify it to,
if you want to include some individual variations.
Then the last thing is pixel-accurate region labels.
And notice that we also provide region labels
for the regions that are occluded by the skin too.
So, this is an example.
It blinks, and we can change the gaze direction.
And notice the glint shape and location
is correctly calculated based on physics low.
And you can also change the intensity
of each region, independently.
So in total, we rendered two million synthetic images,
using Blender.
And the render engine was Cycles.
It's a path tracer that calculates reflection and refraction
accurately, based on physics low.
But when we looked, compared this synthetic image
we rendered to the real images we captured,
there were differences in terms of image characteristics.
For example, these microscopic bumps,
you can see in the skin.
It's not really visible in the real images.
And also the detail from the iris is more blurred
in real images.
So this is where our region label can come handy.
So, you can apply various operations,
such as contrast scaling, blurring, and Gaussian noising.
You can do it either globally,
or you can do it separately for each region,
to create something that looks more similar
to real images.
So you first do pre-processing like this,
and then fit it to the training pipeline.
So now, lemme cover the real captured dataset.
So, our real dataset consists
of one million infrared images binocular.
And it's the highest resolution
among all the available, real datasets.
And each image comes with a label of gaze direction.
And it's worth to note that this is the first dataset
that involves acuity task,
that is to ensure accurate gaze direction control,
and also it reduces fixational eye movements,
which is unconscious eye generate,
that happens all the time.
And we also systematically controlled
the intensity of the scene the viewer is looking at,
to induce pupil size change.
So the pupil center's shift
is included in our dataset.
So one thing we wanted to have, but we couldn't have
in our dataset was
the pupil, iris sclera skin region label.
There's one excellent work that I want to point out here.
Just to be clear, this is not what we did.
I'm just pointing out because this is important.
Please check out OpenEDS for more information
about region labeling in real data.
So this is the setup we used to capture real data.
So you can see the part of the glasses' frame prototype.
And we placed infrared LED's for the places
that we normally do for near-eye gaze estimation.
And on the right figure, you can see the infrared camera,
looking up,
and there's two circular plates
that are infrared mirrors.
It reflects infrared light, so for the infrared cameras,
it's looking our eyes straight on.
But the viewer is looking at the screen directly.
So let's see what the viewer sees on the screen.
per recording of each gaze direction.
So first we start with wide screen.
This will constrict the viewer's pupil,
because the pupil shrinks in size
when seeing a bright light.
So we wait long enough to, until that pupil
become two to three millimeters in diameter.
And then we present one target,
at the desired gaze direction.
And the viewer has to report the orientation of this letter.
The letter was small, so the viewer
could correctly respond, only if the viewer
is directly looking at the target.
So if viewer gives wrong response,
we didn't start the recording.
Instead we presented another target.
And finally the viewer gets the correct answer.
Then, we started the recording.
And we varied the intensity of infrared LED
to include the variations of various
signal to noise ratios in the camera sensor.
And soon after we made a background screen black,
so, responding to this intensity change,
the pupil will start to grow.
So here's one video for one gaze direction.
And finally, let me talk about the neural network
we trained with by using this data.
So I'll talk about pupil localization in this talk.
But you can find other examples in our paper too.
Our network architecture was inspired
by this work done by Laine and colleagues.
It's on facial expression detection.
So the input to the network was a picture of person's face.
The output was the locations of 5000 vertices
in the face mesh.
So we thought this is a good candidate
for gaze estimation because the network
was good at accurately and robustly detect
the local position change in orientation,
orientation change of each part of the face.
So this is the specifics about our network.
It's seven layers of convolutional neural networks,
and each layer was followed by smaller but deeper
convolutional layer.
And after the seven convolutional layers,
there was one fully connected layer.
We evaluated this network performance
on PupilNet dataset.
And as you can see,
the position of the eye image and the size of the eye,
they are all different in each individual photo,
and it's sometimes overlaid with random reflection.
So we included these variations in our training data,
by augmenting our images.
So this is an example of our augmentation
done on one single image.
So we were translating, or applying to the rotation,
or overlaying random image on top of our image.
This shows the inference performance of our network.
Green dot is ground truth,
and the red dot is the inference results
through our network.
And for most cases, it's pretty robust and accurate,
for pupil location and finding the center of the pupil.
And you can also see it's robust
for the difference appearance of the eyes,
also illumination change of the capturing environment.
This plot compares performance
of various pupil localization algorithms,
and each color represents different
pupil localization algorithm,
and it shows distribution because PupilNet dataset
has many different scenarios.
So, it's a spread of detection rates
across different scenarios.
Orange represents our network.
And it's doing quite well, but it's not doing the best.
But I want to point out that this network
was very small, so the size of network
was about eight megabytes, compared to,
the two CBF 20 and CBF 15 required,
about 9.5 gigabyte and three gigabyte of memory.
And the network was also very fast too.
So when we measured speed of executing this network,
it was running below one millisecond
for desktop CPU and this time was including
the data transfer from CPU to GPU,
and then GPU inference and coming back to CPU.
It was also fast on mobile GPU too,
so when we ran it on Jetson TX2,
was running, finishing the operation,
in 3.781 millisecond.
As a conclusion, we are providing a rendered dataset,
but that's based on previous dataset.
And we modified it so that we can closely
reflect the actual anatomy of human eyes,
that are important for near-eye configuration.
And we share real dataset,
that's specifically captured in your eye configured,
geometry of configuration.
And we show one example in this talk,
that's light and accurate and robust for
detecting where the pupil is in the image.
And there are more examples in our paper too.
But there are,
some works to be done, and we couldn't cover
all the problems.
For example, we can model the rotation of the eyeball
and also dynamic deformation of the eye,
when we changes gages, case.
Or, you know, that's for future work.
Thank you for your attention,
and you can find more about our work in this link.
(audience applauds)
- [Moderator] Thank you.
So if you have any question, please raise your hand,
and one of the SP's will make sure
to bring the microphone.
- [Woman in Background] Now come here.
- [Rofer] Thank you, Rofer de Galle,
Human Media Lot.
What you seem to be doing is simply,
finding the pupil center.
Is that right?
- So, yeah. - [Rofer] The red dot
is just the pupil center.
- The example I showed in this talk,
- [Rofer] Yeah.
- was just finding pupil center.
- [Rofer] Yeah.
- We also did end-to-end stage tracking.
But I didn't mention it in this talk,
but you can find it in the paper.
- [Rofer] Yeah, because you know,
finding sort of a black circle in an image like that,
is fairly easy.
You don't necessarily need the neural network for that.
But to map the pupil center to an accurate location,
outside, in the outside world
is really the challenge, right?
- That's true, but in this examples,
we actually had, so yeah.
So finding pupil center in simple images,
is very simple,
but we have very many challenging cases too.
For example, when the eyelid is almost covering the pupil,
you will have only, like 1/3 of the circle,
and we have to find the pupil center there too.
And then you-- - [Rofer] And so did algorithm
perform on that kind of challenge?
- Yeah, so I actually have an example
in which I, you know, let me show that to you.
Just a moment.
I have a lot of slides - [Rofer] Mm, that's
a nice slide. - So let me load it.
I like that slide.
I also like your background.
- Thank you.
(Rofer laughs)
- Does that change with the day?
The time of day?
- No.
(Rofer laughs)
- Just a moment, sorry.
Yeah, so,
this is one example.
So you can see the red dots,
that those are still - [Rofer] Yeah, yeah, yeah.
- the inference points.
- [Rofer] Yeah.
- And these are very strong reflections.
- [Rofer] Right.
- So, you are right.
If it's not having these challenge variations,
then finding pupil center is very easy.
But we want to do that - [Rofer] So this is the real
even when-- contribution here?
- Sorry?
- [Rofer] This is the real contribution here.
- Yeah, I actually removed this slide this morning
because I was trying to be in 15 minutes.
But yeah, so, for example here,
it's finding pupil center.
It's inaccurate because it's very difficult case.
But still pretty close to the ground truth.
- [Moderator] Okay, one more quick question.
Thank you.
- [Woman in Audience] Yeah, senseful little talk.
It's really nice.
So, you know that there is always a saying
that the visual axis is not the same as the optical axis,
and how do you render that,
how that is handled in your rendering system?
And second question's also related.
Since we can't observe the real direction of the gaze,
how will just train the neural who train,
directly figure out where they are looking at,
like the real gaze direction and stuff,
like figuring out pupil center for example,
and what's the thinking there?
Thank you. - So, for the offset,
we just added one parameter that rotates eyeball,
with respect to the optical axis.
So we set it to five degrees in our model.
But if want to use that model,
you can change that value,
then it will reflect that change.
And for the second question, I guess you were asking
about end-to-end phase tracking.
So, when train, we just feed gaze direction
as the output and calculate loss function.
That's all we do.
But is definitely more challenging work,
and our performance was just slightly better
than the state of the arts.