- Thanks for introduction and a good afternoon.
My name is Xucong Zhang.
Today, I'm going to present our work,
Evaluation of Appearance-Based Methods
and Implications for Gaze-Based Applications.
This is a joint work of Max Planck
Institute for Informatics, Germany,
and also Osaka University, Japan.
Please note that it's different with the last talk
that the gaze-based application here is a remote case,
not a head-mounted near eye gaze.
I believe some of you may know that human eye gaze
can be important for the interactive system.
We can use human eye gaze as explicit input.
For example, we can use eye gaze to type the words.
We can also use eye gaze to play the video game.
So, that you can shoot your enemy by just looking at them.
We can also use eye gaze implicitly
to model the user visual behavior.
For example, we can generate the attention map
over the user over the poster or social media.
And now assuming that you have a nice idea
for your next Chi paper,
and you want to use,
you want to estimate the user eye gaze in the project.
And then you will look at the recent Chi paper
to find out what people are using nowadays.
And you will find that most of the gaze based application,
actually using the commercial eye tracker
to retrieve the user eye gaze.
Let's take the Tobii product as an example.
You already have the high resolution camera
and also additionally, for its light source.
Those DYs, of course can provide you
word estimation accuracy something around the 0.05 or
to one degree guess estimation error.
However, first you need to buy such additional DYs.
Some of them can be very expensive.
More importantly,
most of those DYs have
limited operation distance.
That means that you cannot let a user to be too far
or too near to the camera.
Therefore, most of the gaze-based application
with a commercial eye tracker
actually limited themselves to a desktop setting.
If you read some paper from the computer vision community,
you will find another a hot research topic called
Appearance-Based Gaze Estimation.
This group method can accept the input from a single webcam
and process the data with the machine learning technique
and at last the output lucidity gaze direction.
Because this group method can work with a single webcam,
so that it can work not only the desktop setting
to estimate the human eye gaze,
but it can also work with longer distance.
For example, the public display setting
that you have web camera on top of the public display.
It can even work
with head-mounted DYs,
so that to estimate the second person gaze.
This kind of method can work in real-world setting
and auto environment.
Unfortunately, most current state-of-the-art
appearance-based gaze estimation heavily rely
on the calibration and motion techniques.
So that is not accessible or ready for HCI applications.
So, in this paper, we propose a software toolkit
called OpenGaze to bridge this to demand.
Our software toolkit is implementation of current
state-of-art appearance based method
and it can work with a single webcam,
written in C++,
can run on 13 fps with a modern GPU.
It is open source and also available on our website now.
Next, I'm going to breakdown the pipeline
of our OpenGaze software.
So, given an input image from the webcam,
single AGB image, we first do the face
and a facial landmark detection.
And then we crop out the face image
according to the detected facial landmark.
We feed that face image
into a gaze examination model
to out, to, I'm sorry.
To output the 3-D gaze direction.
And with notation of calibration information,
we can project this 3-D gaze direction
to a 2-D screen.
More specifically, for this face
and a facial landmark detection part,
we use the OpenFace 2.0 library.
This is open source library and also,
the state-of-the-art is only kind
for the facial landmark detection.
With the detected facial landmark,
we then perform the detail normalization.
This is a one hour periods of work.
Essentially to say input what your camera
at a fixed distance to the face center
and also rotate the watching camera
so that it can cancel some degree of the height of rotation.
We were able to see it
cancel the head rotation
around the room excess.
And also, it outputted a fixed sized face image
regardless of the distance
between the user and camera.
So that we can have the robot's fixed image
with the fixed size.
And then we fit this fixed image
into gaze estimation model.
This part is another one hour periods of work.
With full face image as input,
we first pass through it
with several convolution layers,
and then we pass it through the special width mechanism,
which efficiently uses the information
from the full face image.
And at last, it output the gaze direction
in the camera code in the system.
For more information, please reference to our previous work.
As I mentioned before,
at last we can project this 3-D case direction
to the 2-D screen.
Okay, now, we have our OpenGaze toolkit,
and next question will be how does this toolkit
compare to the current commercial eye tracker?
Because previously there are no such comparisons
between appearance-based method
and commercial eye tracker.
So that we have to crack our own data set.
We show a visual target on the screen
and then that participant, looking at the visual target
at the same time, we record the affixed image
with AutoShare of the webcam
and also we recorded the case estimate
is auto commercial eye tracker,
which is Tobii EyeX.
In total, we recorded 20 participants
at a different distance from the 180 cm to 30 cm.
For each distance, we recorded 80 samples.
For the experiment, we compared three tactile methods.
The first one is the model-based gaze estimation,
which is GazeML, published last year,
also the current state-of-the-art
model-based method.
The second is appearance-based gaze estimation,
which is one of our peers' work.
Also, it's the implementation inside the proposed
OpenGaze toolkit.
The last one is the commercial eye tracker,
which is the Tobii EyeX.
So, next I'm going to show you two experiments
inside our paper.
The first one will be guess estimation error
across at different distance.
On here, the x-axis is the distance
between the user and a camera
and the y-axis is guess estimation error in degree,
the lower, the better.
And please note, that, for each distance,
we used 60 samples for the personal calibration
and tested on the rest of the 20 samples.
First, I want to show you the performance
from the Tobii EyeX, which is the commercial eye tracker.
As you can see that the Tobii kind of achieved
very good performance
which around 0.5 to 1 degree guess estimation accuracy.
However, the operation distance limited
the unknown guess output for longer distance
or for near distance.
The model-based method account
outputted a case from the user at a different distance.
But when it come to the longer distance,
it will become more difficult to detect
the pupil and the iris
so that the guess estimation error
become much more worse.
In contrast, the appearance-based guess estimation method,
can output the robust performance
at a different distance.
Although the appearance-based guess estimation,
however, a gap with the Tobii EyeX
which is around a two degree guess estimation accuracy.
Previous to that for this experiment, we used a 60 sample
for the personal calibration
and really you cannot collect so many samples
for the personal calibration.
So that our next experiment is about
how many samples we need
for the personal calibration.
Here, for this figure, the x-axis
is the number of calibration sample we use,
and the y-axis still is still the guess estimation error
in degrees, the lower the better.
We pick the distance of 75 cm
because this should be good for the Tobii EyeX.
As you can see from the figure,
if we just had one calibrated sample,
it will cause,
it will cause a much worse performance.
That's mainly because it can cause the overfitting
for all of those models.
But if we have around a four or five samples,
then we can make the performance to be a reasonable level.
Again, the Tobii EyeX achieved the better performance,
and the appearance-based method
kind of achieved the comparable result.
Okay.
So, in summary, that our OpenGaze toolkit
kind of enabled a different new form
of gaze-based application
because it can work with a single webcam,
has a large operation at distances between users and camera.
And last but not least,
also in the edition previously,
this toolkit can work with multiple users.
And in contrast, the commercial eye tracker
can just work with a single user.
And that's where we end my talk.
Thanks for your attention, and please enjoy our demo.
(audience applauds)
- Okay, any questions?
- [Participant] Look, you wanted us to enjoy "Friends" there
playing Bamboozled, right?
Or something like that.
- [Participant] So, but the real problem is
to actually match it to where people are looking, right?
- Yeah, yeah.
- [Participant] So, do you have any thoughts on that?
- You mean the 2-D screen part?
- [Participant] Well, it doesn't matter really.
It could be anywhere in the world, right?
- Well, the output of the toolkit,
a row of the code actually
is 3-D gaze direction camera code in the system.
So, if the tablet is inside calibrated,
inside the camera coding system, then you can get
the output to which target you're looking at.
- [Participant] Okay. Thank you.
- [Participant] I had a technical question,
a bit on slide 14, I noticed something
with the appearance-based method.
It looks like
the arrow is actually going down after one meter.
- Yes.
You mean these ones?
- [Participant] I mean, it's not that significant,
but I find it surprising.
Do you have any insights on that?
- Yes, so please note that for this experiment,
we used 60 samples for calibration at each distance.
That means, for each distance, they have one model.
We train on that distance and test on that distance.
So that has plenty of model.
And another thing is if the user is very far away
from the display,
then the guess range also it shrink.
The total range is a little smaller, so that the aggregate
cannot be that large.
- [Participant] Thank you, very much.
Bryan Hull, University of Michigan.
So I was wondering, between, so one of the,
I think this may go to the first question,
that each eye has a tendency to look differently.
And I'm wondering how you handle,
do you average them together?
Or do you basically attempt to find the intersection point
where the two gazes go together?
- Yeah.
That's a good question, thanks.
So, usually,
the guess estimation method is
inputted single eye,
and then you have the case of direction for each eye,
and then you would intersect them.
But for our previous work,
and this work,
from CVPR Workshop in 2017,
we take the full face as input.
And then we give one gaze direction
for that full face.
- [Moderator] We have a bit more time,
so if you have any,
yes, keep asking.
- [Participant] So do I understand, so yes,
when you showed those graphs with the errors,
the angled errors,
so those are errors off of a position on the screen,
a known position on the screen?
- No, it's a guess estimation error in degree.
- [Participant] Yeah, but relative to what?
- That depends on the distance.
So, two degree, 50 meter, one meter,
on the screen, it's different.
So that we use degree.
- [Participant] No, no, that part, I get.
But, it's, you're comparing it to a known location
on the screen to know what the error is, right?
- No.
We just compared it with the 3-D gaze direction.
We did not compare on the screen coding it.
- [Participant] Okay.
- That depend on the distance.
- [Participant] Okay, yeah, I understand, okay.
Thank you.
- [Participant] I'm curious to know if you have
thought about
if it's possible this too for cameras for a mobile device
and what is the implication of that?
- For the single webcam application?
- [Participant] No, for cameras from a web,
or mobile device.
Smart phones in this case.
- Currently, this toolkit requires GPU to write,
but I know that there are some technique for comprise
the network that you can actually run
on a mobile phone device,
a simple mobile phone device.
And if you can have that
and then you,
I expect that you should have some calibration
on a cell phone because it's too small.
And then you can actually sync, you can distinguish
different regions which part you look at with our toolkit.
Let's say it's possible.
- [Moderator] Okay, I think we can switch gears.
Thank you, Xucong.
Let's thank-- (audience applauds)