- Thanks for the introduction.
I am Xucong Zhang from
the Max Planck Institute for Informatics.
My supervisor is Andreas Bulling
and this also is the joint work with Yusuke Sugano
from the Osaka University.
So, our work is about eye contact detection,
it included two cases.
The first case is object and eye contact
which means that the user looking at the object
and the second case is that human eye contact
which means that the second person
looking at the user.
So, if we can do this eye contact detection
we are able to apply this visual attention cue
for the different application
to estimate if the people looking at the object
or looking at the user, he or herself.
So...(mumbles)
The tracking including some interaction
with the public display,
attentive user interface,
egocentric setting or robotic interaction.
The most related work to ours is that
this work from the last 2013
where they first detected the face
and it didn't detect the eyes
they cropped the eye out to train
one eye contact detector for the camera.
So they are judging whether the other person
looking at the camera or not.
Because they treat the camera as the target object
instead of the object around the camera
so it's the main limitation for this work
and of course, you can also train
one specific eye contact detector
for one specific object with this method
but in that case you need human annotated data
to train the model.
Another intuitive solution will be directly used,
gaze estimation, to estimate the weather,
the person looking at the target.
Here I tried to to use a two state of art
appearance-based gaze estimation
which can be used with ordinary web camera
and this first one is our work from the CVPR 2015
which we apply the commotion on our network
for the appearance-based gaze estimation
most basically, we train the model on the
laboratory second data and then we test it
on the real-world setting data.
So that this model really can be used
for real world application
but the accuracy is about 14 degree.
The adversarial this year in our group
improved the accuracy to be 7.8 degree
but as you can see, the estimation error is pretty large
for this appearance gaze-based estimation
and also, you need the camera and camera-object calibration
to project the gaze direction
in a camera coordinate to the object.
In general, for this group the whole method,
you also need the personal calibration
to really improve the accuracy
which you will also want to avoid.
The goal for this work is that we want to propose
the eye contact detection method
that it can work with ordinary web camera
and we want to apply the unsupervised learning
so that there are no human annotated data.
We don't want any calibration
including personal calibration, camera calibration
or camera-object calibration.
The basic idea is that we assume that the target is
the salient object nearest to the camera,
so that we propose a method that you just need one camera
and put closely to the target object.
For the real-world everyday setting
there will be plenty different objects near from the user.
What do we need to do?
It just put one camera to the target object,
closely to the target object
and we collect the data through the camera
during the daily interaction between the user
under the different objects
and then we label those samples to be positive
and a negative with unsupervised clustering,
most basically how our method works.
With the camera attached to the target object
we collect a bunch of image from the camera
and then we perform face and facial landmark detection.
Now with the face as input
we perform the appearance gaze-based estimation
with one of our peers who work
and for this appearance based estimation architecture,
we output the gaze direction in the camera coordinate
and the we project this camera,
this gaze direction in the camera coordinate
to the camera plane with the tummy camera parameters
which is fixed parameters so that
we don't need the camera calibration at all.
In this gaze, the projector, the gaze point
on the camera plane is not accurate
but we...
The standard method for our method,
we just need to know the relative position.
So for each sample you are get to one gaze location
on the camera plane.
If you have a lot of multiple inputs
you will get a distribution over the gaze location.
Let's take a look of this gaze location
and you can already see that in this
gaze location distribution,
they are different in cluster
belong to the different objects.
Here is the result from the density-based clustering
and the reminder for this camera plane,
its origin is the camera position.
The assumption for our work is that the target
is the second object nearest to the camera.
We just need to pick up the cluster nearest
to the camera to be a positive cluster
and all the sample belong to this positive cluster
will be positive
and the sample belong to other cluster will be negative.
We introduced a so-called safe margin
as indicated here as the green circle here.
It's calculated by the standard deviation
of the positive samples
that all the sample...
All the side of the circle
will be directly labeled as negative.
So that we have more distributed the negative samples.
So far, we get the labels for the samples
as a positive and negative
and besides the gaze location,
we also extract another feature from the
appearance-based gaze estimation architecture
which from the face calibration earlier
and we call it face feature
which is the 4,096 dimension
and we trained the eye contact detector
with this face feature.
In summary, we used the gaze...
We use gaze location to do the, otherwise clustering,
to label the sampling and label all the samples
and then we use the face feature to train the detector.
During the test you just need to
pass through the input image
to the gaze special architecture to extract
the face feature and fiddle in to the eye contact detector
to predict if its eye contact on or off.
To evaluate our method, we click one to the side
under real-world setting.
It included two parts,
the first part is object amount in the data,
which is recorded with everyday working environment.
We have the target object projecting from user
and also corresponding cameras.
It was recording during the normal working day
and we collect the sample from the interaction with...
between the user and the different objects.
In total we have 14 participants,
each of them recorded three objects
which is tabulated...
Tablet, working display, and a clock.
Especially for the working display we have
three camera positions, which is above, below and next to.
The second and the third row are the
example over the data side and also detected a face
and second part of our data side
is called the head-mounted data.
This is recorded with everyday social interaction
and in this case, we let the participant to wear
a head-mounted camera and we asked them to go street
and interview three people.
So they were...
We really talked to different people.
Different people.
In total we have three participants
and we recorded 28 social interactions.
On the right is a sample from our data side
and also detected faces.
During the experiment we used MCC score for the agrimetric
and the 1.0 means a perfect prediction,
zero is a chance of random guess
and the minus one is completely wrong prediction.
For each evaluation we used
the first 75 percentage as training,
and the rest of the 25 percentage as test.
We also manually annotated 5,000 sample from the test set
to report the value
and there are no label for the training part.
For the object-mounted data we treat each recording session
individually and average the results.
And for the head-mounted data
which we did as one recording session.
We take the gaze locking as one of our base limit
so that we treat one of curious work.
We evaluate the user over the gaze lock feature p
and the face feature s.
So for our method, we use the gaze location
for the clustering and the face feature
for the detector training.
For the second method of, called the face clustering,
we used a face feature for post,
clustering and detect training.
For the second baseline, called the gaze classification,
we used the gaze location
for post-clustering and the detect training.
The next baseline is called the gaze projection
which is directly used gaze location p
as the visual attention,
so that it's judged whether, this close gaze location p
is inside the target object or not.
The last piece is called the head orientation projection.
It's used head orientation as a visual cue,
to judge whether this head orientation
is pointing to the target object or not.
Here is the main result from our paper,
we test all the method for the head-mounted data now,
for the object-mounted data and also head-mounted data.
So you can see our measure from far left,
performance best over other baselines.
The most basically, for the face clustering is to use
the face feature for clustering and detect training
we think is not good to use this face feature
to do the clustering because the face appearance
can be different even if the person is looking
at the same target due to the different height causes.
For the face cluster..
Classification, which use gaze location for clustering
and also, they had a training, we think it's not good
to use this gaze location to train the detector
because essentially, this gaze location is just
two-dimension feature, is not able to capture
natural variation on the face appearance.
For the gaze projection and the head orientation
they are suffering from the low estimation error
which is gaze's measurement
and also head orientation measurement.
They also suffer from the camera
and the camera object calibration
which we did manually for this two methods.
At last, the keystroke he also performed very badly
of experiment, it's not because that
this matter is not good, it's just due to the fact that
they have different definite concept over the eye contact.
Here is the example from a training side
over the gaze locking,
the first row are the positive sample
and the second row as negative samples.
The fourth row actually is
the person looking at the camera
and the second row is the just five degree
of the camera looking direction.
The second row actually...
could be positive (mumbles)
in real-world gaze especially
also in our evaluation data side.
The fourth row sets directly looking camera
actually should be negative
because the camera is not our target object.
Next, I want to show you the result
of our unsupervised clustering,
the fourth row as the recording setting,
the blue rectangle is the different attractive object.
The green rectangle is the target object
and the right one is the camera position.
The second row as the clustering result
where the green dot are the positive
and the blue dot negative.
For those samples I don't have the ground truth for him
but we compare those with the ground truth in a test aside
and we found is the quick correct clustering
and you can see in this clustering
that the corresponding to different object,
attractive object.
The last experiment I want to show you is
the performance across different objects
and also the camera position.
For the left, the figure is the performance across the...
For the tablet display and the clock.
Our method to perform the best for the tablet
because it's kind of attractive to the participant
and also it's a reasonable size.
We play the image and the video,
we download it from the internet
and attract the participant's attention.
The performance on the display is worse than tablet
because it has a much larger size.
The performance for the clock performed the worst
because people usually don't look at it.
Right figure shows performance across
different camera positions around the display
which is above, below and next to,
and you can see that our method of
performing robustly for those different camera positions.
We have last demo this evening,
and you are welcome to test our system by yourself.
Now please enjoy our result samples
and thank you for your attention.
(audience applause)
- [Host] Any questions?
For your method, I'll give you the honorable mention
for participating. - Thank you.
- So there's three because there's three authors
Congratulations. - Thank you.
- And there we have it, time for questions.
- [Audience Member] Just a quick one,
you have various clusters that arise
when you set up your objects
and correct me if I'm wrong,
the reason why you're putting a camera under each
of the objects that you want to look at is because
you have no other means of labeling those clusters,
is that right?
- Yes. Exactly. Yes.
- [Audience Member] Okay.
- [Host] Further questions?
I have one, so you chose to use an unsupervised method
and so the question is, when creating a data set
would have it been useful to use
a full-featured eye-tracker
to create a large real-world data set
in order to have sort of, a supervised methods afterwards
or do you think that the unsupervised method
is as good as it gets?
- Yeah, of course you can use some commercial eye-tracker
to do this eye contact detection or you just use the gaze
for the training, right?
Is that what you mean? - [Host] Right. Yeah.
- But the real-world setting that...
It's not...
There are some times the camera position is different,
if you train one model for the
specific object camera position you can not apply it
for other settings,
but for our method you just need to put the camera
to anywhere around the target
and we shall collect the sample
and after a few hours you can get
to the eye contact detector for that gaze.
- [Host] Do we have further questions?
And I think that this be a thank you
and all the speakers.