(inaudible)
- There we go. Much better now.
Okay, thanks for the introduction
and good morning everybody.
Today I will present our proof of concept approach
on estimating workload to pupil diameter
under different lighting conditions.
The work I'm presenting has been conducted in collaboration
with Drea, Albrecht and Andrew.
So, estimating the user's workload is helpful
in many situations where people actually interact
with computing devices or machines.
This is especially the case if you have situations
or if you expect situations where high workload can happen
which you might want to prevent.
So as we can see in this adapted version
of the Yerkes-Dodson Law,
you see that both overload and underload
can actually decrease operator performance.
So high workload,
if we have a look at the car as one example
can happen if you drive on the road, pick up a phone call
and get very involved in the phone call.
This might actually induce a high workload
and decrease driving performance in this case.
Similarly, taking the car again as an example,
underload can happen if you activate all
your assistive systems and don't need to watch
that much what's going on at the road.
Since we, of course, want to keep the driver
in this optimal range with regard to workload,
the goal is to adapt the interface
or the content that is presented in a way
so that we get back to this ideal range.
Understanding workload and adapting to workload
is of interest for multiple domains
and I just picked a few examples over here.
So, as already mentioned, the car is one interesting domain
and we can imagine taking this example
with a phone call again, in the future,
the vehicle might be able to understand
the driver's workload and when you pick up a phone call,
your workload gets too high,
the vehicle might actually take over control
so that you can dedicate all of your attention
to the phone call.
Similarly, we see that in office situations
or also in control rooms,
such as with air traffic controllers
it is of importance to adapt the activities
or the tasks such that you get
or receive ideal operator performance.
Another domain where this is of interest
is the domain of e-learning where you want to make sure
that by adapting the content,
you achieve an ideal learning performance and experience.
Understanding and estimating workload has been done
in research already before.
We see many projects and most of them
employ physiological measures to actually estimate workload.
So, examples are measuring heart rate,
heart rate for reliability,
having a look at the skin conductance level or EEG.
Another method to actually estimate workload
is based on analyzing eye behavior
so one example is the analysis of blink frequencies.
Another method that is based on eye tracking uses
the so-called task-evoked pupillary response
and that's what we used in our experiment as well.
So what is known from previous research
is that pupil diameter actually increases
with the cognitive difficulty of a task.
The changes for pupil diameter are in range
between point one and point five millimeters.
However, there's actually one big challenge
with analyzing just the pupil diameter of a participant.
So what you see in this animation
is what we call the so-called pupillary light reflex.
So the pupil diameter actually decreases
the more light reaches your eye
and this is up to five millimeters of change.
This reaction actually overwhelms task-evoked response
and, therefore, is quite tricky to actually to understand
and distinguish different levels of workload
if illumination is changing.
That's the reason why part of the research
basically evaluated and investigated
task-evoked pupillary response in situations where
the illumination was constant.
But, of course, this is not really applicable
to real world situations
such as when you look at different screen content
or if illumination changes because
then you will also like to use pupil diameter
to actually understand the user's workload.
So the goal, the overall goal, that we tried to achieve
or at least some part of it is to estimate mental workload
by measuring pupil diameter independent of illumination.
So our approach is actually that we analyzed
the combined influence of mental workload
and illumination on pupil diameter
and we did so on a different but fixed levels
of illumination and task difficulty.
So let's have a look at the experiment
that we conducted at the University of Stuttgart.
We conducted a within-subject experiment
with 24 participants with a mean age of 25 years
and unfortunately were to exclude four participants
post-hoc due to technical issues.
The experiments took place in window-less room
with white walls or a computer screen in front of the user.
During the experiment, the participants had to look
at this crosshair that was shown in front of them
and we had two independent variables
which are the level of illumination
as well as level of task difficulty.
We measured pupil diameter using some eye tracker
at 120 hertz and also for the mental task
that the participants had to conduct,
we measured task performance by looking
at the percentage of correctly produced answers.
For subjective feedback, we additionally had to look
at the NASA Task Load Index.
So let's have a closer look at the illumination levels.
On this light, you see that it's basically two types
of illumination levels.
One was related to the aim at illumination in general
so we had one lamp, two lamps
or three lamps illuminating the room
and, in addition to that,
we also had three different conditions
where people had to look at a screen where images
with different gray levels were shown to the participants
and the participants either started
with these three conditions first
or with those conditions and then was counterbalanced.
To induce some kind of mental workload,
we used the so-called auditory delayed digit recall task,
or better known as n-back task.
We actually used it with three levels of difficulty
and the task works as follows:
Over a period of 30 seconds,
a sequence of 20 digits is read out to the participant
and whenever the participant hears a digit,
he or she has to repeat the digit that she heard
n steps back.
So, for instance, if we have a look at the two-back task,
the experiment starts, for instance,
with the system saying four and eight
and when the participant hears the digit six,
he or she should reply with four which is the digit
that was mentioned two steps before.
So each condition took 30 seconds to complete.
Well, with that, that's have a look at the results
that we found during our experiment.
We conducted one way anova and additional post-hoc tests
and what we found is that
there was significant differences
between all pairwise comparisons except for
the zero and one back conditions.
We assume that this was the reason
that the cognitive resources were still sufficient
for both situations but since we look at the workload,
task performance was only one specific aspect.
We did the same to analyze the subjective workload
and what we saw is that also
the subjective workload significantly differed
between our task levels and, of course,
the workload increased with the higher number
of the n-back task.
Okay, so let's have a closer look
at the actual pupil diameter
that we measured during the experiment.
What we did throughout the experiment
is that we calculated the average pupil diameter
for each participant and condition.
So what you see in this diagram,
every dot basically represents one
of these average measurements for one participant
during these conditions.
On this diagram, you see a separation between
the zero, one and two back task.
So let's have a closer look at the data.
If we only have a look at the situations
with the different lamps,
what we see over here are, in each block on the left side,
you see the darkest condition
and on the right side you see the brightest condition.
So what we see clearly in this diagram
is that pupil diameter actually decreases
the more light reaches the eye which is in line
with the prior research.
If we have a look at the screen conditions,
we basically see the same behavior
which was, in a way, expected as well
but what you also see is that this,
the illumination was slightly different
and, therefore, the pupil diameter differed
between conditions for the screen and for just illumination.
So if we now have a look at one specific condition which is,
in this case, the two lamp condition we also see
that pupil diameter slightly increases,
the more difficult the task gets.
So this is what is known from research
which is the task-evoked pupillary response.
However, it is quite tricky
to actually understand workload by just looking at
the participants' pupil diameter.
So what we see over here is that we get almost
the same pupil diameter for one illumination condition
and the zero-back task and for another condition
with the one back task but a different type of illumination.
So this means that it's rather impossible
to just infer workloads
during different illumination conditions
by looking at pupil diameter.
So to still be able to actually distinguish different levels
of workload for different situations of illumination,
we came up with a simple model where we actually composed
the resulting pupil diameter
as a sum of two different components.
So for one specific person, the resulting pupil diameter
is actually composed by the average pupil diameter
that this person was producing during
this lighting condition across
all different task difficulties.
The remaining component is then what we call
the normalized task-evoked pupil diameter
for this specific task difficulty.
Okay, so if we now plot this normalized pupil diameter
the diagram, it looks like this on this slide.
So what we see now is that the pupil diameter,
or the normalized pupil diameter,
now increases with the different task levels.
So it seems that it's now better or more easy,
possible to actually distinguish
between different task levels.
What you already see in this diagram
is like a classification that we performed
to distinguish between the different task levels based
on the normalized pupil diameter.
To do so, we actually had to look at the one-back condition
and usage heuristic approach to define limits,
offering lower limits to distinguish
between zero, one and two-back tasks.
So basically, we compared the values against the lower
and upper limits for the one-back-region
and the heuristic approach to do so was basically
that we calculated the mean value
of the normalized pupil diameter for the one-back situation
and then we added or subtracted one standard deviation
to define those limits.
To evaluate this model,
we performed a k-fold cross validation
and we used three quarters of the data
to actually define thresholds.
With the rest of the data, we performed the test
and in this case for k equals four,
we were able to actually correctly classify 75 percent
of all cases using this model.
This is, of course, just a very first approach,
a simple model that we employed
to actually distinguish workload based on
pupil diameter and illumination.
So we know that there are certainly limitations
and one of the most important being actually
that we used just discrete levels
for illumination and task difficulties.
Of course, if we now have a look at real world situations
you have changing illumination conditions
or like continuously changing illumination conditions
such as when you see the sun outside
or when it gets dark outside.
At the same time, what we did in our experiment
is that we averaged the pupil diameter across all
the conditions or for each condition.
So basically we just had one measurement in 30 seconds.
We did so basically to reduce noise, but in the future,
it might be helpful to actually have a look
at shorter time frames.
In addition to that, we calculated the workload,
or estimated the workload as a post-hoc estimation,
instead of doing so in the real time
which you want to do if you use this in a real system.
So if we now have a look at future work,
and if we imagine we want to use such technology
in the real world, we would need to do the following:
First of all, we might need some eye tracking technology
to actually measure pupil diameter on a permanent basis.
Then we would also need to measure the illumination
that actually reaches the participant's eye
or at least understand which illumination level is produced
by the content that is shown at that position of the screen
while the user looks at it.
In addition, we might need to know
the average pupil diameter of this participant
for current illumination and/or the current workload.
So this would mean that we probably need some kind
of calibration and there are some approaches
how we could actually do so.
In addition to that as already mentioned,
it would be of interest to actually come up
with advanced models that allow
then a final real time measurement
and also to include the continuous changes
with regard to workload and pupil diameter.
With that, I'd like to summarize my talk
and what we presented from our project.
So the idea that we followed was
to estimate mental workload by employing pupil diameter.
With the model that we were developing
and also the data that we collected,
we actually were able to account for illumination
and the model, as a first approach, actually incorporate
both the influence of workload and illumination
on pupil diameter.
With that, I am at the end of my talk
and I'm happy to take your questions.
(audience applause)
- [Audience Member] Hi. Lovely work.
Roel Vertegaal, Queen's University.
I'm actually kind of surprised nobody's done this before.
I'm actually flabbergasted.
There must be other work somewhere in psychology
or something but my question is,
how can we make this practical?
Do you have any thoughts on that?
So, I mean, if you're in an eye tracking situation,
could be either static but I prefer a roaming situation,
or a roving situation, what are your thoughts on,
for example, measuring light conditions?
Like, one approach would be to
if you have a wearable eye tracker
would be relatively straightforward
but if I had a photodiode on there
that sort of like would be placed close enough to the pupil
to see what the light, incoming light is
you could also with a desk mounted eye tracker,
do the opposite and sort of like look
at the light illumination of the face from the other side.
Do you have any thoughts on this?
- Yeah, so you've already provided some
of the answer actually because like
measuring illumination is something which is already done
in many devices today, so if you carry a smartphone with you
it normally adapts to the screen brightness by just sensing
the illumination which is around you
and basically then adapts the brightness of your screen.
So, this could be at least an estimation
of how the overall illumination is like
and on the other hand,
I think one future step that we actually need to do is
basically to see how much pupil diameter changes
with regard to illumination.
If we just have a look at the for real part
that you have a look at on the screen,
so basically just pretty much the area around where
you look at or if it makes a bigger difference
if you actually also change the screen content around it
because we have the assumption that it's really just
the small area around the gaze point where you look at
which is actually important for this fact.
So, if this would be the case, then you could,
of course, employ the eye tracker
to see where the user looks at,
see how the color distribution is around this point
and then take this as additional inputs
to the system, for instance.
- [Roel] Yeah, so I very much appreciate
the empirical approach and doing follow up studies
in terms of the more details
but please build the system, I'd love to see it.
- Yeah, so I guess, that will be the future steps
but I think there's still quite some work to do with it.
- [Moderator] We have time for probably one more question
but I'm sure the speakers will be around after for more.
- [Audience Member] (Inaudible) Texas State University.
Thank you for an interesting talk.
So I have a very short question.
So what eye tracker did you use
and how did you measure the pupil so you had an output
in millimeters from your eye tracking device
or you had an output in bigger units
and then converted to the middle?
- So we used a SMI Red 250 at 120 hertz
and the measurement in millimeters was generated
by the SMI (inaudible)
so I think they first produce it in pixels
and then they automatically convert it to millimeters.
Thanks.
- [Moderator] One more quick question while we --
- Okay, should we swap screens already?
- [Audience Member] I'll make it very quick.
We might get a lot less variance
if you look at relative pupil diameter...