Hi this is the talk on the Low/High
Index of Pupillary Activity for CHI 2020.
It's work by myself and my friends
Krzysztof Krejtz, Nina Gehrer
Tanya Bafna, Per Baekgaard. Krzysztof is in
Poland, Nina is in Germany, Tanya and Per
are in Denmark. We have funding from the
US National Science Foundation for my
part, Bevica Fonden for Denmark and
Polskie Ministerstwo Nauki i
Szkolnictwa Wyższego in Poland.
Alright so the paper is on an eye
tracking measurement of cognitive load
using the Wavelet Transform and looking
at the transient and phasic...the phasic
and tonic signals of the pupil diameter
oscillation. So here we've got some
applications where this could be used
automotive learning domains: it's a paper
2016 at CHI also Peysakhovich did a
PhD dissertation using the low frequency
high frequency ratio of pupil diameter
in his study and some of our algorithmic
developments based on that work that was
in Toulouse, France in 2016, and also
another example is Fridman et al 2018
also a CHI paper of cognitive load
during driving, among other examples
which you can look in the paper for more.
I will skip most of this just to
mention the Cognitive Load Theory
originated by Sweller in 1988 which
highlighted the use and reliance on
short-term memory and which is why often
times cognitive load is measured using
task difficulty, tasks that use some
aspect of memory such as n-back
Oftentimes
eye tracking papers also mention Hess
and Polt 1964 referring back to that work
on pupil diameter, however, it is very
important to mention that in that work
they used a specialized pupillometer,
an Arriflex 16-millimeter which measured
at two frames per second which is very
different from today's eye trackers
which measure at a higher rate and are
usually much farther away from the pupil
and the eye is usually allowed to move.
There is also something called the Task-
Evoked Pupillary Dilation or Pupillary
Response, TEPR, that is often
mentioned and it's important to note
here as per the graph in the original
paper that pupil diameter is often
measured in relation to a baseline
measurement. So usually what happens is
there's a task where there's nothing
happening and that's the baseline, and
then a task of increased difficulty is
given, pupil diameter is measured there,
averaged, and then the difference is
taken between that measurement and the
average during baseline. Peysakhovich
in his study linked pupil diameter to
the LC-NE system where he basically said
that the pupil diameter in response to
task difficulty is the phasic response
whereas the baseline measurement is
linked to the tonic response, and one
thing he also mentioned is that what if
you take the measurement of the pupil
under low-light conditions? Well in that
case the pupil diameter is already
dilated and there's a danger there of
missing the response to cognitive load
with pupil dilation as the expected
response. So it's important to note that
for future studies.
So Peysakhovich did a basic study on a Toulouse
n-back task I won't spend too much time
on this because there are time
constraints for this talk, the point
being though is that his ratio showed an
inverse relationship to cognitive load,
in other words, the LF/HF ratio which
should be lower with an increased load.
Tha's important to remember for later, and
his response showed distinction
between task difficulty. One
thing to note about his task though
is, that, it was taken...the signal was taken
over the entire duration of the of the
measurement whereas what we do and the
pupil diameter and the Wavelet
Transform can be measured
potentially in much shorter increments
perhaps even in real-time. Alright so a
lot of our work is based on our 2018
paper at CHI where we introduce the IPA
which was the Index of Pupillary Activity
based on Sandra Marshall's Index of
Cognitive Activity, something that she
patented but in her version of the ICA
there is only the wavelet coefficient
decomposition mentioned and she does
not mention the scaling function and,
because it is patendted, details of her
implementation are difficult to find
outside of the patent itself. So in 2018 we
discussed how that works and we
presented our own version of the ICA
called the IPA and so here similar to
Sandra Marshall's, we have the wavelet
decomposition, the psi function, but we also
talked about of the scaling function and
both of them are used in combination to
give you the wavelet transform which is
important to note is a multiresolution
analysis of the signal. The j superscript
there indicates which level of
decomposition you want to look for the
changes in the signal. So here is the
discrete version and here is the wavelet
filtering essentially g there is the
convolution of the filter you see in the
table at
left if we look down across the
coefficients to see one and minus one
and so it's a difference between
successive pairs of numbers
meanwhile the scaling function uses the
h filter which is an averaging filter
you see the coefficients there one and
one divided by root two both of them so
you have a sum, it's a sum of products so
you take the first value is multiplied
by one over root two to the second value is one
over root two, add them together and divide and you
get an average signal. Other filters can
be used of course, Daubechies-4 is a
popular one; we ended up using the
symlet-16 wavelet---the longer the
filter is the smoother the effect is of
the scaling function. Here's an example
what you get in 2D and so you have those
same filters but applied over the rows
in which case you get a difference and
you get the vertical edges as you see in
quadrant two in the image at right or if
you take the filters and apply them per
column you will get horizontal edges as
you see in quadrant I believe it would
be one two three third quadrant bottom
left. Meanwhile the scaling function
gives you a subsampled version of the
signal which is at the upper left
quadrant three and if you now take the
next level of the resolution then we
have the next level coefficients and so
you can see that what we're getting is
high frequency content at the original
base of the pyramid and then we get
lower frequency edges as we go up the
pyramid in the decomposition. In 1D the
idea is similar except of course it applies
to the 1d signal which is essentially
what the pupil diameter is, it's just a line.
So in practice we use a symlet-16
wavelet to get a slightly smoother
signal and what we did beyond Sandra
Marshall's ICA is look at the edges or
the changes in moment-to-moment
oscillation of pupil diameter, and
that is accomplished by looking at the
neighbors to the particular value that
we're interested in and if those
neighbors are greater than or equal or
sorry the central value is greater than
or equal to the neighbors and
it is strictly greater than one of them
then we have an edge in the signal.
That's what we find to threshold the
moment-to-moment oscillation of pupil diameter.
The 2018 paper had the code for this which
is still readily available of course in
2020 we updated that with the low
frequency/high frequency ratio and
essentially what it is is just taking
the low frequency component, dividing it
by the high frequency component so the
high frequency components at the bottom
of the pyramid and you take something
like in the middle as the low frequency
component and then obtain the LF/HF
ratio from within the wavelet
decomposition which is I think new; I
haven't seen anybody else do this. As I
said Peysakhovich did this over the entire
course of the signal here we're doing it
within the wavelet domain. That's our
key technical innovation of the paper.
Alright, to validate we applied the LHIPA
as it's called, the LHIPA to the same
study that we did in 2018 which is a
replication of Siegenthaler et al. and
there was number counting, so you either
count backward by 17---the difficult tasks---
or count forward by 2---the easy task.
You can see the paper for the
results that we obtained two years ago,
the study was shown to be indeed
indicative of the difficulty of the task
so accuracy of responses were lower at
high difficulty and it was also
subjectively found to be annoying. You can
refer to that paper to get more details.
So the IPA then and now responded
directly to the difficulty of the task.
Here the LHIPA as expected is showing the
inverse relationship so the greater the
difficulty the lower the LHIPA value.
And so this is what we've seen; neither
can distinguish between easy and control
tasks, however, that's not that
necessarily a bad result because in
a sense not much is happening
cognitively in the easy task and so the
distinction between those two and the
control were nothing's happening is not
so, is not so dire. Alright so
really the innovations beyond the
technical innovations, our paper introduces
two additional tasks. In 2018
somebody in the audience said well did
you use n-back? I said
no because we had to replicate
Seigenthaler et al., so here for 2020 we did
do an n-back task following the
Appel et al. paper at ETRA 2018.
It's a Tuebingen group which did a 1-back
and 2-back or at least that's what
we asked them how they did theirs and we
asked for their guidance in how do we do
the the n-back task here. And so we tried
the 4-back task in a pilot study
but found that remembering back four
steps back actually led to participants
guessing in the pilot study and so we
got results that were no different from
random. So we ended up with 1-back as
the easy task which you just remember
the letter before, and 2-back you
remember two letters before, so this
was Nina's contribution to design an
experiment following at the Appel group.
We used a limited alphabet C, F, H, I think
and maybe S you're referred back
to the paper to see how we designed
this task. Again, it's a replication of
Appel et al.'s ETRA paper. One problem
we had in this setup is
that unfortunately I use low lighting
condition and I guess I'd forgotten
about Peysakhovich's warning that if
you have the pupil already dilated it
might cause problems in finding pupil
difference related to that difference to
the original baseline if you have low
lighting condition. Nevertheless we did
do the 1-back and 2-back tasks here,
counterbalanced, and LHIPA showed, as
expected, a good response whereas the
task was distinguished from baseline. The
IPA, on the other hand, responded contrary
to expectations. In fact it decreased
with task difficulty which is not good.
Well another thing that I we did here
was we varied the gaze point to five
different locations on the screen to
control for that whereas the original
IPA in 2018 used a fixed gaze task when
people looked only in the center of the
screen. So here we're controlling for the
apparent...
the apperance of the pupil off axis which
will be an ellipse when it's off axis to
the camera in the eye tracker. And that had
no effect so that's a good thing.
Here in experiment three was our applied
setting where people were performing an
eye-typing task. Again, two difficulty
levels measured by the LIX readability
score; they had to remember a phrase and
then type it back in an eye typing kind
of application, thats' shown here. If the
LIX was less than 30 then it
was the easy task, greater than 60 it was the
difficult task. Here you can see the
letter T highlighted in red which is
what the user is fixating to type the
letter T. And in this case once again
LHIPA showed results that were
consistent with the other experiments,
in other words LHIPA is inversely
proportional to task difficulty which is
a good thing.
IPA once again gave us results
contrary to expectations. One thing to
note here, eye gaze is free to move as the
person is typing and so this is a very
good robust result of the LHIPA in this
particular example. So that's it we've
got three experiments so it looks like
we have pretty good robust results for
LHIPA. The IPA, on the other hand, only used
one wavelet coefficient band it did not
take into account the tonic/phasic
relation, uh, ratio as the LHIPA does and that
is probably the reason why LHIPA is
responding better in these three tasks
whereas the IPA only responded well in
the original fixed gaze task. So it looks
like it has pretty good potential for
being a robust measure of cognitive load
as well as for potential real-time use.
Thank you very much. Because it's a
limited kind of or virtual conference I
suppose the best way I can
entertain questions might be through
email and I'll be happy to do so I'm
happy to share the code in Python if
you like as well; I'm eager to see how
LHIPA would, how it would work in your
experiments so if you are interested in
this please do let me know and I look
forward to getting in touch, thank you
very much.