Hello and welcome to our presentation on "investigating multimodal measures for
cognitive load detection in e-learning". In traditional learning human teachers
usually try to estimate their students cognitive load and react to it. So, add
further explanations when they feel like their students were overloaded and did
not properly understand everything or move on to more complex topics. While the
e-learning industry is now growing rapidly especially due to the corona
crisis, e-learning platforms still usually rely on some static linear order
of contents, so you just go through one by one, there's not much adaptation to
your current cognitive state only a little bit of customization to you what
you did in the past or how well you performed in the quizzes. So the
literature proposed the concept of "cognition-aware e-learning", where the goal
is that students stay in the optimal range of cognitive load to facilitate
learning and they propose plenty of approaches to estimate cognitive load
for example based on consumer devices or on EEG sensors. Apart from these works
that propose different sensors and correlate them to some measure of
difficulty, like facial expressions, eye tracking, features. Other papers have
focused on good use cases, you know, how can you adapt to the measured
cognitive load. For example you could recommend learning contents that are
most appropriate for your current situation. Or you could adapt the whole
interface. On the other hand, there are some papers that go beyond and try to
really train predictive models. For example a Ridge regression model to
estimate task complexity or SVMs for binary classification if the student
will succeed or fail. However these models usually focus on some few sensors
for example EEG or eye tracking features. And on the other hand they often
don't really have a learning task but more a mental arithmetic task. So you
need to add up some numbers and depending on the difficulty of the task you can
expect the cognitive load to be high or low.
So what we propose instead is to have a more diverse set of
cognitive load measures but also a more realistic study where people really
learn with videos and quizzes as is common in the e-learning setting. And
then we want to try to also investigate which modalities, so heart features or
skin features or eye features, tend to work better than others. For our study we
first handed out at pre-questionnaire to capture demographics but also the
exhaustion, tightness, and stress and background in the topics that we then
chose for the later parts of the experiment. Then we attached all the
different sensors which we'll see in a bit to the participants and showed them
the Moodle environment in which we displayed the videos, where we chose
three different topics, namely integration, vectors, and eigenvectors.
Each with an easy and a hard video, where the easy video captured content
that was part of the high-school certificate and the hard content
is part of the curriculum of the university. And to ensure a very similar
presentation style we use the "Mathe by Daniel Jung" channel, where we have the
same speaker presenting in the same way to limit effects based on a
different teaching style. And after each video we ask them to assess cognitive load
and difficulty. Following the videos there always came a quiz with multiple
questions and again they had to assess the cognitive load and difficulty of
solving the quizzes. Then there was a small break task to omit the
interference between the content items and we did so for all six content so we
counter-balance across the three topics. In the end, we had a small post-
questionnaire to again capture exhaustion, tiredness, and stress. This
image shows the sensor setup that we used, so let's go through that one by one.
First, we have some rating scales, so subjective measures: one for cognitive
load, which is "in solving or studying the preceding problem I invested" "very, very
low mental effort" to "very, very high mental effort". And the difficulty scale,
namely "the preceding content was" "extremely easy" to "extremely difficult".
And both these questions appeared within the Moodle environment after each video
and after each quiz. For the quizzes we also
captured how long it takes the students to solve them and how well they perform
so how many questions they get right and wrong. As a behavioral feature, we use the
Microsoft Kinect v2 to capture the skeleton and with it we calculate the
distance between the head and the screen, hypothesizing that you come closer to
the screen when you are cognitively loaded. For eye tracking, we integrate the
Tobii 4C eye tracker and a webcam With the webcam you just calculate the
openness of the lids which you see on the right so that's the is eye aspect ratio and
with the eye tracker we capture blinks, fixations, saccades and of course the
durations. We reimplemented a paper to capture if the user is searching for
something and we integrate pupil diameter measures such as the index of
cognitive activity. As cardiovascular measures we integrate the Empatica E4,
the Polar H7, and the Garmin Forerunner 935 and with them we capture the heart rate but
also the RR interval, from which we calculate the heart rate variability
measures, so the RMSSD, SDNN and pNN50. And also blood volume pulse measures. As skin measures, we again use the Empatica E4,
the Garmin Forerunner, but also the Microsoft band v2. And with them
we capture the galvanic skin response, which we use then also in the Ledalab
software to calculate some high-level features on them. And we use the skin
temperature as a measure. Here you can again see an overview of all the measures, we we have two
subjective scales, two performance measures, one behavioral measure and lots
of physiological measures for eyes, heart, and skin and of course these can be
combined in several ways. We can distinguish these features into two
categories: on the one hand, "single features", so for examle subjective ratings where we just have
one value across some time interval. And on the other hand we have continuous
streams of data, for example for the heart rate, which we then first
normalized by the participants mean and then we compute five simple features
so the average, the standard deviation, min, max, and the range, which is
just the max minus the min. And this is what we do for all the remaining features that
you know where we have a continuous stream. So overall this gives us 202
features which we of course filtered a bit according to the
literature but also according to visual inspection. As participants we have 21
students, 17 of them male. They were studying computer science so we ensure
that they had already passed the relevant math lectures, so that they have
a comparable background in the topics that we chose for our experiment. They are of
course rather young. Half of them had good or very good e-learning experience on
different platforms. They found that the topic vectors is significantly easier
than the other two topics. In terms of tiredness exhaustion and stress we could
not find a significant difference between before and after the experiment
which is good, so there are no strong tiredness effects that might have
biased our data and they had a high motivation to follow the videos and a
very high motivation to perform well on the quizzes. When we look at the ratings
that the participants assigned to the contents, we see that for vectors eigen
vectors, as well as across the three topics, the differences between the easy
and the hard videos or quizzes were there and were significant. However for
the integration case that was not the case, so we also need to consider the
topics individually. When we look at the correlations between the cognitive load
ratings and the difficulty ratings, we see that they are highly correlated, so
for participants the two things were apparently very strongly related and
almost the same. And if we look at the results and the quizzes we see that
participants were indeed much faster for the easy than for the hard quizzes. But
in terms of how well they answer the questions there was only a marginal
difference between the easy and the hard topics. One very simple thing to
investigate is: how strong is the link between all of these features and our
three target variables, so the difficulty of the content, so easy or hard, as well
as the subjective rating: subjective cognitive load and subject
difficulty. However we have very many features and three targets, so we cannot
really look at all correlations. So what we did as a start was just to look at
the highest correlations of some feature towards these three targets within our
cases: across videos, within the integration topic, within the vector
topic, and within the eigenvector topic. And what we see here is that the
correlations are overall quite good, we have values up to 0.53
and many around 0.4. But for the across video cases, even the highest
correlations are still pretty bad: 0.14 to 0.18. So this suggests that it will be
much harder to predict this case. So let's try to predict: first the intrinsic
difficuly, so this is a binary classification task whether it's
a High School Certificate content or a university degree content. And what is
interesting here is that if we use a very simple Logistic regression model, we
achieve some 80 to 90 percent accuracy for the quizzes but only some 70 to 75
percent accuracy for the videos except for one outlier, for which we
unfortunately don't really have an explanation, we think it's some artifact
on the data. So might it be that the degree of activity really impacts how
well you can predict it? So is it much easier to predict within quizzes where
people are actively engaged. If we look at the regression results, so trying
to predict the subjective difficulty or subjective cognitive load we see that
within topics now we get very low mean square errors especially compared to the
simple baseline that always predicts the the median rating so it seems to work
very well here. Whereas across topics it also works okay
for the quizzes, not as good as within a topic but still quite good, but not at
all basically for the video case, there are only marginal gains to the baseline. And
again this might be due to the active versus passive consumption of videos and
solving quizzes. Or maybe also because when you only passively watch a
video and the hard video of one content might not be much harder than the
easy video of another content, it's of course hard to distinguish that later on.
So those were the results when predicting using all features but we
also are interested in which modality is better and worse. And for this we look
at the intraclass correlation coefficient which is an indication of
the extent to which different participants produce the same values of
a given measure when they are exposed to the same condition. So it tells us which
features generalize well from a small number of participants. And what is quite
interesting is that even though all the measures are proposed in the literature,
we see that only 2.5% of our features can be considered
"excellent" in terms of ICC, 3.5% is "good", 27% is "moderate"
and the majority of 67% is "poor".
However one should note that this can also be the case because we
calculate these five simple features on top of the raw signals
but still it's interesting because most works don't report the ICC and we see
that many sensor features here are actually pretty bad. Well of course a
few are pretty good as well. And if we look at the modalities that
the good ones belong to you we see that almost all belong to the "eye" category -
all excellent ones and the majority of the good ones
plus some moderate ones so this is clearly the best in terms of ICC. Closely
followed them by "heart" and "skin", which both constitute some of the
good and moderate features. Whereas the body posture features can only be considered
rather pure, so that's also quite intereting. So let's look how well we can predict
based on some modalities. In this chart we have the prediction of subjective
cognitive load on the left, subjective difficulty in the middle, and intrinsic
difficulty on the right. So on the right we have a binary classification task, so
that's the accuracy, and the two things on the left are the mean square error. We see in
the blue charts what we saw already before, namely what happens when you use
all features from all modalities simultaneously. The green ones are what
happens when you use only features from one modality and here we see that eye
and heart apparently a bit better than skin features. But we
also see that if you use more than one madality, so if you combine all or
use pairs of two, which would be the yellow case, you get better results than
single modalities and combining the two things namely eye and skin is almost as
good as the multimodal case which has all three. So to conclude, we expored a
very wide range of cognitive load measures in a rather realistic e-learning setting with videos and quizzes with 21 participants. Where we try to
predict subjective cognitive load as well as subjectively assessed difficulty. And the
intrinsic content difficulty, so whether it's a high school or a
university content. Where we were able to achieve high accuracy and low mean sqaured error
so many adaptations from the literature might be possible in practice.
We also saw that findings achieved in some more artificial
mental calculation tasks could be replicated in this more realistic
setting here. And we saw that it's easier to predict during quizzes than during
videos where are people actively engage with the content and not just
passively consume it. In terms of modality analysis, we see that from our considered
features, the eye features perform best followed by heart and last skin. But of
course when we combine several modalities you get better results. And
this extends the findings by Naismith and Cavalcanti who didn't have the
skin in their analysis. In the future we want to try to get a more real-time
analysis to predict the cognitive load based on shorter windows of time, and
to look at the individual features and not just at the individual modalities as
we did in this work. Furthermore you should consider the limitations of the
study: of course we only have 21 participants and 6 contents so further
studies are required in the future. Thank you!