This work is about comparing and combining interaction and eye tracking data
for predicting user cognitive abilities in real-time during visualization processing
and this work was done at the University of British Columbia
Information visualization are more and more prominent in our daily life
and we use them for a wide variety of tasks which include
public engagement, checking disk usage or analyzing social network, and a lot more things.
However there is very strong evidence that user cognitive abilities impact how effectively and confidently user can process such visualizations.
This has been shown for perceptual speed, working memory and a few more abilities.
And there is evidence that eye tracking can be used in to predict some of those cognitive abilities in real time during visualization processing.
Altogether these two findings show that there is potential for designing intelligent user adaptive visualizations that can first
detect at runtime the levels of cognitive abilities of the user
and provide them with a more personalized or adaptive interaction to facilitate visualization processing.
However the prediction of user cognitive abilities has been limited so far to the simple setting shown on this picture
namely one bar or radar graph
and fictional question-answering questions which do not mimic or visualizations are used in our daily life.
So we wanted to extend these works in two different ways.
First by using a more realistic settings.
namely by using MetroQuest which is a commercial platform for public engagement widely used in North-America.
and second by using interaction data in addition to eye tracking data to perform such predictions.
First to understand if this can improve the accuracy of the predictions
And second to understand if an eye-tracker is required at all to perform such prediction of cognitive abilities.
MetroQuest is typically used to educate the public about urban planning scenarios.
In this particular study we use the MetroQuest shown on this screen.
namely users were provided with maps and deviations charts to compare three public transit scenarios.
And these transit scenarios are actually currently studied by the city of Vancouver, so this is a real project
widely covered by the news so people know about it.
And all of the data shown in the map and the char tare real data published by the city.
And in addition the 166 participants recruited for this study
all belong to the community affected by the public transit scenarios
so that they actual care about the data shown in the interface.
In the study we first tested the cognitive abilities of the users.
And then we let them interact with MetroQuest as much as they want and with no training
to explore and rate the different public transit scenarios.
This is how MetroQuest is used in real life.
We tracked the gaze of the users using a Tabii T120 eye tracker.
Afterward we assess the experience of the users with MetroQuest using interviews and questionnaires.
Ina first round of evaluation we actually found that 4 cognitive abilities significantly impact how user processed MetroQuest and their experience with the system.
and they are listed on the slides.
The first is perceptual speed at performing simple visual tasks.
the second is visual working memory, to store visual information.
Same for spatial memory to store spatial information.
And visual scanning is the speed at locating objects in our surroundings.
Some of the results we found for instance and there is more in the paper,
High visual working memory users preferred the charts over the maps and they found the maps difficult to process.
so there users may need some adaptive help to process the map in a more effective and confident way.
and then another example is that users with a lower perceptual speed made fewer comparisons of the public transit scenarios
although it is an important part of the MetroQuest system
so these users may need more hints on cues to perform such comparisons.
Altogether what we want to do in this work is to
predict these 4 cognitive abilities in real time to see if we can enable the type of intelligent support I just described.
We define the prediction as binary classification tasks
namely for each of the cognitive abilities we split users in a High or Low group
depending on their levels of the cognitive abilities
and we use a median split to do so.
To classify users as High or Low we leverage a set of features computed over their eye tracking and interaction data
Those features are meant to capture the overall levels of activity of the users
and their attention to specific areas or interactive features in the interface.
We also use a combined feature sets that is basically the addition of the eye-tracking and interaction feature sets.
We train random forest classifiers over 25-runs, 10-fold cross validation over users for each of the cognitive abilities.
This is a high level overview of the setting and it is described in much more details in the paper.
Some of the main results we found are shown of this slide.
We have four charts, one for each cognitive ability.
On the charts the X-axis is called Window length and
and it is meant to capture how much data is used to train and predict the cognitive abilities.
So for instance at window 10 we use only 10% of the interaction to make the prediction and at window 100 we use the entire interaction.
This is to know how soon we can make accurate predictions during interaction with MetroQuest.
The Y-axis is the prediction accuracy.
the horizontal brown lines is the majority-class baseline and then
the other lines are the accuracy of the random forest classifiers for each of the
combined feature set, eye-tracking feature set and interaction feature set.
Some of the findings we found is that eye-tracking is still found to be better for three of the 4 cognitive abilities
where you see the red targets, so perceptual speed, spatial memory and visual working memory.
In addition the Combined classifiers were not more accurate than eye-tracking alone
This tends to show that an eye-tracker is still required to predict most of our cognitive abilities.
This said the interaction classifiers were still found to be very promising for visual scanning at the bottom left
because it can make accurate prediction at the very first window which is very promising.
And in general we still found that the interaction classifiers were overall more accurate than the majority-class baseline
which shows that there is potential of interaction data to predict cognitive abilities with a bit more research
Another important finding we found is that very early prediction is possible for three of the cognitive abilities where you see the red targets again,
right at window 10% for perceptual speed and visual scanning, and 30% for visual working memory.
This is very promising because the sooner we can make prediction of the user cognitive abilities,
the sooner we can provide them with more personalized or adaptive support or interaction.
And even for the last ability, spatial memory, we can still make accurate prediction halfway in the task
which still leaves some time to provide personalized support.
This was an overview of our results and there is more in the paper including a discussion of the classifiers.
To conclude we showed that different cognitive abilities that are known to impact visualization processing
can be predicted from both eye tracking and interaction data.
although we found that eye-tracking is still more accurate overall on most of the target cognitive abilities.
But still interaction data can be promising for some specific abilities with the extra-value that there is no need for an external device to track the user's gaze.
Altogether this work extends previous results that were found in simpler settings and tasks, with different visualizations and cognitive abilities.
here we show that prediction of cognitive abilities is possible in more realistic settings, using MetroQuest
which is a real system using real data and visualizations.
Altogether this contributes to showing that
that it is possible to design intelligent user adaptive visualizations.
As a matter of fact our current work is to study how we can design some forms of adaptive support in metroQuest
by using our classifiers in real-time t recognize the levels of cognitive abilities of the user
and provide them with adaptive support in real time.
Thanks for watching!