Hi, I'm Robert Bixler from the University of Notre Dame
and I worked with Sidney D'mello at the University of Colorado Boulder on this research.
I'll be talking about building cross domain models of mind wandering using eye gaze features.
You've probably had the experience of going about your day engaged in some task only to
find you've started thinking about something else and forgot what you were doing.
This phenomonon is commonly called Mind Wandering.
Mind wandering is generally characterized by a decoupling between internal thoughts and the task at hand.
The frequency of mind wandering varies depending on the task and environment, but ranges between 20-50%
Research shows a negative relationship between mind wandering and comprehension
which indicates that tasks such as reading or learning are negatively impacted by mind wandering.
User interfaces that facilitate tasks requiring attention such as online learning systems
could therefore benefit from a method to mitigate the negative effects of mind wandering.
One potential method to address mind wandering is to reorient attention
back to the task at hand using a reactive intervention.
For example, a user interface system could display a message to notify a user
that they should repeat some portion of the task.
This type of intervention would need to be triggered when mind wandering occurs, which
requires a mind wandering detector.
Research into mind wandering detection has expanded significantly in the last decade.
Mind wandering detectors have been built using a variety of modalities and have been applied to a variety of domains.
Collecting data in order to build a new model for a new stimulus can be expensive
so it would be useful if similar pre-existing data could be leveraged to build a cross-domain model.
Building a cross-domain model that uses data from one domain to classify data from a different domain
is a type of transfer learning called domain adaptation.
The goal of our work was to build a cross-domain mind wandering detector using eye-gaze features.
We focus on eye gaze to detect mind wandering due to the eye-mind link,
which suggests a close coupling between internal processing and eye movements.
Mind wandering is characterized by a breakdown between internal thoughts and external stimuli,
which should be reflected in distinct eye movement patterns.
The current study facilitated cross-domian models by collecting data from the same participants
across stimuli from different domains.
We collected eye gaze data and mind wandering reports from 132 students while they viewed
7 stimuli in a random order for 6 minutes each.
We used two different eye tracking setups. The first used a Tobii EyeX eye tracker affixed below a laptop.
The second consisted of a monitor, an Eyelink 1000 eye tracker, and a tower mounted headrest spaced 60 cm away.
Mind wandering reports were collected using thought probes.
When a probe occurred, participants were asked to press a key if they were mind wandering.
There were 3 probes per stimulus spaced randomly between 90-120 seconds apart,
resulting in one thousand nine hundred and eighty total reports.
Two of the stimuli were controls and weren't analyzed here.
The five stimuli we did analyze have been used in previous research and consisted of
an illustrated text with two diagrams of everyday devices along with expository text displayed
for three minutes each
the first six minutes of a French narrative film about a boy following a balloon
the first six minutes of a video lecture on population growth
a series of six static images of various city scenes shown for 1 minute each
and six minutes of a sherlock holmes novel with pages presented for 5 to 15 seconds.
Mind wandering rates differed between the five domains analyzed.
The narrative film was the most engaging with a mind wandering rate of .27
and mind wandering occurred most frequently during the static scenes, with a rate of .59.
We built models using data from short time windows prior to each probe.
Each type of model used windows of the same size ranging from 10 to 50 seconds long.
We computed fixations and saccades using standard eye movement detection algorithms
and compared them using qualitative and quantitative scores.
We settled on a dispersion threshold algorithm with a threshold of 57 pixels.
Our first set of features were global features that were not dependent on stimuli.
The bulk of these were descriptive statistics of various eye movement metrics during the window of data.
We used these features for supervised machine learning.
We built models with a variety of hyperparameters as shown in this table.
We selected our final set of parameters using the area under the receiver operating characteristic, or AUROC, as our metric.
We built the models using participant-level-5-fold cross-validation.
For a given fold and iteration the training set consisted of data from the source domain
and from participants that were not in the fold.
The testing set consisted of data from the held-out participants from the target domain.
Additional data processing steps on the training set in each fold included
removal of multicollinear features, within domain normalization, and removal of outliers
We explored three additional approaches to build cross-domain mind wandering models.
The first of these was to compute locality features which captured the proportional
spatial distribution of eye movements with respect to the stimulus.
Each cell is a separate feature, so here there would be 28 different features.
We also implemented two different domain adaptation techniques.
The first was instance-based domain adaptation, which determines how well a given instance
from the source domain can predict data from the target domain.
Our method involved weighting each source domain training set instance using a meta-classifier
trained on unlabeled instances from the target domain
and training a weighted classifier to classify a different set of instances
from the target domain using the weighted instances.
Feature level domain adaptation entailed reweighting features based on
how their distribution differs between the source and target domain without considering the class labels.
Features with similar distributions between each domain received higher weightings.
Within domain models were evaluated using AUROC.
We evaluated cross domain models using the transfer ratio, or TR.
TR measures the performance of a cross domain model targeting a given domain
in comparison to the within-domain model for the target domain.
A TR of 1 or greater indicates a cross-domain model with the same or superior performance
as the within domain model, while a TR less than 1 indicates inferior performance.
The diagram set resulted in the best within domain models, resulting in an AUROC of .72.
These results overall are comparable to those from previous studies, shown in the table below.
For our cross domain models we report the AUROC for each domain pair on the left and the TR on the right.
Each cell shows performance of a given model trained using data from the domain in the rows
and evaluated using data from the domain in the columns.
Within-domain models are on the diagonal.
Primary take aways are that the illustrated text, scene, and lecture domains all transferred well among one another,
the film and text domains transferred well to one another,
and the scene domain transferred well to all other domains.
We also built composite models that were trained using two domains and evaluated on a third.
We compared the average performance of all of these models to the average performance of the
cross-domain models that were trained using a single domain.
The source and target average TR reflects the average of all models that were trained
with a given source domain or evaluated on a given target domain, respectively.
On average, the composite models performed better than the baseline cross-domain models.
We used this same type of comparison to compare the cross domain models using the global features
to two different types of models using locality features
and models using domain adaptation techniques.
On average, the locality features with a grid size of 4x4 performed better than the
global features, with improvements for all domains except the film domain.
However, neither domain adaptation technique was successful at performing better than the global cross domain models.
Our main finding was that we could transfer across domains, with
global eye gaze features achieving an average TR of .54.
We were able to improve on these results on average by building models using multiple
domains or using locality features,
but the domain adaptation techniques we implemented did not result in any improvements.
One of the main limitations was a low volume of data, which made it more difficult to use deep learning techniques.
We also were not able to effectively classify smooth pursuits because our calibration procedure
only included static fixation points.
Finally, our data was collected in a lab and included a relatively homogenous sample of university students,
so it is not clear how well these results will generalize to other populations.
Some of our future work should be aimed at addressing these limitations.
Collecting a larger volume of data from a more diverse population
and improving the calibration procedure could improve generalizability.
Future work could also explore domain adaptation techniques in more depth.
Finally, integrating cross domain models into attention-aware technologies
would help to research effective interventions.
Thank you!