Hello everyone, my name is Rafael Wampfler,
and I’m a postdoctoral researcher at the Computer Graphics Lab of ETH Zurich.
I will present our paper on affective state prediction from smartphone touch
and sensor data in the wild.
Our novel system predicts affective states during smartphone-based interactions.
Our system can be useful as part of therapeutic chatbots to improve the overall mood of users.
Based on touch and sensor data collected during chat conversations,
our system can predict various affective states.
A therapeutic chatbot can then adapt the conversation to the predicted affective state of the user.
The predicted affective states can also be visualized using a pie chart.
The segments are filled according to the level of the corresponding affective dimension.
This type of visualization can be useful to convey information to others about the user's
current affective state like status messages.
Affective states are psycho-physiological constructs used to characterize the emotions
and moods that arise and are experienced while individuals are engaged with a stimulus.
In our work, we are representing an affective state using this three-dimensional model
consisting of valence, arousal, and dominance.
Valence indicates if a feeling is positive or negative,
arousal describes the intensity of the feeling,
and dominance indicates the degree of control over the situation.
The contributions of our work are threefold.
First, we collected a large-scale dataset of smartphone touch and sensor data from a
user study with 82 participants conducted in the wild over 10 weeks.
Second, we present a deep convolutional affective state classification model that is trained
on two-dimensional keystroke and sensor heat maps.
Finally, we assessed our model based on the collected dataset and we show that we can
accurately predict affective states from inertial sensor data alone.
We conducted an experiment in the wild to collect a large-scale dataset of smartphone touch and sensor data.
Eighty-two Participants used our Android application on their smartphones for ten weeks.
The application recorded sensor data, context data, and usage logs when the phone was unlocked.
Using the app, participants could also track their progress in terms of self-reports and
statistics including their rank in relation to other users.
Participants were also required to use our custom keyboard during the study.
From the keyboard, we recorded the touch positions.
By pressing the star button at the top of the keyboard, participants could start a self-report.
The self-reports measured valence, arousal, and dominance on a 5-point scale.
We also measured the presence or absence of the basic emotions and stress.
In total, we collected over 30’000 self-reports.
Our model consists of three blocks.
First, as input to our model, we used heat maps created from keystroke and sensor data.
Second, we trained a convolutional neural network on the heat maps to automatically
extract meaningful features from the heat maps.
Third, for the classification of the affective states, we used the features learned by the
convolutional neural network as input to fully connected layers.
To generate two-dimensional heat maps, we aggregated typing characteristics covering all bigrams of 38 keys.
We encoded all possible bigrams in a 38 times 38 heat map.
The first and second keystrokes in a bigram are encoded in the row and column, respectively.
We placed more frequently used keys in the middle of the heat map.
For example, the typing speed between the first two characters of the quote, that is L and I,
are encoded in the L row and I column.
In this way, the heat map is built up for all consecutive bigrams.
We averaged all the values for each cell in the heat map.
In addition, we standardized the heat maps based on the mean heat map during a baseline typing period.
In the heat maps, we encoded three typing metrics.
The first type measures the time of moving from one key to the next key.
The second type measures the time of moving between keys as well as the hold time of the first keystroke.
The third type considers the time of moving between keys and the hold time of the first and second keystroke.
We extracted these three types of heat maps using a window of 80 keystrokes.
For creating the sensor heat maps, we considered the gyroscope
and linear acceleration measurements of smartphones.
We encoded the three-axis combinations into three heat maps.
Using a window of 30 seconds, we binned the absolute sensor values into logarithmically
spaced bins and counted the number of values in each bin.
Again, we standardized all heat maps based on the mean heat map during a baseline period.
We assessed our model by training on 81 participants and testing on 1 left out participant.
We have repeated this for all 82 participants.
Using the combination of keystroke and sensor heat maps, our model can accurately predict
three levels of valence, arousal, and dominance with an AUC of up to 0.85.
For predicting valence, the keystroke heat maps outperformed the sensor heat maps.
On the other hand, for predicting arousal, sensor heat maps performed best.
For dominance, the keystroke and sensor heat maps performed comparably.
Our model is also able to predict stress and the basic emotions anger, happiness, disgust,
sadness, surprise, and fear with high accuracy.
We have also assessed our model on heat maps extracted on larger windows
ranging from 2 minutes to 30 minutes.
For a window size of 5 minutes, peak performance is reached.
The performance decreases with larger window sizes.
Performance can be further improved by fine-tuning the network per user.
Peak performance is reached when we fine-tune the network on the first 40 to 60 self-reports
per user and test on the remaining self-reports.
In this work, we have presented a pipeline using automatic feature extraction from heat maps
generated from typing and sensor data.
We assessed our pipeline on data collected in the wild from 82 participants over 10 weeks.
The sensor heat maps performed comparably to the keystroke heat maps.
As such, we believe that sensor data can be an alternative to touch data to protect privacy better.
We have also shown that the performance increases by fine-tuning the model per user.
Of course, our results also have some limitations.
We only considered windows containing typing and our data collection was restricted to a population
consisting of German-speaking bachelor and master students.
Thank you very much for your attention!