Hello everyone, my name is Rafael Wampfler, and I'm a Ph.D. student at the Computer Graphics
Lab of ETH Zurich.
Today, I will present our paper on affective state prediction based on semi-supervised
learning from smartphone touch data.
Our novel system predicts affective states of users during smartphone-based chat conversations
by using touch data.
Our system works unobtrusively and in real-time in the background of a smartphone.
The predicted affective states can, for example, be visualized using a pie chart.
The segments are filled according to the level of the corresponding affective dimension.
This type of visualization can be useful to convey information to others about the user's
current affective state like status messages.
Our approach can also be used to increase self-awareness of users or as part of therapeutic
chatbots such as Woebot, which aims to improve the overall mood of users.
Woebot infers the mood of users from chat conversations by using natural language processing.
Woebot then adapts the conversation to the predicted mood of the user.
By considering also touch data with our approach, the bot would be able to increase the mood
prediction accuracy.
Affective states are psycho-physiological constructs used to characterize the emotions
and moods that arise and are experienced while individuals are engaged with a stimulus.
In our work, we are representing an affective state using this three-dimensional model shown
here consisting of valence, arousal, and dominance.
Valence indicates if a feeling is positive or negative, arousal describes the intensity
of the feeling, and dominance indicates the degree of control over the situation.
The majority of systems to detect affective states rely on biosensor data such as heart
rate and skin conductance, or camera data to infer emotions from facial expressions.
However, most of these setups are invasive and potentially costly, which can limit their
widespread application.
The quality of affective prediction models inherently depends on the quality and amount
of data available to train them.
Unfortunately, user studies are often expensive to run, so many times we have not quite as
much labeled data as we wanted to have.
From this limited amount of labeled data, most systems extract features manually.
Manual feature extraction relies on domain knowledge and does not leverage the larger
amount of available unlabeled data.
Our lightweight and less invasive system extracts touch input characteristics of users while
typing on smartphones and aggregates these metrics into two-dimensional heat maps.
Then, a semi-supervised classification pipeline dynamically predicts affective states of the
user based on automatically extracted features from the heat maps.
For automatic feature extraction, we employ autoencoders on the heat maps.
In the past, different work used different data sources to predict affective states on
smartphones.
Likamwa and colleagues used multimodal data consisting of social interactions, browser
history data, application, and location data.
They assessed their system with 32 participants.
On the other hand, Gao and colleagues used touch data such as pressure and speed of touch
to predict affective states while users played a game.
They assessed their approach with 15 participants.
In contrast to these previous works, we conducted a large-scale experiment with 70 participants,
and we used chat conversations.
Chat applications are especially relevant as they currently rank as the most used applications
on smartphones.
By using two-dimensional heat maps in contrast to raw data, we are also considering the spatial
distribution of the data.
Finally, in contrast to manually crafting features, we employ autoencoders to automatically
extract features from the touch data.
In the past, autoencoders have shown to provide large improvements in classification accuracy
in other domains.
For example, Kingma and colleagues were able to improve the classification accuracy of
handwritten digits by up to 26% by employing autoencoders.
As another example, a method for the classification of real-world images reached a reduction of
the error by 50% using autoencoders.
Our semi-supervised classification pipeline for predicting affective states based on touch
data consists of the following three steps.
First, we extract two-dimensional heat maps from touch data.
Then, we train a variational autoencoder on the extracted heat maps.
Finally, we classify affective states based on automatically extracted features from the
autoencoder.
We use a sliding-window approach to aggregate individual touch inputs into three types of
heat maps.
The first type represents the touch pressure during typing.
The second type provides information about the typing speed and is equal to the time
difference between two consecutive touch downs normalized by the distance.
The third type provides information about the speed between touch events.
For the down-down speed and up-down speed metrics, we assign their value to every pixel
on a straight line between the events.
All metrics are standardized based on the mean and standard deviation during a baseline
typing period.
While touch data is continuously available, ground truth is typically only available in
certain intervals (for example, from self-reports).
To make use of the large amount of unlabeled data, we employ variational autoencoders to
infer meaningful low-dimensional embeddings from two-dimensional heat maps.
A variational auto-encoder consists of an encoder and a decoder.
The encoder network learns an efficient compression of the heat maps into a low-dimensional space
using a deep neural network.
The decoder network then reconstructs the input based on sampling from the distribution
of the latent space.
For the encoder and decoder, we used convolutional and deconvolutional layers.
We modeled the latent distribution with a multivariate Gaussian parametrized by mean
and variance.
In the end, we reduced the input dimensionality from 80 x 64, that means 5120 dimensions,
to 20 dimensions in the latent space.
For the last step of our pipeline - the classification - we then added a fully connected classification
layer to the learned data encoder and fine-tuned the entire network for the classification
of affective states.
We have trained an autoencoder for each type of heat map and stacked the latent space of
the individual autoencoders.
To assess our pipeline, we have conducted a lab experiment with 70 participants between
the ages of 18 and 31 from 20 different departments of ETH Zurich and the University of Zurich.
We captured more than 70 hours of touch data.
From this data, we extracted over 44'000 heat maps for each type of heat map.
After the participants have been instructed, the experimenter exited the room and used
one of the Skype contacts to start a short small talk conversation with the participants.
This conversation was used to make the participants comfortable with the keyboard and the handling
of the smartphone.
Next, participants were instructed to watch a nature video for five minutes on the smartphone
that was used as relaxation and allowed them to acclimate to the room environment.
At the end of the nature video, participants were asked to type two well-known pangrams
that served as a baseline for touch input during the modeling stage.
To trigger different affective states, participants engaged in four different Skype conversations
with four different contacts.
Participants were told before the experiment started that the four contacts were real people
sitting next door.
The order of conversations was randomized across participants.
The conversations consisted of a shocking conversation about the Rohingya genocide,
an exciting conversation about the most beautiful holiday experience of the participants,
a rude conversation,
and a confusing conversation with a chatbot.
Every 90 seconds, the participants had to fill in a self-report measuring the affective state
in terms of valence, arousal, and dominance on a 9-point scale.
In addition, we measured the presence or absence of stress and the basic emotions anger, sadness,
happiness, and surprise.
In total, we collected 1893 self-reports.
Here, we see the changes in valence, arousal, and dominance during the four chat conversations
for one participant.
The figure shows that valence and dominance increase during the exciting and confusing
conversations and decrease for the other two conversations.
For arousal, we see the opposite pattern.
We have evaluated our model using leave-one-user-out cross-validation.
Here we present the ROC curves for predicting valence, arousal, and dominance on three levels.
The red, green, and orange curves represent the performance we achieved using only pressure,
down-down speed, and up-down speed heat maps, respectively.
The blue curve represents the performance when using all heat maps in combination.
Our results show that we can accurately predict three levels of valence, arousal, and dominance
with an AUC of up to 0.84.
For valence, the typing speed heat maps performed best.
For arousal, pressure heat maps showed the best performance.
And for dominance, all types of heat maps performed equally well.
Altogether, the combination of the heat maps provided only marginal improvements over the
individual heat maps.
Here we see the confusion matrices for valence, arousal, and dominance based on the combination
of all heat maps
For valence, arousal, and dominance, the low
and high classes were often wrongly predicted as the medium class.
As expected, the larger the distance between the classes, the easier it is to differentiate
them for our model.
That means the low class was only rarely confused with the high class and vice versa.
Interestingly, for arousal, the medium class was most often wrongly predicted as the low
class, but medium dominance was more often confused with high dominance.
Our system is also able to predict anger, sadness, happiness, surprise, and stress on
two levels with an AUC up to 0.88.
Usually, affective states change over time.
We hypothesized that the performance of our classifier could be affected by the period
over which affective states are constant.
For example, if affective states are alternating in short time, it can be much harder to make
an accurate prediction compared to when affective states are constant over a longer period.
We recalculated the accuracy measure by considering only the data points for which the affective state
was constant over a certain period.
That means a specific number of preceding data points with the same class.
A sequence length of zero corresponds to considering all data while sequence lengths of one, two,
and three imply that we only considered data points having at least one, two, and three
preceding data points with the same label.
By excluding only immediate jumps, which means a sequence length of one, we observe a steep
increase in accuracy.
In contrast, increasing the sequence length to two or three preceding data points provides
only marginal improvements.
We have presented a semi-supervised pipeline using automatic feature extraction from touch
data based on autoencoders.
In a lab experiment with 70 participants, we have shown that our pipeline predicts affective states well.
Our system is lightweight, less invasive, and can be used on different types of mobile devices.
In addition, our system can make two new predictions every second.
We have also shown that accuracy depends on the sequence of previous affective states
and that accuracy tends to increase if affective states are constant over a longer period.
Of course, our results also have some limitations.
We have used a lab setting, which means a controlled environment.
Going into the wild will be the next step.
And our setting is restricted to chat conversations.
Thank you very much for your attention.