Hello and welcome to today's presentation of our work "Quantifying the Relationships
between Everyday Objects and Emotional States through Deep Learning Based Image Analysis
Using Smartphones.
Today's short presentation will have two parts: in the first part I'll be talking about the
background of our work and also the methodology that we used in our study. Then, in the second
part, I will be introducing the results that we obtained and also discussing their implications.
So to get started, I'd like to give you a bit of background and discuss our methodology.
Essentially, in the past few years we've seen quite a few works that link emotions on the
one hand and various smartphone and sensor measurements on the other. This has included
data such as location and mobility patterns, smartphone interaction data, and social media
data.
What we do in this work is that we instead propose to use another modality: visual information
from everyday scenes. Specifically, we looked at the objects that appear in these scenes.
The main question that we are trying to answer in this work is: "Are everyday objects linked
to emotions?".
In terms of the methodology of our study, we've had participants submit mood reports
together with pictures of their surroundings. In these reports, we asked them to tell us
their levels of tense arousal, energetic arousal, and valence on 7-point Likert scales. In the
questionnaires themselves we used the more commonplace terms "stress", "activeness",
and "happiness". Then, we applied deep learning based object detection methods to extract
the objects that appear in these photos automatically. Finally, we analyzed the associations between
emotional states and the objects that appear in these pictures.
There wasn't a dataset that already existed which allowed us to test these hypotheses,
so we developed an Android app for the purpose of this study. This app was used by participants
to record ESM reports: we've asked them to submit 4 of these a day. With the photos that
they took with these reports, we asked them to tap the photos to indicate the objects
they feel are relevant to their emotional states.
We've had 22 participants and around 3000 mood reports over a minimum duration of one
week. The study ran from September 2018 to October 2019. We also added some additional
features in the app to encourage participation such as being able to view mood statistics,
a gallery of photos that they have submitted and also a "mood map" which allowed participants
to view their emotions at different locations during the study.
To summarise this work in one picture, users submitted photos using the MyMood app together
with reports of their emotional states. Then, we took all these photos and fed them through
an object detection method based on deep learning. Then, finally, we analyzed the relationships
between these objects and the emotional states to figure out relationships that hold across
all the participants.
In terms of results, I will have to be quite brief but you're more than welcome to check
out the paper in case you would like more details.
In terms of the frequency of objects that we saw appear in these photos, we noticed
that they follow a power law distribution with a very long tail: that is to say, there
are many objects that appear very rarely while few objects appear very frequently. Additionally,
we found that indoor objects are much more common in these reports: for example, computer
keyboard and laptop were the most frequent object classes that appeared in these reports.
In terms of the associations between objects and mood, we found for example that (in this
leftmost column) work-related objects associate with increased levels of stress, while objects
that you would expect to appear in a home associate with decreased levels of stress.
In terms of activeness, we also saw different correlations between objects that you would
expect to find indoors such as "bed" and "furniture", while outdoor objects correlate with increased
levels of activeness. We also found that relationships with valence or "happiness" were quite a bit
weaker: it is more difficult to associate this with objects in the external environment.
Finally, we also found that these associations become stronger when location is taken into
account. For example, a computer might be associated with increased levels of stress
when the user is at work, but it is typically associated with decreased levels of stress
when the user is at home, suggesting they might be engaging in leisure activities (such
as playing computer games).
In terms of the implications of this work, we've presented another modality that can
be used for mood-aware intelligent applications. We think that the findings of this study are
relevant to computational psychology studies and also more broadly to architecture (for
the design of indoor and outdoor spaces for human use). In terms of future work, while
in this study we considered the collection of photos using smartphones, it would be much
easier to gather this data using unobtrusive devices such as smart glasses and IoT cameras.
That is the end of my presentation, my contact details are there if you would like to reach
out with questions -- I would be happy to answer. I would also like to thank my collaborators
Laura, Abhinav, and Mirco and obviously yourselves for your attention. Thank you very much.