Hello, my name is Stelios Asteriadis, I am an associate professor at the university of
Maastricht in the Netherlands and today i'm going to present a paper that we published about one
year ago at the ACM transactions on interactive intelligent systems (ACM TiiS) and we have had the
happiness to be invited also in IUI 2021 by the organizers in order to present it here. Actually,
this is a work of my phd candidate Dario Dotti and the title of the work is 'Being
the center of attention a person-context CNN framework for personality recognition'.
So, I am not going to start with any groundbreaking information for you
and I am just going to say that our personality is manifested in a number of ways among which
is also our nonverbal behavior. So, behavior that is coming from our facial expressions
or body movements, etc.. However, it is a fact that, although personality computing
is an area that has received a lot of attention in the last years, when it comes to making use
of computer vision and artificial intelligence techniques for infering a user's personality,
most works are coming from the from the branch of facial analysis, so, facial expressivity etc.
This means that we only have a small body of works
that deal with bodily expressivity and how this relates to our personality traits manifestation.
So, in this work, our goal is to propose a system that looks into
body dynamics in two different contexts, in order to examine whether we can generalize as much as
possible so we are going to look into different sorts of data. One piece of data is coming from
a social event and our other direction in terms of data analysis is coming from a non-social context.
So, a person being alone in one room interacting with objects so, no social interactions.
To give a broad overview, we are splitting our data representation in three different branches
and in the end we are fusing all these together making use of a deep neural network,
a CNN in particular. So, the three different branches are the following:
the context proxemics descriptor, which is a descriptor that is
describing how the target person, so the person whose personality we want to predict,
relates to other individuals or even other objects in the environment. The person descriptor,
here in the middle, that has to do with how the individual motion characteristics of a person (so,
no relation with other people or the environment) can tell something about their personality. And,
finally, we focus on social groups, so, groups of people that stand close to each other and
talk to each other, so two or three people, not considering the rest of the people in the room,
for example, in order to see how motion dynamics work within this group and how an individual can
impact the rest of the group and, then, can we say something about this individual's personality?
Right, let's start from person motion features, then. So, features that have to do only with
one's own bodily expressivity. We have used the Openpose library to extract a number of
joints' coordinates on the body and then we are looking into comparing those joints with four
fixed reference points, as we say. In our case we use the shoulders and the hips and the relation
of each of the remaining joints with regards to the reference
points is then transformed into cylindrical coordinates. We, of course, extract
these features across time as well and we end up with these three 2d map representations,
each one standing for the radius, the theta, and the z value of the cylindrical coordinates. So,
these three 2d maps encapsulate information that has to do with how each joint relates
to four reference joints on the body. Next, we are extracting features that have to do with
the the subgroups (the groups of people that talk to each other, for example, in a cocktail party,
as we will see later on) and what is happening here is that we have used clustering,
an unsupervised approach in order to split the people in the scene into a number of clusters,
which correspond to groups of people talking to each other. And, then, excluding the target
person, we see how the motion features of the rest of the people work. So, we take the motion
features as we saw in the previous slide, of each individual in the social subgroup, and
we apply a pooling strategy - max pooling or average pooling - in order to acquire
a compact descriptor for the whole group,
except for the person in question. We want to see how the group behaves in relation to who
our target person is. Right. Thirdly, we are looking into how our target person's
physical coordinates relate to those of other individuals in the scene. So, for example, if we
are talking about a shy person we would expect that this person would not be close to people
all the time. If we are talking about a highly agreeable person then we would expect that this
person would be close to people, close to other individuals. And for this reason we want to see
what are the relationships of this person with regards to other people in the group,
and for this reason we are again using cylindrical coordinates but not on a bodily basis but by
comparing the coordinates of individuals with each other using just one reference point on the body
of each individual separately. And, of course, it doesn't make sense to use the z
coordinate here because we don't really care about how tall or short a person is, but we only care
about how far they are from each other. Similarly to this, but looking a little bit more tricky,
we are making usage of the space, by an individual when it comes to non-social settings. So,
as i said before, these are settings where there is an individual alone in a room and they interact
with the environment. In order to see what are the important areas of every environment
with which this individual interacts, we apply an unsupervised method by looking
into hand motion features. Unfortunately, I don't have time to explain this now,
but you can look for more details in the paper. So, we're looking at how the motion of the hands
differs across different areas of the scene and we apply an unsupervised clustering approach here
making use of Gaussian mixture models in order to tell what are the regions of interest in the
scene. And we then apply exactly the same thing like we applied in the previous slide but in terms
of other people in the scene you have objects in the scene. Now, how does this all come together?
For bringing all these together, we have made use of a pre-trained VGG19 model,
but here we stopped at the vgg5 layer and then we trained our own
two layer fully connected network followed by a softmax classifier in order to point directly to
the most possible personality trait. So, depending on the problem, we fuse these three descriptors
coming from the three CNNs taking as input the descriptors - the cylindrical coordinates as
we described them before. So we have the representations that we discussed before,
passing through CNN networks and then fused all together in the end through
concatenating them and passing them through a two-layer fully connected network and softmax.
Finally, for our experiments, we used two datasets. The first one was the Salsa dataset
which consists of a social context where you have individuals interacting with each other
in a poster session like we would have if we were able to meet for this conference,
and also the cocktail party - so, the coffee break - of this conference was used. Data from
the coffee break of this conference were used for analyzing the dynamics
in the non-verbal behaviors of the individuals. Secondly, we are using a dataset that we recorded
a few years ago where we asked 45 individuals to interact with
their environment. We left them alone in the room and then we asked them to do a number of tasks
like make some tea, look for some objects, etc. And, then, by analyzing their bodily behavior we
want to tell something about their personalities. Unfortunately, my time is finishing soon, so i
will simply mention here that we made use of two personality trait models: The big-5 personality
traits which is probably the most common model used in personality computing and then the three
personality traits that are actually a combination of the big-5 traits here. So, the
big five have to do with neuroticism, openness, contentiousness, extroversion and agreeableness,
whereas the personality types model consists of the undercontrolled, resilient and overcontrolled
traits. Here are some results. We conducted an ablation study by mixing
and matching different descriptors from the three ones that we discussed earlier
and we also compared with previous approaches which have been applied in similar problems but
it was the first time here that we applied them on these specific datasets. It is worthy to seay here
that we followed a leave-one-subject-out protocol meaning that the test subject had nothing to do
with the rest of the training data. So, every time we were using (for testing) a new subject
that the system hadn't seen during training. Some more results on the non-social dataset here and
here are results on the big-5 model which i would like to very briefly comment on.
So, as we see here for the Salsa poster session -so, for the social
dataset - we see that the openness to experience trait has the highest accuracy,
which means that this part of this dataset is very well fitted for this kind of feature
because we are talking about academics here, we are talking about researchers here, so openness to
experience is probably something that shows really easily in a poster session, whereas extroversion
and neuroticism showed mostly in the cocktail party (as they call this part of the dataset).
On the contrary, extroversion was not so apparent in the non-social dataset
which means that we don't show how extroverted we are probably when we are dealing with stuff
in a room by ourselves. So, this seems to be also in accordance with psychology.
And this concludes my presentation and my video. If you want to have
more details about our work and our works in general I would encourage you to visit our website
and get to know more about us. Thank you very much!