hello everyone my name is stephanie tan i am a phd student at the socially perceptive computing
lab at delft university of technology today i'm going to present my work for the topic multimodal
joint head orientation estimation in interaction groups via proximics and interaction dynamics
we motivate our approach from this typical in the wild social interaction scene human
behaviors are a dynamic and multimodal phenomena previously head orientations have been studied in
lab controlled or pedestrian settings but for in the wild settings it remains a challenging topic
our motivation is three-fold to better estimate head orientations in the wild
we argue that firstly we can leverage temporal and multimodal data where different modalities
complement each other we use a small number of room mounted cameras and wearable sensors worn
around the neck like smart id badges that record body inertial motion speaking status and proximity
from the cameras we extract positions and relative orientations of the subjects secondly we propose
a social context pulling in our neural network architecture in order to account for the fact
that human behaviors are inherently coupled with each other the intuition is that in the context
of conversations humans tend to orient their heads based on the flow of the conversation
and lastly we formulate the head orientation estimation problem as a regression problem
as opposed to a class classification which is a the mainstream approach to ensure a finer
granular estimate of the head orientations which are crucial in social interactions in the wild
to support our vision of leveraging multimodal data we propose the feature set including speaking
status triaxial acceleration body orientation and positions for estimating head orientations
we chose 10 second sequences for these signals as inputs which is a design choice by observing
speech durations in the interactions to accommodate the sequential nature of the
inputs and to capture the group context we adopt a shared lstm network-based approach
for social context pooling here we exemplify the recurrent step of the proposed model from
t to t plus one the information stored in the hidden states associated with each individual
are combined together to obtain a group level context represented by this
k vector afterwards this group level context is concatenated with the respective hidden states
of each individual and passed on to the next lstm cell
for the last point our argument in favor of regression is two-fold firstly modeling
orientations in a classification setting assumes that the classes are independent from one another
while angles are ordinal this lends naturally to a regression formulation secondly the half movements
can can be completely undetected if orientation are expressed in terms of classes as they could
be bend into the same class in our example here we show that the discretization results
in a loss of valuable information and hence a regression setting tackles these two problems
to showcase our performance of the proposed model we designed several baselines
first we devise a rule-based method to capture a specific type of dynamics that is listeners
tend to orient their head towards the speaker a speaker's head orientation is given by the
circular mean of the orientation of the positional vectors from the speaker to each of the listener
next without including the temporal information we propose a frame based method the sequence bay
method method is a simplified version of the proposed model which does not pull the hidden
states of other members of the group these two methods capture only the individual perspective
in our proposed method which we consider as a group model
we jointly estimate the head orientations of all members in an interaction group
by considering the relative information between all possible pairs of individuals
and context pooling the hidden states between the subsequent steps of the lstm
lastly we show that our proposed method outperforms these baseline methods
to conclude our contributions include first we introduced a social context pooling model in the
shared lstm architecture to capture the group context secondly our newly proposed feature set
for head orientation estimation without any visual features is a step towards a purely sensor based
and non-intrusive approach for head orientation estimation in crowded and in the wild settings we
also show the advent the advantage of regression over classification and multi-modal data
and lastly we contributed significantly to an existing data set by providing upper body
key points uh head body orientations and group memberships uh and group membership annotations
to get access please visit visit the link in the footnote which is also in the paper for more
detailed results please refer to the paper and its auxiliary material this concludes my presentation
thank you for your attention and i would welcome any questions at this point