hello my name is kshitij and today i will be presenting our paper uh with title assessing
cognitive performance using physiological and facial features generaliziing across context
this is a paper that i wrote with three of my colleagues in Evangelos
mikhail vasslis uh from ntnu tu delft and university of Melbourne so let's start with what we did in
this paper so in this paper we started moving from context dependency to generalizability of features
and these features were computed from facial and physiological responses of
participants and unlike other papers where they use even if they use a bunch of studies they the
studies are very similar in context so the context independent uh features are a bit difficult
to find and in this paper we use four studies from very diverse contexts and then we applied machine
learning pipelines to find generalizable features i want to emphasize that this is not a paper on
transfer learning and i will come back to this towards the end of the presentation we propose a
new feature generalizability index to quantify the generalizability of a given feature based on how
well it performs in a different context and finally there is a wrapper context here which
is cognitive performance because we do not claim that we have found features that are
universally generalizable we can say that within the context of cognitive performance
we have achieved significant generalizability of features now the question is why to switch
generalizability so important so feature generalizability is important because
um a priori knowledge about which features to compute and how in a new complex is very important
given that um in ubicom psituations there are certain contexts where you don't have some time
to play with the data and to define to define which features to compute in an institute
experiment um we used facial data in this paper because uh emotions have a very close proximity to
cognitive performance and recent studies have shown that physiological responses also have nice
proxies to cognitive performance now to achieve this goal of finding generalizable features or
finding a method to claim that if a feature is generalizable or not we use a highly diverse
set of contracts so for example we use some games some self-assessment tests some debugging problems
and we engineer features from these contexts we engineer features from the data collected from
these contexts especially facial and physiological responses um then we trained uh the model on data
set a and tested it on data set b and based on the training and testing performances the comparison
of the two performances um in terms of uh residual error we um come up with a generalizable
set of features so the first data set is uh when our participants were claimed pac-man
the second data set is when our participants were answering an adaptive self-assessment test
so this test adapts the difficulty of questions based on how you have performed until now
then the third data set consists of the data when participants were solving five rebounding problems
and finally the last data set that we used was about an eye tracking game so the player was
controlling the avatar using their eyes and this dataset was kept uh separate
uh to do a final test of generalizable features so the first three were used to figure out the
generalizable features and with the training on one and testing one another and the fourth data
set was kept aside so that we can finally test the set of generalizable features from our exercises
a quick overview of which features we extracted so from facial uh features we
extracted how our muscles move uh on face so uh the action units and also the each frame of the
face shield video was passed to a vg 16 that gave us thousand features at the end from time series
data of physiological features we computed value and spectral histograms we also computed auto
regressive moving average but we then moved on to a little bit more advanced features that were
large so it's basically similar to rmr models but with the small difference that when our model
captured the mean of the time series garage models capture the variance of the time series
and also we captured information about the shape of the signal both in time and frequency domain
once we were done with the feature extraction basically we were done with the steps until
feature instruction in this figure we went two different ways one was feature selection with
linear lasso or non-linear random forest and the dimensionality reduction uh processes uh a simple
pc or a non-linear pcr once we had our training and testing uh results we what we wanted to come
up as generalizability index is how similar these two sets of results are and now for that we used a
uh user statistical test called annosim intuitively anusing works exactly like anova
but instead of analyzing the variance between two sets it analyzes the similarity between
the two sets so if the amusing gives us um significant results that means the two sets
are similar and this is what we wanted to see from the training and testing sets that i said before
and feature generalizability index is r in this case and the p value represents the uh
significance of uno sim and the nas have same data sets for training and testing so one five
and nine have same data for training and testing so it doesn't make sense to claim generalizability
of any sorts but when the data sets are different in training and testing stages then there is some
sense in claiming generalizability and here we can see that when we are training on one
industry or other there is no generalizability to be claimed but when we are training on more
than one data sets and testing on one then the sim turns out to be significant that means the
pipelines are generalizable and by extension the features used in those pipelines are generalizable
and finally we also show that if we use only the generalizable set of features we get a
very decent uh prediction of below ten percent error um with the dataset of eye tracking game
so what does all this mean basically so the first thing that we found is guard features were very
generalizable that means capturing the variance in the time series is important because the variance
as it connects with cognitive performance is generalizable across context uh what
we found also that emotions are very complex to specific because actions turned out to be
context specific as i said before that this is not transfer learning because
um the idea of transfer learning is to transfer the whole model from one context to other
and still the contexts have to be similar to each other in our case the data sets don't have to be
from similar context and we are transferring only the features actually the names of the features
and how to compute them with that note i will stop here and i hope you have questions for
me in the question in the session or you can write to me on this given thank you