- Yeah.
So, very good afternoon to all.
So the title of the presentation is,
Does Emotion Influence the Use
of Auto-suggest during Smartphone Typing?
So, this is a joint work with
my supervisors at IIT Kharagpur
in collaboration with Georgia Southern University.
So, text-entry in small touch-based devices
like smartphone is relatively difficult.
The reason is there is limited space.
So, to facilitate typing in these environments,
different techniques like auto-complete,
auto-suggest are used.
However, there are two key challenges,
or two key issues with such techniques.
First, they occupy a fixed amount of space
in the keyboard layout.
So, what is happening is that,
so, in the keyboard layout,
a fixed amount of space is reserved
even though the user is not going to use it.
And second, that in order to use such facilities,
user needs to select and read the set of suggestions
shown to her, and then she needs to select
which words she is going to use.
So, these kind of increases the cognitive load
and disrupts the flow of typing.
So, as found in earlier studies.
So, if these suggestions are showed
only when the users are going to use them,
then the keyboard layout may be used more effectively.
So, this is the, but there can be different factors,
which can guide the auto-suggest usage.
For example, typing volume.
So, if the user is typing more and more,
then she may be more inclined to use the suggestions.
Similarly, a specific type of applications,
the user may be willing to use the auto-suggestions.
So, one key factor that is relatively ignored in,
or mostly overlooked in literature, is user emotion.
But, it is found that user emotion in generally
influences the typing behavior, from earlier studies.
So, in this paper, the key question that we ask,
Can auto-suggestion usage be determined
based on human emotion?
or does human emotion influence the auto-suggest usage?
So, this is the only question
that we are asking in this paper.
So, let's try to understand the auto-suggest scenario.
Suppose, a user starts typing at WhatsApp at t1,
and he continues to do so till t2.
So, here t1 to t2 is a session.
So, within this session, each black bar
denotes a key-pressing event.
And as the user is pressing the key,
the different suggestions are shown to him,
and he can select or reject the suggestion shown to him.
And at the end of these sessions,
the user can be probed
that what is his emotion during the session.
So, in order to model this kind of scenario,
we need to track three things.
First, we need to trace user's typing
and provide him the suggestions.
Second, we need to label whether the user
has used any auto-suggestions during the scenario.
So, we have defined it in this way that,
in a session, if a user has at least used one suggestions,
then it is marked as "accepted",
and if he uses no suggestions,
then it is marked as "skipped".
And similarly, we asked the user to
let us know what is his emotion during the session.
So, with this requirement to design this
experiment apparatus,
so it is a standard Q-W-E-R-T-Y keyboard.
So, in this keyboard, the suggestions as the user is,
so it's a custom keyboard,
and as the user is performing typing,
we show her the suggestions based on
an English dictionary,
and we also ask the emotion self-report.
So, we ask for four type of emotions,
happy, sad, stressed and relaxed.
So, this is based on the Circumplex model of emotion.
So, in this emotions, so we select each emotion
from different quadrants,
so that the user can self-report them easily.
And, now the question is that we have to build
the emotion or the auto-suggest usage prediction model.
So that we have constructed a personalized model
based on random forest,
and we have two classes to predict,
it is whether the user is going to accept the suggestion
or he is going to skip.
And we use only emotion related features
to build the model.
So, if you use three feature,
one is emotion associated with the current session,
the emotion associated with the previous session,
and the elapsed time between these two emotions.
So, these are the features that we have used.
So, since we are looking for the role of emotion
on the auto-suggest usage,
we have used only the emotion-related features.
So, with this, we conducted a three weeks study
in our university campus.
Approximately 20 students participated,
and we installed the app in their smartphone,
and asked them to use it for typing,
but we have got data from 13 participants,
because seven participant did not provide enough data.
So, we have discarded their data.
So, this is the final dataset
that we have observed,
or we observed approximately 3000 sessions
from all the users,
out of which 28% sessions are tad bit accepted sessions
and remaining are tad bit skipped session,
and this is the distribution of each users in which
they have accepted or skipped the suggestions.
So, now it asked the first question.
So, does auto-suggest usage vary across emotions?
So, we had four emotions,
happy, sad, stressed, relaxed.
And we checked what is the percentage of cases,
the user has accepted or rejected auto-suggestions.
And we see, in all the cases, the rejection is more,
but in case of sad and relaxed,
the amount of auto-suggest acceptance is higher.
We investigated what could be the possible reason for it,
and if we trace it back to the Circumplex model,
we find that sad and relaxed are kind of the state,
when the activity level is low.
So, the user is kind of lethargic,
or maybe he is low active state.
So, this is probably the reason that may have used
may influence more auto-suggest usage.
We also investigated the same for individual users,
so earlier result was based on the aggregated data,
so here we looked for individual user, what is happening.
So, if you look at this figure,
for every user, there are two bars.
So, first bar shows the distribution
of different emotions for accepted cases,
and second bar shows the distribution
of different emotions for skipped cases.
We compare if the distribution of the emotions
for accepted and skipped cases,
is significantly different or not.
And we find that for every user using chi-square test,
that the distribution of emotions
is significantly different.
So this motivates us that
using emotion, it may be possible
to determine the auto-suggest usage.
So with this we come to the evaluation.
We use a 10-fold cross validation
for evaluation purpose,
and we use the Metrics: A-U-C-R-O-C and F-score,
And we compare our method with two baselines.
The first baseline is the most-represented emotion.
So, in this emotion,
we observed that for most of the users,
there is one emotion state in which
the auto-suggest usage is highest.
So, probably whenever this emotion is treated,
we determine that this is the state,
when the user will use the auto-suggest.
And generalized is the case where we combine
the data from all the users,
and we use leave-one-participant-out cross validation,
to validate this model.
And then we investigate how accurate is our model,
and we find that, we obtained an average accuracy
of 73%, and it outperforms, and the proposed model
outperforms both the baselines.
And we checked what is the influence of each
of the emotion-related features that we have used,
and we find that, current emotion and previous emotions
are the most influential one.
They have very close information gain,
and the other one is also having
relatively less but also having significant influence.
And then we tried to investigate,
how our accuracy was 73%,
how we can improve it,
and we find that the one of the reasons
that our accuracy is low, is that we have
an imbalance in the accepted and skipped classes.
Because we have more skipped instances
rather than accepted classes.
So, what we do, we kind of balance
both accepted and skipped classes using
SMOTE,
the synthetic minority over-sampling technique.
And we observed that the performance improves significantly.
The accuracy ranges to 82%,
and both the F-score for accepted and skipped classes
also increases.
So, in summary we can say that the auto-suggest usage
is related with human emotion and users
are more likely to use emotions in low-activity status,
when they are sad or relaxed.
And based on emotions, auto-suggest usage
can be predicted with an accuracy of 82%.
So, with this I thank everyone,
and I thanks especially Microsoft Research,
LRN Foundation and IUI for supporting me the travel grant,
and IME member of Complex Network Research Group at India.
And, this is my webpage, the further details related
to this can be found here.
So, if you have any questions, I can answer.
(audience clap)
- [Organizer] We have a question over here.
- [Girl From Audience] Hi, can you go back
to the Circumplex model of affect?
So, my question is, Circumplex model is a
two-dimensional representation across
your arousal and your valence.
So, what is the reason for choosing
happy, stressed, sad and relaxed,
because they are not on extremes
like happy is lower on arousal,
and little to the normative side,
and little more extreme on the valence side,
and your sad is, like they are not proportional.
So, what is the reason for choosing these four?
- Okay.
So, the reason is that,
if you consider this an X-axis and Y-axis,
so this is the first quadrant, second, third and fourth.
So, there in this quadrant, the emotions lies
are having both high pleasure and high up.
So, we have selected such a way,
that from every quadrant, one emotion is represented.
So, happy is one of them, sad is one such,
stressed is one, and relaxed is one.
So, that is why we wanted to select one emotion
which is representative from each of these quadrant.
So, that is why we selected these four emotions.
- [Girl From Audience] My question was around like
how did you choose that happy was the representative of
first quadrant, because if I look at this representation,
excited is at more normative level to anxious.
So, my concern is that they are not at the same level
even with respect to each quadrant.
So, is there a reason you chose that,
like within each quadrant?
- No, actually, so if you look at this,
so, we have given happy or excited, right?
So, the reason we have given,
so if we give two emotions from the same quadrant,
it will be difficult to distinguish.
So, that is why we have selected one emotion,
and whether not astonished or happy,
that is our choice.
We based on that we have selected it.
There is no reason that,
but we feel that if we look at the
basic category, discrete category also,
the happy, sad, those are the predominantly
represented there also.
So, we have selected based on that.
- [Girl From Audience] The other question I had was around
the accepted and skipped.
- Okay.
- [Girl From Audience] So, your definition
of accepted is that
even if it's 1%, like if there's just one instance
of using suggestion, then you are considering it accepted.
- Right.
- [Girl From Audience] So, do you think you are masking
some information because the number of suggestions that you,
like someone who uses like 10 suggestions,
compared to someone who uses only one suggestions,
they are all being combined in one class.
- Yeah. So, it's a good question.
But the problem is that we have to set
some threshold, right?
So, whether it can be one, or five or 10, we do not know.
So, for simplicity, and we do not want to miss any single
instance that whether the user is skipped
the auto-suggestion usage.
We have selected one as the number. Okay.
- [Lady From Audience] Following up to the first question
that was asked about the choice of the labels, right?
coz your results are,
I don't know,
it's hard for me to explain why would someone use
auto-completion more when you are sad.
And in a sense because you forced the users to choose
among these four labels, you know you can imagine that
at some point, there was something
that was more related to their actual feeling,
they would choose it, but is not, I don't think
it's necessarily sound to say that you found
the connection between sad and auto-completion
without having a clear justification
of why do you think that emotion would be related
to the usage of this particular tool,
coz if you use this model, you could also just imagine
that to get more into what are possible causes of effective
states, that would influence usage of this tool,
you might even just simply give, you know,
the ratings of activation and pleasure
that users can select so you avoid putting labels
on the states, that are somehow fictitious, right,
and then maybe you can try to come up with
could be that it's really activation,
that might have the influence,
whether you're kind of in a hurry or overloaded
or that's when you tend to use it or not,
depending on how well it works for you, right?
But, if you wanna really get into some sort
of a causal relationship between states and usage,
you might wanna find something that is a little-bit
less contrived than these labels.
- Yeah, very well, good, very well suggestion.
So, the causality analysis
that probably you are trying to point out,
so, we have not done that,
and here also we say that it maybe the reason
that in low-activity level, users prefer to use
the auto-suggestions more.
But yes, the usage of the causality analysis
will probably reveal the actual picture,
why in these emotions,
the users are using more auto suggestions--
- [Lady From Audience] Yeah, you always have to confirm
whether your labels are appropriate for this task, right?
As opposed to if you have just using the dimensions.
- Right, right.
- [Lady from Audience] Your voice making forcing
a word into the user's head.
- Yeah, but. Yeah, so.
- [Man From Audience] Hi, I am (mumbles)
from Cambridge University.
I kind of have a question that approximately follows
the previous two because
without being an expert in affect,
but wouldn't the intensity
of an emotion can strongly influence
whether they can
use or not use it to correct.
I can see how if you're particularly sad,
you'd just completely not text at all,
or if you're particularly happy,
you'll take your phone out and message your friends.
But the difference between little bit sad
and little bit happy, already,
given the fact that
all this discussion about
which one of the quadrant,
emotions we will be having.
I'm just wondering
you're getting somewhere,
but to what extent is not
representative of how
people go through their everyday lives,
where their emotions change a lot,
and have like some elements of
sometimes of high sadness,
sometimes of high happiness,
but most of the time,
they're somewhere in the middle,
somewhere neutral.
- Yeah. So, that is also a good suggestions,
but the problem is that
if we try to collect the intensity of the
different emotions, right?
So, we've tried that experiment as well,
but what we've found that most of the time
the intensities do not vary much.
So, most of the times, they're kind of clustered
in some regions.
So that is not that in the daily practice,
so these are primarily the university students,
that at some point of times they will give rating
like -1, -10 and at the other point of time,
they will give ratings at 10.
So, there is not a huge variation in this spectrum.
So, that is why probably we can select these
kind of model. But, as you have suggested,
so maybe granularity if we increase,
so this may give better results.
So that can be possible.
- Okay. So--