Hello, I'm going to present our work exploring smartphone
keyboard interactions for experience sampling method driven
probe generation.
My name is Surjya Ghosh, and I am currently working as a postdoctoral
researcher at CWI Amsterdam, and this is a joint work with
researchers from Indian Institute of Technology, Kharagpur
and Microsoft.
A smartphone keyboard interaction based emotion detection
has many applications, for example, in mental state tracking
or adaptive interface design.
But at the core of this value added services,
there is a supervised machine learning model for
emotion inference.
Now, to construct this model, we need emotion ground truth
labels, which are typically collected as a long term user
study known as experience sampling method.
But manual selfreport collection is time consuming and fatigue
inducing, so we always require intelligent probing strategies,
which will probe the user at Opportune moment so that there
is less fatigue.
So with this background, let us try to understand the research
question. So if we pay close attention to smartphone keyboard
typing, we find that there are two key facets.
One is the timing, that is how fast you are typing, how much
errors you are making.
And at the same time there is a rhythm in which you type,
so both of which translate to the time domain characteristics
and frequency domain characteristics.
So we'll leverage or we ask this question, Can we leverage
those two kind of characteristics to determine or design
intelligent probing strategies, which will probe at the Opportune
moments? So let us try to understand the keyboard interaction,
this probing scenario in the right hand side diagram, you
see that there is a text entry session from T1 to T2,
where the user is typing in whatsapp, and once he completes
typing, he is asked, How was his feeling by sending an ESM
probe? Now, while answering this probe here, he can mention
that whether the probing momentous Opportune or not, and from
the text entry sessions, we can extract several typing related
features, not text related features, but typing related features
like pressure and typing speed, et cetera.
And then we can combine those labels with the typing features,
which is to determine whether the probing is done at the
Opportune moment or not.
So now, as we all know that standard Google keyboard does
not provide this facility to track typing, we have developed
our own keyboard, which is similar to Android keyboard, and
it allows the user to type in the same way.
And we also collect the emotion self reports where the user
can ensure the the probing moment is Opportune or not, he
can mention just no response by saying selecting the no response
pattern, and it indicates that the probing moment was in
Opportune. We have ran a three week in-the-wild study involving
22 participants, and we have collected close to the 3500 sessions
and out of which 83% was Opportune and 17% was in Opportune.
Now, with this data of what we have done, we have represented
every session as a set of IDs, so a set of it a ITD indicates
the time interval between two consecutive typing events.
Now, after we represent every session as a set of ITD, we
extract set of time domain characteristics like session length,
session duration, et cetera, and we apply DfT or discrete
Fourier transform to convert the its into frequency domain.
From there, we extract four features, number of peaks and
three amplitude, the peak amplitude, one, two and three, and
then we compare.
Is there any difference between the between these time domain
parameters and frequency domain parameters when compared
for inopportune and opportune moment?
We see that for all the timing domain characteristics like
session Length, session duration, session speed, and error rate
vary significantly.
Similarly, the number of peaks and first peak amplitudes
and second and third peak amishold also vary significantly
between opportune and inopportune moments, so leveraging
those, we have developed a machine learning based model known
as Actually, we have developed multiple models.
One is TD, which is based on only time domain parameters,
FD, which is based on only frequency domain parameters, and
then the combined, which combines both the time domain and
frequency domain characteristics.
And we evaluated this model.
We obtain an average F score of 93%, and we have seen while
running the explainability analysis is that session length
and peak amplitude are the top-two discriminating factor
between in opportune and opportune moment.
So in summary, we can say that a smartphone keyboard interaction
pattern has both time domain and frequency domain characteristics,
and those vary significantly between the opportune and inopportune
probing moments.
And combining those, we can develop a machine learning based
model which will probe at the opportune moments and make
sure that the survey fatigue is reduced.
We have developed this model, and we obtained and average
accuracy of average F-score of 93%.
So these are some of the references that we have used in
our work.
We can refer to our papers for more detailed discussions.
And with this, I thank everyone.
And if you have any questions, we can take those.