- Alright. Thanks for the introduction.
Alright, I'll start talk by giving a simple explanation
of what is meant by swipe error.
Alright, have you ever come across
this situation in general?
A swipe error can be as simple as swiping
to the direction that was not meant to be.
Depends on the context of interaction,
swipe error can be tolerated,
but however in some cases it may cause an annoyance
and sometime embarrass...
Why'd it do that?
- [Man In Audience] What?
- Yeah, probably can't.
It's the HDMI?
Yep, yeah, this is a...
Right, sorry for that technical issue.
Alright, so in general, swipe error can be as simple
as swiping to the direction that was not meant to be.
Depends on the context of the interaction,
Swipe error can be tolerated,
but in some cases might cause an annoyance
and sometime embarrassment,
for instance in Tinder application
where every swipe matters.
Alright, in Tinder, swiping to the wrong direction
may result in liking or disliking someone unintentionally,
and this is a common mistake when using Tinder app.
Alright, people at Tinder acknowledged this
and has come out with the Tinder+
which has premium feature which allow users
to undo their swipe in the event of a mistake.
Right.
Swiping are a common touch gesture
to interact with touchscreen.
Despite of its simplicity, users still make swipe mistake.
User are likely to make swipe mistake
when they are in attention-demanding situations,
such as reading, or replying text message,
or when they are playing game on the mobile device.
So why are we interested in detecting swipe mistake?
If swipe mistake can be detected,
it can be used to support graceful recovery from error.
However, this requires timely detection of an error signal,
come from sensor measurement
that correlates strongly with swipe error.
In BCI most of the effort to detect and recognize
interaction error were achieved
through brain-computer interface.
So, through this technique,
when user commit a mistake during interaction
a specific signal called error related negativity
will appear in the brain about 100 millisecond
to 300 millisecond after the mistake.
So researcher have used this signal
to detect interaction error.
Whilst error detection using brain-computer
interface is intuitive,
this technique actually require
additional peripheral to be worn by user.
For example, this one.
Wearing a headset like this may not
be practical for mobile devices.
And perhaps not like this one.
Alright, in this work,
our hypothesis is that when error is made on the touchscreen
there is a physical changes in the hand grip
and movement of the device.
So these subconscious, subtle flinches
can be detected using contact and inertial sensors.
In order to detect this error,
we apply machine learning techniques on the sensor data.
So therefore our detection technique
consist of an input for mobile device sensor
which are then passed to the machine learning algorithm,
and we derive a swipe error model
which is then used to make swipe error prediction
on new and unseen mobile sensor input.
Here, we used two sensor input,
hand grip and hand movement data.
So in this work we are particularly interested
to find out if these two inputs,
namely hand grip and hand movement,
can be used to detect swipe error.
We also are interested to find if
this technique can be generalized across
user and across swiping task.
Given the two input, we are also interested
to see if detection can be improved
by commanding both inputs.
So, to answer the research question
that I just mentioned just now,
we ran a couple of experiments.
To do so, we designed swiping task based on
three classic cognitive tests,
namely Flanker, STROOP and SART.
So in this swiping task, the goal is
to swipe and respond to the simulation on the screen.
For Flanker, the swipe direction must follow
what is being shown on the screen,
but for STROOP and SART the task is
to swipe to the right if correct stimulus is shown
and swipe to the left if otherwise.
We also used another response,
which is hold response where
when we show unrelated stimulus on the screen.
We gave feedback in the form
of correct message after every swipe.
The goal of the experiment to induce swipe mistake
in the most realistic manner possible.
So here are the example of the stimulus
that we used in our experiment
for Flanker, STROOP, and SART.
For Flanker, the goal is to swipe
according to the middle error surrounded
by the rest of the errors.
For STROOP the goal is to swipe to the right
if the word matches the color,
or swipe to the left if otherwise.
And for SART the goal is to swipe to the right
if the number appears in this series
or swipe left if otherwise.
So each swipe trial start with a two second
waiting time followed by the swipe itself.
Here we increased and decreased the swiping time
based on the number of previous
correct and incorrect response made by a user.
Respond feedback is shown for about two second
after swipe before proceeding to the next swipe trial.
Alright, in order to collect hand grip
and hand movement data we used two Nokia N9s,
which has been modified to include back-of-device
capacitive sensor and movement...
Movement is captured through built-in accelerometer.
We recruited 20 participants which
18 of them are right handed,
and two of them are left handed.
We ran the experiment in three session
in which each session we collected 30 swipe trials.
So therefore we had 270 swipe for each participant.
We recorded time-stamps, accelerations,
and backward device capacitive reading from each swipe.
So here is the N9 prototype that we used in this work.
Shown on the left, there is a 20 capacitive sensor
lit on the flexible PCV across the back
and inside of the phone.
Middle image is the back of the N9 prototype,
where we attached the capacitive sensing.
And we used silicone cases during the experiment
to protect the sensor from dirt and grime
from affecting the sensor readings.
So for each swipe we pull out a segment
of interest from signal time series
that is one second before and after response feedback.
And then we normalize this segment
to eliminate any long term trend.
And from the normalized sensory
we extracted two types of features,
sensor total magnitude which act...
act as a proxy
to grip and hand movement strength,
and raw multi sensor readings.
Alright, shown here is the magnitude
of back-of-device sensor for four consecutive STROOP task
for one participant.
As you can see here the total magnitude increased gradually
as we show (mumbles) to the user.
We also see here that the data spike in total magnitude
indicating that there are physical changes
in the hand grip around this time.
Alright, shown here is the average
of swipe attempts from all participants
for both sensor between one second
before and after the feedback.
So the vertical dash line is the feedback time
and the red and blue lines denote the correct
and incorrect swipe made by the participants.
So as we can see here the difference between
correct and incorrect swipe are quite clear
and in STROOP task and for back-of-device data.
It's also clear that the differences appear
at about different time for back-of-device
and accelerometer data.
Alright, from the collected, sorted data
the goal now is to distinguish between
incorrect and correct swipe response.
So here we use Random Forest classifier
to detect swipe errors.
So to do so, we trained a classifier with swipe collected
on the first and second sessions
and test it with the data collected on the third sessions.
So we evaluated our classification performance
at different time after the feedback
to see how the performance varies with the time.
So we used four performance metrics,
the Area Under ROC curve, the accuracy,
the False-Accept rate, and False-Reject rate.
So we begin by showing the classification result
of using individual model,
that is we trained a classifier using participant data
from the first and the second session
and test it using data from the third session.
Alright, shown here is the average from all participants
classification AUC at different time after the feedback.
Purple lines are the AUC for back-of-device data
and the green lines are the AUC for hand movement data.
The red lines are the base line AUC
which is the chance of getting swipes correctly classified.
So in this work we are particularly interested
at the performance achieved at about 200ms
because we believe it is possible
for user to make any explicit correction within this time,
which makes the detection of the swipe error useful.
Also we can see here that the performance
does not vary that much with time.
Alright, here is the average classification,
result is in pooled model for all participants.
To build the pooled we removed data from a particular user
and we randomly selected swipe data
from the rest of the participants
and test it against the data removed
from the particular user.
As you can see here,
the AUC, Accuracy, and False-Accept rate
of the pooled model for both hand grip
and hand movement data give better performance compared
to using just the individual model.
However, the False-Reject rate for pooled model
is much higher than the individual model.
Next is the average classification AUC
from using model constructed using both data.
In both individual and pooled settings,
we can see that there is an improvement
when combining both data.
This suggests that the accelerometer
and back-of-device sensor contribute
independent information about the error signal.
And here is the average classification AUC
of training on specific model
and testing it with mismatched task.
So for instance, training on Flanker
and testing on the SART,
or training on the STROOP
and testing on the SART as well.
So this is to show how well we can
generalize this standing across tasks.
As can be seen on the table, the performance
is very good for accelerometer data
where we can see here AUC is approximately
between 0.8 to 0.9.
However this is weaker for the back-of-device data.
And interestingly, training with multitask model
where we combine Flanker, STROOP
and SART together improve the test performance
over training on just the matching task.
This suggest that swipe error are
generated across these task.
So from all the result we ask these question,
can we classify swipe errors automatically?
Can we classify swipe errors early?
Yeah, we can do that about 200 millisecond.
Can we build classifiers that do no require
individually trained model?
As I showed just now, the pooled model
outperformed the performance from using
just the individual model.
So we also can improve the performance
by combining both back-of-device sensor
and accelerometer data.
And, lastly, can we generalize the swipe error?
Well, from the result I showed you previously,
by combining all model that we built
from Flanker, SART, and STROOP,
we can get better performance compared
to just using the match task performance.
Alright, to summary, using our technique,
we showed that swipe errors can be automatically
detected using hand grip and movement data
as early as 200ms after feedback.
We also showed that the swipe error patterns
are not user and task specific,
however it should note that this study was done
in a controlled lab setting using swipe data
from a small population,
and finally...
and finally we show that using both input trained model,
we can get better swipe error detection performance.
Thank you.
(audience applauses)
- [Man In Audience] And without further,
three minutes for questions.
Please line up to the mic.
Meanwhile, please can I ask speaker to test equipment?
I think I will start with one quick question for you.
- All right.
- When you tried to analyze the difference,
more yes to (mumbles) swipes?
- Come again?
- For instance, when you analyzed
your obligation to pinch gesture?
- Well, that's a good question.
Currently, we just looking at
the simple swipe task at the moment,
but we also might look into different
gesture like you said, pinching or dragging,
and also probably scrolling.
So we already have the data for that,
but at the moment we just trying to keep the,
trying to model a simple basic touchscreen gesture
at the moment.
So that's probably interesting things to do in the future.
- You have another questioner, by the way.
- [Woman In Audience] Hi.
- Hi.
- [Woman In Audience] I'm Paulina
from University of Notre Dame.
Yeah, this is very interesting.
Again, I wanted to know how you would incorporate this,
like after you detect the swipe pattern,
how would you or in what ways would you
incorporate that information?
Wouldn't a simple,
if there is a provision for a user
to undo whatever they've done, like data,
would that be sufficient
or would this give an added advantage over?
- So you meant by--
- [Paulina] You mentioned that Tinder app, right?
So you said now that they have an undo button.
You think detecting the swipe pattern can
give an advantage that the undo button cannot?
Or would it be better than that, or?
Like not just put Tinder--
- Yeah, yeah, I understand that.
Okay, in our results we can make a good prediction
whether error has been made at about 200 millisecond,
so that is way, way before the user acknowledge
that the error has been made.
So for Tinder for example they have the undo button
that is need to be clicked manually after the user
realized that an error has been made.
So in our technique the focus is to detect
automatically rather than initiate the user
to undo the task, I mean the swipe,
or whatever error that has been made.
Does that answer your question?
- [Paulina] Yeah, thank you.
- Alright, thanks.
- We have time for one quick question.
- [Man In Audience] The examples you've shown,
there's a clear right and wrong answer in the experiment.
But in the Tinder app, there isn't a right or wrong answer,
it's either you like or dislike someone.
Do you feel a model like that--
- Well, that is a good question
because our experiment is based
purely on the cognitive error rather than
the Tinder error, specific error.
So that's going to be interesting if we can
do experiment on the Tinder app itself
where we try to correlate what kind of signal
that could be produced from the grip
or movement data when Tinder error's been made.
So I think that that is going to be like a next step,
when we focus on the specific application.