- It worked, I'm not sure what's going on,
why it didn't work.
Oh, okay, I'm gonna have to have a two setup thing here,
with notes here, presentation here.
We're gonna make the best of it, so let's just start.
My name is Martez Mott.
I am a PhD student in the Information School
at the University of Washington,
and I am presenting on behalf of my coauthors,
Radu-Daniel Vatavu, Shaun Kane, and Jacob Wobbrock,
and the title of our paper is
Smart Touch: Improving Touch Accuracy
for People with Motor Impairments with Template Matching.
So touch is one of the most dominant ways
users interact with modern computing devices,
and the predominance of touch can really be attributed
to the proliferation of touch-enabled devices,
such as smartphones and tablets.
So many of us here have smartphones and tablets.
Some of you right now are on your smartphones and tablets.
That's okay, I won't hold it against you.
But there's also these other touch-enabled devices
that we interact with on a semi-regular basis.
So, perhaps some of you checked in to your flight
on your way to Chi using a touch-enabled kiosk
at a airport, and perhaps some of you use
touch-enabled devices at your local grocery store
if you go to the self check-out lanes.
But for many people with motor impairment, however,
touch-enabled devices are inaccessible and inoperable.
So for people with motor impairing disabilities,
such as cerebral palsy or muscular dystrophy,
touch-enabled devices just seem not to work very well.
The problem with touch-enabled devices
doesn't lie with the user, however.
It lies with the implicit assumptions
touch screens make about users and their abilities.
For example, if a user wants to touch a target,
touch screens presume users can suspend an arm,
extend a finger, and land and lift
within the bounds of their target.
But for many people with motor impairments,
extending a single finger can be difficult or impossible.
So instead, they may touch with
the back or side of their hand, with multiple fingers,
or with their knuckles, and, as a result,
the system can't infer this type of touch input.
And, in addition to that, it's also difficult
for many users with motor impairments
to land and lift within the bound of the target
because that requires a great deal of fine motor control.
So our goal is to allow users to touch
in however their abilities allow,
and for the system to correctly infer
the users' intended touch locations.
To accomplish this goal,
we took an ability-based design approach
to touch screen accessibility.
We did that by doing two things.
First, we did an exploratory study
of the touch behavior of people with motor impairments.
And then, using that information, we created and evaluated
a novel template-matching algorithm
to improve touch accuracy for people with motor impairments
that we call Smart Touch.
So the first thing I'm gonna describe
is an exploratory study of touch behavior
of people with motor impairments.
So we did a lab study where we recruited 10 people
with motor impairments.
They came in the lab and
did a couple of different touch trials with us.
And all of our participants had
some motor impairing disability
that impacted their ability to control
either their fingers, their hand, or their arm.
So, for the study procedure, we used
the Microsoft interactive table, the PixelSense,
and we presented a crosshairs to the users,
and we asked our participants
to touch the crosshairs however they wanted.
From our study, we collected all this touch data.
We had a couple of different measures of interest,
some things that we were really interested to see,
so that's what I'm about to describe to you now.
The first is concurrent touches.
Typically, when a person touches a touch screen,
a single finger touches the screen,
and a contact point is registered.
And the centroid of that contact point
is taken as the user's intended touch location.
However, if you're a person with a motor impairment,
and you touch with the side or the back of your hand,
multiple concurrent touches get registered.
That's represented by the semi-transparent blue ellipses
that you see on the screen here.
So then the question becomes, of all of these touch points,
which of these touch points represent
the user's intended touch location?
We were interested to know, on average,
how many concurrent touches get generated
by the touch behavior of our participants?
And the answer is, there's a lot.
You can see here, from this box plot,
the distribution of the number of concurrent touches
generated by our participants.
And you can see that there's quite a variety here, right?
So you can see, for participant four,
they were the only one who actually touched in such a way
that only produced one concurrent touch at a given time.
And compare that to participant 10,
whose range went somewhere
between five and eight concurrent touches per trial.
The next measure of interest was land-on distance.
The land-on distance is the distance between
the centroid of the first touch
to the center of the crosshairs for each trial.
And, for that, we were also interested to see,
on average, what was average land-on distance
that our participants were able to achieve
on the touch screen.
And we can see here that there's quite a big spread here.
Note that the distance to target here
is measured in centimeters, not millimeters.
So you'll also see a huge range of abilities here.
So, for example, participant one, their land-on distances
were somewhere between five and around 45 centimeters,
and we're back, okay.
(audience laughs)
So. (Martez laughs)
Compare that to, this is funny,
compare that to participant four,
whose land-on distance spread was much shorter.
So similarly, we looked at lift-off distance.
The lift-off distance is the distance
between the centroid of last touch
to the center of the crosshair.
And you can see here,
there results are very similar to land-on.
We see quite a big spread here,
very large average lift-off distances.
So the key findings here
is that the mean land-on and lift-off distances
were much higher than those reported for non-disabled users.
And this is a problem
'cause remember, for a user to acquire a target,
they have to both land and lift
within the bounds of the target,
and so, from the numbers we just saw,
those ranges are much too big to accommodate the size
of current touch screen targets.
And the second thing we observed
was that touching with various parts of the hands
create varied multi-point touches.
Right, so that's the problem.
The system can't infer where the user's trying to touch.
So the key challenge is,
we must consider the users' entire touch process.
We know from our data that we can't rely solely
on the initial contact point or the last contact
that the user makes with the screen
to infer where the user's intending to touch.
And the second key challenge is,
how do we take all of these multiple concurrent touches
and from that, be able to,
how can we map multiple touch contact areas
to a user's intended x,y target location?
To do that, we developed a new approach we call Smart Touch.
Smart Touch is a template-matching approach
to improve touch accuracy.
And what I'm gonna explain to you now
were the steps involved in our Smart Touch algorithm.
The first step is pose extraction.
We have all of this touch data, the idea is that
we wanna extract only the most relevant touch data.
We don't want everything.
We just want what we consider to be the most relevant.
So first thing we do is we segment
the touch process into frames.
The touch process is every touch that occurs
from touch-down to lift-off,
and a new frame is created every time a touch is added,
every time a registered touch is updated,
lost, things like that.
Then the next thing we need to do is,
so, from our study, we realized that
people would tend to dwell,
when they were getting near their target.
So we decided that we needed this notion of stability.
That's the time where we believe users felt like
they were actually acquiring their target.
So we created two stability scores,
movement stability and shape stability.
We go through each frame of the touch process,
and we label each as either movement of shape stability,
either movement stable or shape stable,
and then we look for overlap.
So which frames are both movement and shape stable?
And then, from that, we take the touch data
from the frame that's both movement and shape stable
and has the longest lifespan.
Next I'm gonna talk about the pose-matching process.
The idea here is that,
(laughs)
so, that's okay.
Thanks for that.
Okay.
All right.
The idea here is that
we wanna be able to compare
touches that occurred at run time
to previously observed touches
that we would collect through, for example,
like through a training phase.
To compare the touch that occurred at run time
to previously observed touches,
we have to normalize them so that they can be comparable.
So we do a simple, two-step normalization process.
We just put a bounding box and translate
the top-left corner of the bounding box to the origin.
And then, to actually do the matching process,
we extended the functionality of the $P recognizer.
So the $P recognizer is a point cloud recognizer
that matches points to other points.
But, since we have ellipses,
$P can't handle can't handle
the additional properties of ellipses,
so the additional properties being area,
being orientation, so forth.
So there's more detail in the paper
about the pose-matching process,
but essentially, what we do is we line up all the templates,
and we go through and step through, and we compare each one
until we find which template is the best match.
So that's the template with the lowest distance score.
This template is most closely resembles
the pose that we just extracted from the touch data.
Now, I've explained to you that we've
extracted this touch data,
we've lined up some ellipses.
But how do we actually predict
the user's intended touch point?
We take the best best matched template,
and since these were created during the training phase,
we know where the user's intended target was
when this template was created.
So we know the offset from the weighted centroid
of the best match template,
and what we do is we take that offset,
we apply it to the weighted centroid
of the pose that was created at run time,
and that gives us our predicted touch location.
Just to recap, Smart Touch is a three-step process.
First, we have to extract the most relevant touch data,
and then we go through a pose-matching process,
and then, finally, using data collected
from the best match template from the pose-matching process,
we actually do our touch-point prediction.
So finally, I'm gonna talk about
the evaluation of Smart Touch.
We performed an evaluation of Smart Touch
using the touch data collected from our 10 participants
that were inside of our exploratory study of touch behavior.
First thing we have to ask ourselves
is which number of templates is Smart Touch most effective?
So we did an analysis of that,
and we discovered that at 30 templates,
we were able to achieve the best performance.
So that's what we used.
And for our main analysis, we asked ourselves,
how accurate is Smart Touch compared to land-on or lift-off?
In terms of accuracy, we look at the offset distance
between the land-on and lift-off points to the crosshairs,
and the distance between the predicted Smart Touch point
to the center of the crosshairs.
In our evaluation here, you can see,
so things kinda got a little messed up here,
we found that Smart Touch was significantly more accurate
than both land-on and lift-off,
and you can see here, that
for, yeah, those got messed up.
So that's supposed to be over participant one, six, and 10.
You can see here that, for those participants,
Smart Touch was able to provide
a huge increase in performance.
And,
even with participants in which the distances
between land-on and lift-off were much closer,
as, for example, as you can see
in participants four and seven here,
Smart Touch was still able to provide some benefit.
So, just one part for discussion here,
it's important to note that, even with Smart Touch,
these error offsets are still in centimeters,
so we're not to the accuracy points
that we wanna get to quite yet.
But we're gettin' there,
and that's what our continued research is gonna work on.
We hope to eliminate any performance differences
when it comes to touch accuracy
between people with motor impairments
and between non-disabled users.
So for our future work,
we also wanna look at improving the accessibility
of mobile touch-enable devices,
so there's gonna be some differences
between the touch behaviors compared to a table top,
compared to a mobile device, like a smartphone or tablet.
We wanna be able to find those differences
and design to eliminate them.
And then we're also interested in improving
touch interaction for users with situational impairments.
So if you're on your smartphone and you're walking,
we know that you're less accurate,
so we also wanna see if we can provide benefits
for people in those situations as well.
So in our conclusion,
I talked about a study that,
like an exploratory study of touch behavior
of 10 people with motor impairments,
and I also described the creation
and evaluation of Smart Touch.
Thank you.
(audience applauds)
I'm pretty much done, right?
- Thank you very much.
I think we don't have time for questions,
but all the coauthor was be available in the coffee break,
or, so thank.