hi everybody thanks for coming to my
talk today i'm johan wenzel i'm a phd
student at the university of waterloo
and i'm here presenting understanding
how people with limited mobility use
multimodal input which is a project i
completed as part of an internship with
microsoft i finished this project with
the help of my co-authors sasha
yanuzovich james devine john porter and
martes mott
so mostly as a commentary on my own
attention span for presentations i'm
going to summarize our findings here
so first we find that when people with
limited mobility encounter accessibility
barriers one of the most common remedies
is to combine input devices
second we find that these input
combinations are strongly based on
individual mobility needs with a wide
range of combinations to suit a wide
range of people
these accessibility barriers are
generally caused by mismatches between
the user and the device or between the
device and the application
when people with limited mobility are
looking for ways to combine devices they
often look to social media or word of
mouth recommendations
and finally we make recommendations for
increasing accessibility specifically by
adding tools to discover and configure
input combinations
so i'm going to start things up with a
quick primer on degrees of freedom and
multimodality think about the different
input scenarios that you encounter every
day you might browse the internet mess
around the 3d models play a game or even
combinations of these at the same time
each of these situations takes in user
input at a different level of complexity
and as such they demand input with
increasing degrees of freedom
in this context degrees of freedom
refers to the domain of input provided
by each different device for example a
keyboard with 101 keys can provide 101
sources of binary switch input the key
is pressed or it's released a mouse
provides data for its x and y motion on
your desk as well as click events and up
and down scroll events and any other
buttons on a fancy gaming mouse
likewise a gamepad has a complex
combination of buttons triggers binary
or analog and joysticks
now think about controlling an
application like unity it's a complex
program that involves input in three
dimensions mouse interaction keyboard
movement often at the same time the more
complex your interaction is the more
degrees of freedom of input it expects
but what if users can't provide that
input with the devices they have in the
fidelity the application expects for
example consider a user that has trouble
holding and using a controller but in
their normal computer setup they use a
keyboard a switch setup and a mouse to
provide that same level of input for
several games and applications why not
approximate or recreate that input using
devices that users already have this is
what's called multimodal inputs
combining devices with multiple degrees
of freedom into one cohesive form of
input
so the overall motivation for this
project is how do people with
accessibility needs utilize
multi-modality and multimodal input in
their everyday computer usage and how
can designers use it to make more
accessible interfaces this has a few sub
questions which we investigate in the
paper
namely which devices do people with
limited mobility use and how are they
combined
next
how are these choices affected by
context like application type hardware
or other factors
and finally how can developers and
designers accommodate the wide variety
of existing setups
we surveyed 43 people with mobility
limitations to look at their preferred
devices and input configurations first
we ask respondents which categories of
computing device they own
so this includes smartphones desktop
computers laptop computers things like
that
next we ask them to list all the input
devices they use with these computing
devices
so this includes stuff like mouse voice
keyboard touch screen note that in this
categorization we split out controller
and voice because there are special
categories of input that kind of don't
really fall into the categories of
discrete and continuous
so after the surveys we decided to get a
deeper insight into the context and the
conditions that affect how people create
these multimodal input configurations
so we found that most people preferred
slower paced games as they were
generally more accommodating for
different styles of use and as well here
are some notable quotes from the
interviews
i'd rather find a game i can already
play than try to buy special hardware
for just one game
sometimes finding a game that works is
luck of the draw there's often no
preview and no way to know if i can
change the controls
vr's lack of accessibility makes me feel
disabled all over again
and i don't usually play in groups
because i usually don't understand why
i'm moving slow
the issues and remedies associated with
combining input devices can be broken
down into a two-step matching process
between the user and the device and then
the device and the application
so in the mismatches between the user
and the device people often employed
alternate usage styles which include
conventional usage with alternate
control schemes adapted grip which is a
different usage posture so say changing
how you hold or use the device adapted
device which is a hardware modification
or alternate device which is swapping
the input device for another entirely so
say swapping an xbox controller for a
quad stick
and then remedying device application
mismatches people use two things
virtualization which is emulating a more
compatible device so say using a
software keyboard will display it on a
screen or remapping which is changing
input functionality entirely so say
tilting a joystick to the left to
emulate left click
but one of the biggest things we found
in the interviews was that people didn't
create multi-device configurations out
of nowhere in most cases they sought
outside help from either people that
they knew recreational therapists or
most commonly social media like youtube
or reddit
as such the third part of our
investigation aimed to simulate that
process of searching social media we
search youtube for all combinations of
60 accessibility related keywords and
eight gaming related keywords like xbox
playstation things like that after
establishing a data set and narrowing it
down using preset inclusion and
exclusion criteria we ended up with 74
videos from 66 unique youtube channels
we categorized the videos based on usage
style platform number of input devices
user position and device position and
then finally we used cohen's kappa as a
measure for integrator agreement in this
analysis
i'm not going to list it all here but
our main findings are that most videos
showed one device being used but the
vast majority of those one device videos
showed users playing on a game console
videos showing console gamers had a
bigger variety in usage styles but the
videos showing pc gamers showed more
alternate devices being used videos of
console gamers showed them using
multiple body parts more often
in terms of the lesson learned the
differences in device compatibility
between console and pc games affect how
users engage with these games from body
position to device choice
so there are three primary takeaways
from this analysis
in terms of general support designers
need to design their apps to accommodate
a wide variety of hardware and
combinations especially including
options for remapping like you see here
which is the steam layout screen
next discovery tools to help users
discover and implement multi-device
configurations will make a significant
difference to a system's usability and
finally many users reported using
virtual devices like a software keyboard
to supplement real ones researchers need
to consider virtual devices as a
meaningful part of future device spaces
so that brings us to the end of the
presentation and now i'd love to take
any questions you might have otherwise
feel free to reach out via twitter or
email thank you