Welcome!
My name is Vishnu, and I am a PhD student
at the Computer-Enabled Abilities Lab
at Columbia University.
Today, I will be presenting NavStick:
Making video games blind-accessible
via the ability to look around.
We all know that video games are
a pastime for many people,
and 3D adventure video games
(like the ones on your screen right now)
often feature large, fantastical worlds for
players to explore.
Yet, these game worlds are often inaccessible
to players with visual impairments,
and many existing games created specifically for
players with vision impairments
greatly simplify the experience for players,
making the game less fun.
We believe that providing access to the same
gaming experiences as sighted players is
important to providing visually-impaired players
with an enhanced sense of fun
and an increased sense of agency in gaming.
To this end, we introduce NavStick,
an audio-based tool for virtual environments
that allows visually-impaired players
to "look around" themselves within the game world.
NavStick's operation is simple:
Tilting the right stick on a game controller
in any direction
prompts the game to announce what lies in that
direction with respect to the player via line-of-sight
like this:
NavStick: Bananas.
NavStick: Cling wrap. (beep)
NavStick: Dark chocolate. (beep)
NavStick: Chocolate chip cookies.
NavStick: Butter. (beep)
NavStick: Croissants. (beep)
Vishnu: And as you may have noticed just now,
these announcements are made via spatialized (or 3D) sound
from the direction being queried.
You can think of NavStick as representing a circle or a pie
within which the player is at the center.
Any object that's within the player's line-of-sight
takes up some slice on this pie
with the size of the slice determined by how big
and how far the object is from the player.
We call this underlying data structure the "NavPie."
Tilting the stick then queries the slice that lies in
the direction that the stick is being tilted in.
We use the right stick specifically because
many video games use it to control the in-game camera,
and as the in-game camera isn't very useful for
visually-impaired players,
we instead repurpose it for NavStick.
We believe that granting players with visual impairments
the ability to look around as NavStick does
can make 3D games more blind-accessible,
and provide them with the same exciting experiences
that sighted players are so often afforded.
To evaluate NavStick,
we performed two fully-remote studies.
One to compare NavStick with the status quo means
of surveying an environment,
and another to explore NavStick's potential for use
within 3D adventure video games.
For Study 1, we wanted to compare NavStick
with the status quo means
of surveying an environment:
traversing a list of items.
Using a list to survey an environment
is very common in games created for
players with vision impairments.
So, we use this study to compare NavStick
with list-based surveying
via a menu-based surveying tool we created
called "NavMenu."
We compared both tools with each other
and explored each tool's advantages and disadvantages.
We put players within an aisle
of a virtual grocery store.
This aisle had eight distinct item slots
(one item per slot),
and participants used NavStick and NavMenu
to survey the eight items within the aisle
in order to answer some questions.
The goal was to evaluate NavStick and NavMenu
within the context of Darken and Sibert’s classification
of wayfinding tasks for virtual worlds
in order to determine how well both tools
perform for each task.
The currently active task dictated whether
we asked questions about
the content or the layout of the aisle,
or whether we asked them the question
before or after they surveyed it.
This setup gave us 5 distinct tasks.
We also added a very simple video game level
as our sixth task.
This level was a recreation of a level
from the classic blind-accessible game, Terraformers.
As for some of our notable findings:
We found that participants significantly felt
that they were able to form a better mental map
with NavStick than with NavMenu,
as we can see in this graph which shows
participants' average impression as to the quality
of their mental map.
We also found that participants had significantly
more fun in the simple video game task
with NavStick than with NavMenu,
as we can see in this graph which shows how fun
participants found each tool to use
in the simple video game task.
However, we also found that participants generally
did not prefer NavStick for every task.
As we can see in this graph which shows
more participants selected NavMenu as
their preferred tool of choice for non-directional tasks —
tasks that don’t require a sense of knowing
the physical direction of an item —
and many participants explicitly noted that NavMenu was
very good at simply communicating the presence of items
(that is, whether or not they're just there).
This suggests that NavStick and NavMenu
could complement each other,
and indeed participants agreed that they would
love to have access to both tools within a game,
so much so that one participant even refused to pick
a favorite tool at the very end for this reason.
So, those were some of our findings from Study 1.
Study 2 was an exploratory study in which
we explored NavStick’s potential in making
3D games made for sighted players more blind-accessible,
as well as the extent to which NavStick works
for complex situations found in modern 3D adventure games.
To do this, we created a 3D video game called
The Explorer, derived from Unity's 3D Game Kit.
Using The Explorer as our study’s testbed,
we had seven participants use NavStick to help them
traverse a 3D game world that we created,
avoiding obstructions and
defeating enemies as they desired using a laser gun.
The game world consisted of two large areas
comprising a total of eight segments, or shorter levels.
We designed the eight segments to possess
environmental characteristics found in existing 3D games
that could present a challenge for NavStick.
These were enemy movement,
time pressure,
and environmental occlusions.
We had the eight segments then feature
every possible on/off combination of these characteristics,
giving us eight combinations.
As for some of our notable findings from this study:
Every participant completed all levels of the game.
No one failed to complete a single level.
And we were also able to witness
how NavStick offered players with an
enhanced sense of agency.
This figure shows the paths that
participants took in Segment 8 of the game.
We can see how players employed different strategies
to get through the level and even roamed around
near the end of the level as they waited
for the door at the end to open.
Participants explicitly noted how NavStick
gave them an enhanced sense of agency.
However, we also found that the various
environmental characteristics that we integrated
into the levels caused participants some trouble.
This box plot shows levels of frustration
on 20-point scales as given by players who used
NavStick in a level with that characteristic.
We can see how each characteristic prompted
progressively higher levels of frustration in participants,
with levels featuring occlusion having
some of the highest frustration scores.
This points to future work which should focus on
improving NavStick to overcome these challenges
or even creating new tools to solve them.
So, our studies yielded several implications
for future blind-accessible games.
In particular, we see two main perceptual challenges.
The first deals with the fact that NavStick
only announces objects within a player's line-of-sight.
Yet, many participants desired some form of
"x-ray vision" — that is, being able to
see the world beyond their line of sight.
Sighted players usually have access to
a world map or a minimap that communicates information
about the greater world they are in.
So, how can one create an equivalent to these tools
for players with visual impairments to give
them more information about the world?
The other challenge deals with aesthetics.
Many participants wanted to know what
the overall world within The Explorer looked like
beyond merely knowing what objects it contained.
3D game worlds are often richly detailed and fantastical.
So, how can we go beyond merely communicating points
to communicating what the world actually looks like?
So in this project, we put forward a few contributions:
We introduce NavStick, an audio-based tool which allows
players with visual impairments to "look around"
within virtual environments to make 3D games
more blind-accessible.
We performed one study comparing NavStick
with menu-based surveying.
We found that NavStick has distinct advantages
over menu-based surveying
but that the two can co-exist.
We performed another study to assess NavStick's
potential within a 3D adventure video game
and find that it grants players an enhanced sense
of agency and fun not seen in current games
for visually-impaired players.
And we uncovered perceptual challenges that must be
overcome to make 3D adventure video games
even more blind-accessible.
So to wrap up, I would like to sincerely thank
my co-authors and everyone else who helped
for their work on the project.
And of course thank you for watching.