Hi my name is Lisa Elkin I'm a PhD
student at the University of Washington
today I am presenting work I did during
my master's degree at the University of
Waterloo with jean-baptiste beau, Géry Casiez,
and my master's advisor Daniel Vogel
our paper is called manipulation
learning and recall with tangible pen
like input most pieces of software have
many different commands that can be
Selected sometimes these are in
hierarchical menus that can be
frustrating to navigate sometimes these
take up tons of precious screen real
estate particularly on tablets and
command selection and parameter
manipulation are completely decoupled
for example selecting a pen width and
drawing a pen stroke are two completely
different actions and so imagine if
instead of selecting commands like this
we selected commands like this here a
device is tapped with one edge to select
the color purple and then a thick line
is drawn with the corner colors change
to green and then medium dots are drawn
with a different corner the idea of this
device comes from Vogel and Casiez
work about a device called Conté they
drew inspiration from an artist Conté
crayon which is that stick of coal that
artists use where they can draw
different lines of different thicknesses
depending on how it contacts the paper
and so they asked what's the digital
analogue of this device and if you look
at its extruded rectangle shape if you
count up all the corners and all of the
edges and all of the sides you get 26
different contacts and so what if you
could create a device where you could
detect which contact was touching the
screen and that would mean that you
could map 26 different commands to each
of the 26 contacts so they built a
prototype that used infrared light and
diffuse illumination table and they use
computer vision to classify which
contact was touching the screen based on
its shape but because they used shape they
could only distinguish between contacts
that looked different which meant that they
couldn't tell which corner was touching
although they could tell that a corner
was touching and this meant that they
could only distinguish between ten
different contacts so their work left
some questions unanswered first of all
how can we actually build this 26
contact prototype that realizes the full
vision of this idea that they had they
claimed the device would be fast to
manipulate but this isn't tested and we
can make a prototype of many different
sizes
don't know how size impacts manipulation
time and finally probably the most
burning question is can anyone learn to
use this thing can people remember the
location of 26 different commands
located on spatial landmarks on the
outside of a 3d input device so our
first order of business was to make a
prototype that can actually distinguish
between 26 different contacts we wanted
it to work on any normal capacitive
touch display like an iPad or a
touchscreen laptop so we made a
conductive device and what that means is
that a touchscreen device can't tell the
difference between conte touching the
screen and your finger touching the
screen so a 3d printed a case out of PLA
plastic and we covered it in copper tape
to make it conductive now designing and
3d printing the end points were a little
bit trickier we wanted the corners to
feel pen-like but touch screens filter
out small touches so that random
conductive particles in the air don't
count as touches so if we made the
corners really pointy they would be
dismissed as noise so to overcome this
we 3d modeled air pockets into the corners
and we 3d printed them out of squishy
conductive material so they felt pen
like when they touch the screen the air
pockets allowed them to expand in
surface area
touch the screen with more surface area
so they wouldn't be dismissed as noise
then we put Hardware inside the device
we have an IMU that tells us the angular
position of the device and that is
attached to a microcontroller with Wi-Fi
capabilities so that those angles can be
sent over to a computer and then there's
a small lipo battery inside the device
so that is completely wireless now we
were so proud of our conductive Conté
prototype that we took it to the grocery
store to weigh it and we found its
weight as comparable to a standard
stylus which is 21 grams and Conté
weighs 25 grams so we have this IMU
data that's sent over from Conté to a
computer running software that we wrote
and that software has a decision tree
that uses the roll and the pitch of
Conté to determine which contact is
touching the screen then there's a
touchscreen device that can be the same
device size that computer but it doesn't
have to be and it sends touch events
over to the computer now just like we
had to make the corners touch the screen
with a large enough surface area that
they wouldn't be dismissed as noise we
had to make sure that none of the
contacts were so big that they would be
dismissed as a palm
so we have to make some interesting
design decisions with Conté and this
meant that we had to do some touch
smoothing and we needed to know which
contact was touching the screen in order
to do the touch smoothing properly so
the software that we wrote takes that
touch information takes the classified
contact does some filtering and packages
all packages it all together in Conté
messages which are down events up events
move events and air events and air events
just say the angle that Conté is that so
that the software can use that
information when it's not touching the
screen if it wants to and that's all
sent back to the touchscreen device so
that Conté applications can be
interactive so since Vogel and Casiez
were inspired by an artist crayon art
applications are sort of a natural use
case for Conté but we built a demo to
show probably a more widespread use
which is a PDF annotation application
so here Conte is used with one corner to
draw and red a different corner is used
to highlight and perhaps my favorite
part of the demo we are certifying that
this is a Conté demo by using one end to
stamp signature and one end to stamp a
date all right so we have our conductive
26 contact input device and now it's
time for experiments so throughout the
evolution of Conté there have been three
different sizes small size which is the
artists contact crayon the medium size
which is vocal and Casiez's prototype and
a large size which is our conductive
Conté prototype and so we wanted to test
the impact of device size on
manipulation time we 3d printed three
different mock-ups in our small medium
and large size and we painted each side
with a different color which we'll
explain later and we also had a pen
mock-up which we also painted different
sides with different colors as a
baseline comparison we use the term
manipulation to mean when a device is
rotated from touching the screen with
one contact to touching the screen with
another contact and back again
now if you count up all the possible
pairs of contacts it's 26 choose 2 which
is the hundreds of contacts and that's
not really realistic and real that's not
really realistic in any sort of
reasonable experiment time and so in
order to mimic real use where a corner
would be used as an inking corner we
designated one corner as start corner
and then all manipulations start from
start corner to a different contact and
back again and we use the term
manipulation corner
to describe the contact in a
manipulation that is not start corner so
we built the lightbox with some light
bulbs a camera at the bottom and a
plexiglass top and there was this
resistive touch sensor on top of the
plexiglass so the resistive touch sensor
allowed us to know when a mock-up was
touching the screen and we used the
current frame of the camera to classify
which contact was touching now remember
how we painted each side a different
color we measured the relative
proportions of each color in the frame
when the mock-up was touching the screen
to determine which contact was touching
the screen the user was shown two images
side by side representing each contact
in the manipulation from the user's
perspective and the active contact was
highlighted in green and that just meant
that was the contact the user was
supposed to touch the screen with next
we had 12 participants all right-handed
four sizes small medium large and pen
each participant used all four sizes and
size was counter balanced using a four
by four a Latin Square and then for each
participant for each size manipulation
contact was random and they
completed all manipulations that started
from start corner and one of the tasks
was for each size for each participant
they had to complete as many
manipulations as were required to
complete two consecutive error-free
manipulations per contact so our
hypothesis was that the smaller devices
would be faster to manipulate and this
made sense because contacts on the
smaller devices are closer together so
we were quite surprised when we found that
the largest device size is fastest so we
found a significant main effect of size
on log transform manipulation time and
post hoc tests found all pairs of sizes
to be significantly different from each
other with small being the slowest
then medium then large being the fastest
we also compared to a pen and a pen has
five different contacts the nib the
barrel the back the edge between the
Nib and the barrel and the edge between
the barrel on the back and we designated
the edge between the nib and the barrel
as start corner because that represents
a typical inking position so in order to
make a fair comparison between a pen and
Conte we doesn't we determined five
contacts on Conte that were similar to
the five contacts on a pen and then we
compared manipulation times between
pen and their corresponding contact
and we only found one pair to
actually be slower with Conte and so
this suggests that a pen isn't any
faster to manipulate than Conte and it
only has five contacts where Conte has
26 all right so now we're on to the most
burning question learning can anyone
learn to use this thing so we used
our conductive Conté prototype to test
this and it only had three visual
markings there is a white marking on one
end a white marking on the top and a
white marking on one side and this was
the smallest possible amount of visual
information that we could give
participants so that they could
distinguish different contacts different
edges and different sides from each
other but it gave them no extra
information and didn't tell them what
any of the contacts actually did because
Conte is a 3d spatial interface which
hasn't been studied we wanted to compare
against some baseline so we compared
against a 2d spatial interface we
we drew inspiration from Gutwin
et al.'s fast tap which is a grid based
Spatial interface in which commands are
located in grid cells and a grid cell is
tapped to pick the command and then in
it and then it is executed we also have
to have some way to teach people where
commands were and we again drew
inspiration from Gutwin et al. with
follow-up work that they did on fast tap
where they made a game to teach people
commands so we map 26 toy commands to
each of the 26 contacts on Conte and
each toy command had a corresponding
icon so the way the game works is an
icon would fall from the top of the
screen to the bottom of the screen and
the user had to execute the
corresponding command before it reached
the bottom now expert users who had
learned throughout this training process
where commands were could just execute
the command immediately
novice users were able to open a novice
guide after a short delay to find the
corresponding command and then they had
to close the novice guide and they could
execute it there was a small penalty in
the game to discourage people from using
the novice guide with the hopes that
they would become experts by the end of
the study the grid interface works
similarly commands were located in grid
cells and novice users could open a
novice guide which would show them where
the commands were after a short delay
and they could execute the command but
expert users could select the command
immediately and again there was a small
points penalty in
the game to encourage people to become
experts we use 26 commands on Conte we
use the same 26 commands on the grid
interface and we also wanted to test it a
simpler version with nine commands on
one end and we did this because we
thought it might be really difficult to
learn the location of 26 commands and
that we might get more meaningful
information from nine commands we had 32
participants six left-handed four
conditions Conte 26 Conte 9 Grid 26
Grid 9 um the study was between
subjects with 8 people per condition and
the task was there were 24 training
blocks where participants would play the
game blocks had different speeds which
we elaborate on in the paper and within
one block each command fell from the top
to the bottom in random order after the
24 training blocks there were three
retention blocks one zero minutes after
one ten minutes after and one 24 hours
after so we can see how many commands
participants remembered during a
retention block each icon matching a
command showed up in the middle of the
screen exactly once it didn't move and
participants had as much time as they
needed but the novice guide wasn't
allowed and they were given no feedback
about whether or not they selected the
right command so people can learn to use
Conté after zero minutes and after 10
minutes there was no significant effect
of condition on recall rate and all of
the recall rates were really high after
24 hours there was a significant main
effect of condition on recall rate the
post hoc tests found that only grid 9
and grid 26 were significantly different
now it's true Conte 26 didn't beat grid
26 but we never set out to do better we
thought the grid interface would be much
easier to learn than Conté and we just
wanted to measure how much easier it was
to learn so we think that the lack of
significant difference is really just
due to the fact that all participants
were able to learn all of the different
conditions and as you can see the recall
rates for all of them are really high
but we did do a power analysis to see
how much of an effect we could have
measured and we have enough statistical
power to have found a medium to large
effect if one existed so this says that
if there is a difference between Conté
and Grid it can only be a small
difference in the paper we also discuss
completion time expert usage rate and
error rate during the training blocks in
summary we created a conductive 26
contact Conté device found larger sizes
are faster to manipulate and people can
learn and recall command locations thank
you so much for watching and we hope you
read our paper