hi there my name is chris creed from the
dmt lab at birmingham city university
in the uk and thanks very much for your
interest in our work around multimodal
gaze interaction
for creative design so there's been a
lot of work around eye gaze interaction
over the past
several decades and in particular has
been lots of research around
eye typing selecting small targets which
is a known challenge in this field
due to some of the the limitations in
accuracy of
eye trackers and also work around
document navigation and
other other areas too but one field
where there's been a lack of work is
around
creative visual design work using gaze
interaction
and this can actually result in some
user groups being excluded
for example people with physical
impairments who might not have the
option to use a mouse and keyboard for
creative work
if they're reliant on gaze interaction
as their their primary input approach
and we don't have accessible interfaces
to support them
they can essentially be excluded from
this this type of
of work and also just incorporating gaze
interaction into existing
design commercial design applications is
unlikely to be an optimal approach
and just an example of that i've got a
screenshot here of adobe xd
and and we can see there are lots of
small targets essentially within the the
interface so for example on the left
we've got all the
icons for different tools we've got the
controls on the right
that's fine they're they're
straightforward to select using a
traditional input device like a mouse
but if we try and use our eyes to select
these tools it becomes much more
challenging so we've been focusing very
much on how can we make design tools
more accessible
for people who are using gaze
interaction as their primary method of
input and to explore that we've
developed a new application called
Sakura which is a multi-modal
gaze design application and by
multi-modal i mean that there's a
combination of using gaze for pointing
at targets within the the application
and then a mechanical switch is used for
performing selections
and there's a screenshot of the
interface here so and and this was
informed
by the design of commercial
applications and
we can see on the left we've got a
tools menu
in the center is the design canvas and
at the bottom we've got the different
controls for each
tool and just to run through the
tools quickly
from top to bottom so at the top number
one we have the object selection tool
number two is the text creation and
editing tool
the third one is for adding shapes and
images to the canvas
the fourth is for document navigation
the fifth is for
object alignment and the sixth is the
last one
is for designing sorry previewing
your design
in terms of the controls at the
bottom i'll come on to those in a
later
video just demonstrating how the
application works
so one important point is just the
interaction approach that we've taken
within Sakura so
objects can can only be manipulated via
external control so the controls at the
bottom of the application
as opposed to direct fix gaze fixations
and we took that that decision again
just due to some of the inherent
limitations of eye tracking technology
and in terms of the accuracy of that and
and pixel level
precision and accuracy is very important
obviously in
creative tasks and design tasks so we
took the decision to make
to ensure that objects can only be
manipulated via external controls
so unfortunately there isn't sufficient
time to show all of the main features of
sakura but there are three key features
i'd like to focus on where there's
been little work
in the in the field to date around
and around the gaze interaction in
more
design focused domains so they are grid
a new grid-based
object selection approach that we've
developed how you can manipulate the
size and location of objects
and also how you can align objects
within Sakura
so initially then in terms of our
grid-based object selection approach
of course object selection is a
fundamental requirement for doing
creative work and that has to be
an efficient process but there are some
unique challenges with this so
there could be distracting backgrounds
there might be overlapping objects
of course objects can be a variety of
sizes and shapes and that can make
performing selections more challenging
and just to show you how the
grid based approach works i've got a
short video and what the user is trying
to do in this video is select the winter
wilderness text
just under the third image on the
right and to do that they'll initially
need to select the object
selection tool in the top left that will
display the the grid
on top of the canvas the user then
needs to select the cell
where their object is
the first object in that cell will
be highlighted and then there'll be some
left and right
arrows or controls where the user can
then navigate between the different
objects within that
that cell so i'll just play the video
now so you can see that working
so the object selection tool has been
selected the grid is then displayed
so the user then navigates to the
appropriate cell the first object is
highlighted with a red border around it
and then the controls are used the right
arrow is used to go onto the next
object within that cell
and then that selects the
appropriate object
so the key benefit of this approach is
it reduces the impact of target size
in particular small targets influencing
selection accuracy
there's just a consistent approach for
selecting targets of
of any size so the second area i'd like
to focus on
is object manipulation how you can
manipulate the size and
location of objects and in traditional
commercial applications
handles for example are commonly used
like the ones circled here in adobe xd
and these are efficient to use
if you're able to use a mouse to
manipulate those objects but if you're
using your eyes then of course that's
that's going to present challenges
because it's a very small target so in
terms of
demonstrating this there's another
short video here
and just to touch on the controls at the
bottom so we can see that the
image on the right
is currently highlighted so it's got
a red border around it
the controls at the bottom the move
tool so the second from the left is
currently selected
the one next to that on the left of it
is the resize tool
next to those two tools we have the
directional arrow arrows which enable
users to
move objects in different directions
or resize them in different directions
and we've also got an object
transformation size
feature which is the button with 100
in it
and when the user selects that it brings
up another controls menu where
users can select the size of
transformations that they would like to
perform
on an object so i'll just play the
video now to demonstrate how
this this works
so you can see the the right arrow is
being used to move the object to the
right the left arrow
up and up and down the user then
switches to
the the resize control
the object transformation size can be
changed so 10 has been selected here
and then by default it resizes objects
bigger if you select the resize tool
again it goes red
and that means you can resize an object
smaller
and then you can close the menu
so the benefit of this approach is that
you have full control over the
an object's location and size without
having to fixate directly on specific
pixels within the canvas that of course
is always going to be a challenge
because of the inherent limitations
around the accuracy of eye tracking at
the moment
whereas by using external controls you
have full control over the positioning
and size of objects
so finally object alignment so
alignment guides are very commonly used
in design applications to help produce
more professional
outputs and that can be through guides
or grids
but as we can see in this screenshot of
photoshop which has some vertical and
horizontal guides
they're going to be challenging to
select via eye gaze interaction because
they're essentially very small
targets so just to demonstrate how this
works with Sakura
so you can see the middle image here
isn't
snapped against a guide at the moment a
vertical guide
so what the object what the user
initially needs to do is select the
alignment tool which is the second from
bottom they then need to select the
guide that they want to
snap an object to they then select the
object that they want to snap to a guide
and then they use the snap control
to actually perform that snapping action
so i'll just play the video for this to
demonstrate how that works
so the alignment tool has been selected
so the user will then want the
select a guide tool so they're going to
select a vertical guide
and they can toggle between the guides
by using the controls there
so that guide's been selected
the user now is to select the
appropriate object they want to snap
so they've done that now and then they
use the snap control to actually then
snap that object to the
to the appropriate guide so after
developing Sakura we wanted to evaluate
whether it was an appropriate approach
for doing design work and we conducted a
lab-based study to do that with 24
non-disabled participants we used a
windows 10 pc with a 22 inch monitor and
of course Sakura was installed on the pc
we used the eye tribe eye tracker used a
jelly bean
switch to enable people to make
selections
we developed this design using sakura
and we
essentially broke this in different ways
and then asked participants to fix the
issues
so for example in one task we took out
the text professional freelance
photographer and then they had to add
that back in
and in another task we misaligned the
images or some of the text and asked
them to
to align it using the the
alignment guides
and there were four tasks in total
that they were asked to complete
and then we administered an online
survey where we asked about each of the
the key features developed
and in terms of the grid-based object
selection approach
22 participants provided positive
comments and there's a real emphasis on
it being a fast and
efficient approach seven participants
did highlight that shifting their gaze
between the canvas and the controls was
potentially an issue
so if you're focusing on the
controls at the bottom of the
application you can't necessarily see
what's happening
within the canvas in terms of object
manipulation
20 participants provided positive
comments around this approach
again there's an emphasis on the
controls being simple easy and intuitive
to use
for object alignment a similar theme 19
participants provided positive feedback
and there was an emphasis on it being a
fast easy to use and helpful tool
although three participants did state
that they found it a bit
time consuming and the fact that they
required numerous selections
and wasn't wasn't ideal for them we also
administered sus
and received a positive average
score across all participants
for that so given the positive
feedback we received
we wanted to do a follow-up study with
some disabled participants so we
conducted a study with six
participants with physical impairments so
five of those had
motor neurone disease and one had
multiple sclerosis
we conducted the study with four
participants in their homes one was a
social meetup for people with
mnd and and one was our university lab
and the task we set people was to create
this design
from from scratch from a from a blank
canvas so we gave this design to them on
paper
and then they had to start working
towards that but we encouraged them to
to deviate from this if they
wanted to
here's a photo just of one of the
participants in in their home
we used a laptop for this study you can
see the the switch
just in front of the laptop so that
that's an example set up
for for this study sessions lasted
between 90 to 120 minutes
and all participants we found were able
to utilize secures features to
create an initial design participant
five and and participant six their
designs contained more canvas elements
and more extensive use of
alignment guides and just an example of
that is participant six design here
you can see there's there's a lot of use
of guides alignment guides
images different text formatting and
obviously they had to move objects
around to line this up appropriately
so this design took participant six 15
minutes to
to complete the other participants
were able to utilize all of the
tools and controls
their designs typically just had fewer
design elements in them
so after participants completed the task
we did a semi-structured interview with
each of them
we have positive feedback again so
participant six for example said the
system was
intuitive simple and and straightforward
to use
but some of the participants did have
some did have some issues so in terms of
the switch selection process some of
them found it
challenging to perform a quick tap to
actually
accurately select a target within the
within the application this
issue again came through about
that disconnect between when you're
looking at a control you can't see
what's happening on the
on the canvas and the starting
position of new objects that was
something that was highlighted by two
participants so by default
objects are placed new objects are
placed towards the top left of the
viewable canvas
whereas actually two of the participants
said it would be better to actually
initially look approximately where you
want the object to be placed
and then you can use the controls to
nudge it into the correct final
position
we again administered sus at the end
of the study
and again achieved a positive
overall average score across
all participants in terms of perceived
levels of
usability so in terms of future work
moving forward there were some
interesting
points made by participants across both
studies and one of them was around
a multi-selection of objects and then
performing transformations
simultaneously and that's an area
where there's not been much work
around gaze interaction for creative
work
also using guides for
as a means for determining the size of
an object not just for snapping objects
to a guide for layout purposes but
also using them to help support the
size of
of objects and also the the creation
of a pen tool so that users can create
custom shapes via gaze input and
again there's been a lack of work in the
field
around this this particular area our
future plans moving forward we're
currently doing more iterative work on
sakura
and our plans are in the near future to
make this freely available online for
others to use so thank you very much for
for listening if you have any questions
about the paper and our work then please
do
get in touch with me and it would be
great to hear from you
thank you very much