Hi,I am Bharat Paudyal, PhD student at Birmingham City University. Today I will be talking about
our work Voiceye, a multimodal inclusive environment that allows users with physical impairments
to perform coding activities. This is a joint work conducted with Dr Chris Creed, Dr Maite
Frutos and Dr Ian Williams.
This short presentation will discuss the background of the research, our developed prototype called
Voiceye and a short demo of Voiceye. This will also cover two studies which we conducted
with both disabled and non-disabled users. Finally this presentation will talk about
the contribution of our study.
Programming involves multiple activities which includes designing, writing code, debugging,
etc. . It heavily rely on using traditional input methods such as keyboard and mouse,
which can be challenging for people with physical impairments. Assistive tools such as eye gaze
alternative input device. In terms of software industry, gaze and speech have been used independently
tracker, speech recogniser, hand gestures are becoming popular and are being used as
to support people in development work.
Speech based coding tools such as Talon voice, voice code are being widely used and presenting
new opportunities for programmers with physical impairments by reducing the dependency on
the traditional input devices. However these tools are primarily targeted for experienced
users and have been rarely evaluated with physically impaired users. Similarly, tools
like EyeDe, EyeNav and CodeGazer have been developed to explore to use of gaze as pointing
device for navigation purposes in coding environment. A number of research have suggested that the
combination of gaze and voice can be beneficial and can provide more effective method of interaction
as it addresses the limitation of each modality. Gaze interaction can address some of the misrecognition
issues of speech while speech input can help to overcome the issues of small targets selection
issues of eye gaze. Research have been carried out to explore the combination of voice and
gaze in non-programming fields such as controlling desktop application, word processor, web browsing,
creative software. However, no prior work has been conducted to use speech and gaze
interaction in coding environment.
To address the lack of research exploring the potential of multimodal interaction approach
and to assist coding activities for disabled users, we developed a code editor called Voiceye
that can be operated using the combination of gaze, speech, and mechanical switches.
The system uses voice input for verbal command such as selecting, navigating and removing
code as well as for dictating longer forms of non-coding text such as comments.
Gaze is used to provide a more controlled approach to write code via an on-screen keyboard . To
address issues with slow typing speed we integrated Emmet code that allows user to write HTML
CSS code via a shorthand notation. Meanwhile mechanical switches are used for performing
selection and triggering actions. The prototype provides fluid design with visual feedback
of the spoken commands as well as selected letters.
We will go through the demonstration of the prototype
An evaluation was conducted with 29 non-disabled participants to investigate the feasibility
of multimodal interaction in a coding environment. Through this evaluation, we want to evaluate
if participants will be able to use gaze to type, voice to control the interface, and
operate switches for selection and triggering the actions. 64 tasks including deleting,
writing, selecting and editing HTML and CSS codes were used. All participants completed
the tasks and Voiceye received the usability score of 68.1, which can be labeled as OK.
The prototype received positive feedback and majority of participants found the prototype
to be useful. Based on the feedback , some updates were made to the system including moving the
cursor to the center of the line and moving the cursor to x position left and right etc.
Voiceye was evaluated with Five physically impaired developers to explore their experiences in
writing HTML and CSS code using a real world scenario. Voiceye received an average usability
to utilize the features within Voiceye to write code. Majority of participants reported
score of 74 which can be labeled as good and usable. All the participants were able
that Voiceye was better than their existing coding tool and provided positive comments
about multimodal interaction approach.
Previous researches have independently explored the potential of gaze and speech. This study
presents a new system that integrates these methods of interaction to provide more efficient
approach of coding. The results from two user studies were positive and demonstrated the
feasibility of combining gaze, speech, and mechanical switch interaction for supporting
development work. The majority of participants from the first study found coding via multimodal
input controls to be an intuitive and simple. Physically impaired developers from the second
advantages over their existing assistive methods of interaction.
study also made similar positive comments with several stating that this approach provided
This prototype is loosely coupled and can be extended to include more coding features
such as navigation, debugging in near future.
We are grateful to Microsoft for supporting this project through an “AI for Accessibility”
grant. We also thank London RSI Support Center for supporting the recruitment of physically
impaired developers.
Thank you very much for your time