Hello everyone, welcome to my presentation of our paper: Enhancing Interactions for In-Car
Voice User Interface with Gestural Input on the Steering Wheel. I’m Cui Zhitong,
a PhD student at Zhejiang University. Voice user interfaces are becoming
indispensable in car for offering drivers the opportunity to make distraction-free inputs and
conduct complex tasks. However, the usability and control efficiency of today’s VUI remain
to be enhanced due to its sequential nature. A typical usability issue of VUI is lacking the
flexibility to control interfaces. For example, there are no shortcuts to perform
familiar actions as interacting with a keyboard, which causes a reduction in efficiency. Also,
VUI has not been well-designed to help people recover from errors. Additionally, users lack
support to reduce the cognitive load required to retain the critical information in voice outputs.
Prior research has acknowledged that the steering wheel as a promising design space
for enriched gestural interactions due to its easily accessible location helps reduce
visual demand, thereby improving driving safety. Therefore, we see an opportunity to enhance the
usability and interaction efficiency of in-car VUI by incorporating flexible gestural input on the
steering wheel as a supplementary input channel. Here is an interaction scenario of our system:
a user invokes the VUI system using a double-tap gesture and makes a request by saying,
"I wanna buy two tickets to the Oscar Cinema to watch the movie Mulan on Saturday at 8 PM".
Since this action would be difficult to undo, the system explicitly double-checks by saying, "All
right, there are two reserved tickets in the Oscar Cinema for the movie Mulan on Sunday at 8 PM.
Did I get the message right?". Because the system mis-recognized the word "Saturday" to “Sunday",
the user could correct the wrong word by drawing the first letter ’S’ of it to select, and then
re-speak the word “Saturday” to let the system know. Then the system prompts again, "Got it. That
is Saturday instead. I have found the best spot for two people. Would you like to get these?".
Next, the user can perform a checkmark gesture to confirm the purchase. Then the system
replies, "It’s done. Enjoy your movie!" The above figure shows the user flow when
interacting with a typical VUI system, consisting of 5 states (S1 to S5): invoking VUI, making
speech request, listening to response, confirming system inquiry, and correcting speech request.
Corresponding to different states of user interaction with VUI,
we proposed novel interaction opportunities and gestural commands on the steering wheel,
aiming to supplement the usability issues in VUI with enriched control of the interfaces.
Gestural commands are listed in the Table. Taking the state of making speech request for example,
speech command is poorly suited to fine-grained and real-time control of continuous actions (e.g.,
adjusting window height) or analog parameters (e.g., volume or ambient light). Therefore, we
expect gestures to fine-tuning a parameter after the voice command has activated the function.
To better understand drivers’ touch behaviors on the steering wheel while maintaining driving
safety, we conducted a user-elicitation study where participants were asked to create gestures
for the above gestural commands during simulated driving. The left figure shows the distribution of
touch locations on the steering wheel. Based on this distribution, the right figure shows the
placement of our sensors to recognize gestures. This figure shows our elicited user-defined
gesture set, which was developed in light of the most occurrences of gestures for each command.
Leveraging what we have learned from drivers’ touch behavior, we implemented
a proof-of-concept prototype consisting of two customized capacitive sensors attached to the
steering wheel to recognize all the user-defined gestures. The sensors are flexible to adapt to
the irregular shape of the steering wheel and facilitate intuitive gestural input.
We designed our prototype consisting of SensorA and SensorB. SensorA recognizes thumb input, and
SensorB recognizes index finger input. We used the classical two-layered row-column diamond pattern
to fabricate the two sensors. Next, we developed a Processing library to convert the raw capacitance
values into touch coordinates robustly, and then classified them into our user-defined gestures.
Figure a is the layer-by-layer materials of our sensors. Figure b demonstrates that our
sensors are flexible to use. Figure c is the hardware circuit diagram for mutual-capacitance
sensing. Figure d is the final prototype we used in the evaluation study.
We then conducted a controlled user study to compare our system with conventional VUI based
on three secondary tasks in a simulated driving environment. In the primary driving task, we
utilized the Lane Change Test (LCT) to test driver distraction caused by in-vehicle secondary tasks.
To provide consistent primary task workload, participants were instructed to change lanes
immediately according to signs on both sides and perform good lane keeping. We designed
three secondary VUI tasks: fine-tuning the ambient light, listening to a message, and buying movie
tickets. Participants conducted these tasks in two interaction conditions. One interaction condition
is using conventional VUI. The other is using VUI + our gestural commands and user-defined gestures.
Then, we evaluated our system’s usability regarding driving performance, interaction
efficiency, cognitive workload and user feedback. We used the Line Change Test (LCT) to measure
driving performance, and results indicated that performing gestures while steering may
slightly affect the driving performance than just using VUI, without bringing about a significant
reduction. We also measured interaction efficiency and results demonstrated that our system
significantly improves the control efficiency of conventional VUI. Meanwhile, results from NASA-TLX
questionnaire showed the overall cognitive workload was reduced with enriched gestural
control of the interfaces. We envision that our work could support drivers to conduct more complex
tasks by integrating these modalities in the car. I’d like to thank all my collaborators who
contributed along the way. Thanks for listening and please refer to our paper for more details.