Have you ever used a voice interface while driving?
Were you satisfied?
Recently, voice user interfaces have become more widely used.
However, there are still some issues that make voice user interfaces uncomfortable,
particularly while driving.
I would like to present my research on improving the use of the voice interface while driving
by using a tactile input and output interface.
In-vehicle voice recognition is making a comeback, fueled by the recent rise in voice recognition
systems in consumer technologies such as Siri and Google Assistant.
In the driving environment, it is very important to maintain visual focus on the road conditions.
Therefore, voice interfaces, which can be used without looking, are particularly beneficial
to the driving environment.
Recent in-vehicle voice interfaces provide not only navigation information, but also
information such as nearby restaurants, today’s schedule, and mail contents.
In addition, a wide variety of additional features including sending messages and
adjusting sound volume or brightness are offered.
Voice interfaces enable you to perform these functions without looking at the monitor.
However, there are usability issues in voice user interfaces due to the sequential and
temporal nature of speech.
The first is the turn-taking problem.
A user sometimes misrecognizes whether the interface is listening or not.
The second issue is short-term memory load.
It can be difficult for the user to remember multiple items when the result is too long.
The third is inefficient situations.
For example, to fine-tune the volume or brightness,
the user must repeat initiating Siri and saying the commands.
The last issue is error correction.
To correct a single word in a long message,
the user is required to repeat the whole message again.
The last two issues can be categorized as re-speaking whole command issues.
To overcome these issues, we can think of visual or auditory feedback.
However, considering the driving situation, the visual method is not appropriate.
In the case of the auditory channel, auditory feedback should be designed as temporary sounds
in order to not be a disturbance to the speech interaction.
However, previous studies noted that persistent feedback is better because temporary feedback
in driving situations may not be recognized by drivers.
Therefore, we design a persistent tactile feedback with manual input interactions to
overcome these issues.
For these three usability issue categories, the corresponding interaction themes are designed:
state feedback, output control, and input adjustment.
For the turn-taking problem, persistent tactile feedback was designed.
For the listening state, wave patterns are generated.
Wave patterns converge from the outside to the inside, as if a radar is searching for
the user’s voice input.
The listening state also has a timeout that automatically returns to the idle state if
the user’s voice is not detected for a certain period of time.
To deliver the time remaining until this timeout, these wave patterns move faster and faster
if no voice is detected.
When a user’s voice is detected, the wave patterns change direction to inform the detection.
The wave pattern that travels outward mimics the shape of the user’s voice spreading
as a sound wave.
For the processing state, a pattern representing a rotating fan is provided to users.
This rotating shape mimics a GUI widget that rotates to indicate that it is loading.
We considered various dynamic patterns to finally choose these patterns.
These selected patterns were easy for users to remember because they were metaphorical,
as explained earlier.
The three patterns were different in the direction of the stimulus or the order of the stimulated
fingers so that the user could distinguish them well without looking at the device.
To solve the short-term memory load issue, we designed an output control.
When a user searches an item list such as a list of train times or today’s schedule,
a ladder-shaped tactile pattern is printed at the index finger position.
A square box in this ladder pattern means an item.
By scrolling through this pattern, the user can listen to the previous item again or skip
the unnecessary item and move on to the next item.
Users can also interact with the item through a double-tab gesture.
When the user’s hand leaves the device and then touches it again, these patterns are
moved to the new touch position.
This feature allows the user to engage without looking at the device when clutching to scroll the list
or when the user wants to use the device again after taking the hand off from
the device for both-handed driving.
For the re-speaking whole message issue, input adjustment interactions were designed.
To fine-tune the sound volume or brightness level, once the user says a command, a tactile
jog pattern appears.
For example, if the user says ‘set volume to 70’, a tactile jog shuttle will appear
at the location of the index finger.
The user can then fine-tune the value by moving the finger along the jog shuttle, instead
of repeating similar, commands such as ‘set volume to 80’ or ‘set volume to 75’.
On the right is the interaction that corrects the error in a message.
The user said ‘I’ll be there in thirteen minutes’,
but the system misrecognized ‘thirteen’ as ‘thirty’.
In this case, in a typical voice interface, the user must repeat the entire sentence.
If the user repeats the whole message, other words that were previously properly recognized
may sometimes be misrecognized.
We designed the tactile I/O interaction so that users can correct the wrong words instead
of repeating the whole message.
A horizontal ladder pattern appears to review the words in the sentences.
Each square represents a word.
If the user finds a word that is recognized incorrectly, the user can correct it by saying
a new word while touching it.
Finger Feedforward is a common interface element to ensure learnability.
The number of fingers used in each interaction step varies.
In the State Feedback interaction stage, three fingers are placed on the touchpad to recognize
various patterns.
In other interaction stages, one or two fingers are used for manual input.
For example, in Output Control, one finger is used.
The system should provide a cue to help users learn which finger (or fingers) to use in
the current stage.
For this purpose, the interface delivers a tickling stimulus under the index finger.
When the user feels a tickle, the other fingers are lifted to perform single-finger operations
that browse the list of train times.
We conducted a user study to evaluate the scenarios.
There were two conditions: voice-only and Voice+Tactile
In the voice-only condition, users interacted with the typical voice interface without the
tactile device.
There were two tasks: train reservation task for output control
and message sending task for error correction.
We measured task completion time, task load score, and driving performance of a lane changing task.
As a result, the Voice+Tactile shortened the completion time.
This is in line with our expectations.
When the users were browsing the train list in Voice+Tactile, they didn’t have to hear
the list from the beginning just because they forgot an item, and they were able to quickly
skip the unwanted items.
In the message-sending task, the wrong word locations could be selected and corrected
faster than saying the entire message again.
For NASA-TLX scores, there was no additional task load in Voice+Tactile,
when compared to voice-only.
The Voice+Tactile condition required the use of a new tactile device and the operation
of the steering wheel with only one hand.
Nevertheless, there was no difference in the task load.
Also, there was also no significant difference in the driving performance of the Lane-changing task.
For future work, there are potential issues in real driving situations and the position
of tactile display.
Real driving situations such as being blocked by a vehicle or having to respond quickly
to avoid an accident may be more complex.
In Voice+Tactile interactions, decoding the tactile patterns requires perception and cognition.
In stressful driving situations, this can measurably affect driving performance.
This may require future studies with more realistic driving situations.
Drivers sometimes must return their right hand to the steering wheel, such as when making turns.
If the driver was using Voice+Tactile, the hand operation on the touchpad would be interrupted.
Voice+Tactile should tolerate such interruptions.
Indeed, this requirement is already supported in some Voice+Tactile interactions.
However, the effectiveness of this seamless transition is yet to be tested.
A user study should verify whether users could adapt well to the transition flows.
In this study, the tactile display is located near the armrest.
Different positions may be possible.
The center of a steering wheel is wide and close to the steering hand.
However, its area is not rest-able, and therefore the touching posture may be less stable.
The steering wheel rim is closest to the steering hand.
However, previous studies imply that the information bandwidth may be too small to support the
Voice+Tactile interaction.
Meanwhile, using other positions may be explored by future research.
We proposed the Voice+Tactile concept, which augments voice interface via tactile interaction.
We designed three Voice+Tactile interaction themes: State Feedback, Input Adjustment,
and Output Control.
Voice+Tactile showed better efficiency and improved user experiences overall.