Hi everyone, my name is Maozheng Zhao, a phd student from Stony Brook University supervised
by professor Xiaojun Bi.
This work is in collaboration with Shumin Zhai from Google.
In this paper, we designed and implemented a voice and touch based error-tolerant multimodal
text editing and correction system for smartphones,
we call this system VT.
Changing text input such as editing text and correcting errors is a core activity we perform
daily on smartphones.
However it is difficult to perform.
The bottleneck lies in the need for precise and repetitive manual control.
In this paper we propose VT which only requires users to tap the approximate location and
speak the or correction content or editing command.
VT can automatically infer the exact location based on the voice and touch input and the
text context.
This is a demo about using VT to edit and correct text.
Attached document
Cut this paragraph
Paste
Bold
Highlight this sentence
Your assistance
VT can accommodate ambiguity and uncertainty in text correction
In this demo, the user will insert the word “better” to the editing sentence.
The user does not need to put the cursor exactly between the two words.
He just needs to tap any of the two words adjacent to the inserting location then speak
the inserting word.
Better
The user can say some context words around the insertion word to help to locate the insertion
location.
Some better news
In this demo, the user substitutes the two error words with “it was”.
The user does not need to precisely select the error phrase,
he just needs to tap the phrase or the word adjacent to the phrase then speak the new
phrase.
It was
The user can also speak some context words around the substitution phrase.
It was very
VT supports word level, sentence level and paragraph level text editing.
Cut two paragraphs
Paste
Underline this sentence
Highlight three words
Highlight
Bold
Highlight
Next, I’ll introduce the workflow of VT.
With touch input “t” and voice input “s”,
if the voice input starts with one of the editing keyword,
the inputs will be used for text editing.
Otherwise, they will be used for text correction.
For text correction, the first step is to generate correcting candidates.
For example, the user tried to correct this sentence to “when do you have to be there”.
The user tapped the mistyped word and spoke “do you have”.
VT will generate 3 kinds of candidates, insertion only, substitution only, insertion and substitution
candidates.
The second step is to compute scores for each candidate.
The scores are computed based on a tri-gram language model, word embedding similarity,
editing distance and the location of the new phrase.
The third step is to generate suggestions based on the combination of those scores.
In this example, this sentence is the top suggestion, it is the user’s correction
intention, “when do you have to be there”.
The top suggestion will directly replace the original sentence in the text view, the second
and third suggestions will be shown as alternatives.
For text editing, there are two modes. Single command mode and compound command mode.
Underline
Underline this sentence
​​We did two user studies to compare VT with the existing techniques.
The first user study is to compare the performance of VT with the touch-only method.
In touch only condition, users used existing touch based method in Android OS.
In the VT condition, users can use VT method or touch-only method.
This is because VT is proposed to augment rather than replace the touch-only method.
Here are the results. This is the task completion time for text editing and correction.
VT reduced about 30 percent time for both text editing and correction.
We did another user study to compare VT with IOS voice control, which is an existing voice
and touch based method on iPhone.
For text correction, VT reduced around 48% time from iOS voice control.
For text editing, VT reduced around 30% time.
To investigate where VT saved time, we divided the task completion time into two parts, text-selecting
time and text-changing time.
We can see that VT mainly reduced the text selection time.
This is because VT does not require precise text selection.
It will infer the actual location based on the input and text context.
Conclusion
First, we have designed and implemented a voice and touch based error-tolerant multimodal
text editing and correction system for smartphones
Second, VT can accommodate ambiguity and uncertainty in text editing and correction.
Third, VT greatly improves the performance of text editing and correction over the touch-only
method and the iOS’s Voice Control method.
This is a demo about using VT to edit and correct text.
Attached document
Cut this paragraph
Paste
Bold
Highlight this sentence
Your assistance
Thank you for your attention!