Hey, Ray, how's the dinner tonight?
Hey, um, I enjoyed the dinner very much.
The insert curry chicken emoji was delicious!
What?
What are you talking about?
Oh, I'm just using Voicemoji to speak emojis!
Hello.
I'm Mingrui Zhang from the University of Washington.
And today I'm going to introduce Voicemoji to you.
Voicemoji is a speech based emoji entry system for people with visual impairments.
This work is conducted by me, Ruolin Wang Xuhai Xu, Qisheng Li, Ather
Sharif and my advisor, Jacob Wobbrock.
People with visual impairments also use emojis quite often.
In our interview with 12 blind and low vision BLV people, we found that
eight users used emojis every day.
However, the current emoji entry experience can be
quite unsatisfying for them.
As we know, BLV users, use screen readers to interact with their
smartphones based on the voice feedback.
And the most common way to input emoji is to use an emoji keyboard where a lot of
emoji options are listed on the screen.
And we need to manually pick one, but for BLV users, they have to listen
to the emoji description one by one, which is very time consuming.
(Voiceover speaking) Smiles and people category.
Selected smiles and people category smiles and people grinning face 31.
grinning face with big eyes, 32 grinning face with smiling eyes, 33
Another method is to use the emojis suggestions for example on the iOS keyboard.
If you type a certain word, the keyboard will suggest
you with some related emojis.
(Voiceover speaking) B, cap B, E, d f a a r r bear, original text.
Prediction.
However to do this, you have to memorize the exact keyword for the
emoji and for some emoji with long descriptions, such as "face with
symbols on mouth" or " cat with tears of joy", there is no keyword for them.
And based on the literature and our interview, we found that
BLV users prefer the voice input as a natural interaction.
So the question is, can we speak emojis?
And the answer is, yes.
Let's look at the demonstration of Voicemoji
Voicemoji is a speech based emoji entry system for blind or low vision users.
It has four main functions.
Searching emojis by saying "emoji search" plus the emoji keyword,
"emoji search" rainy day emoji.
(Voiceover speaking) Possible emojis are: first emoji cloud with rain emoji,
second umbrella with rain drops emoji,
third emoji sun behind the rainy cloud emoji, fourth emoji cloud with thunder emoji,
fifth thunder cloud with rain emoji, more available.
(Voiceover speaking) Emoji sun behind the cloud ... Added emoji sun behind the cloud with rain emoji
Directly insert emoji by saying the emoji keyword plus the name.
(User speaking) "The insert chicken curry emoji was delicious."
(Voiceover speaking) The emoji "curry with rice" emoji was
delicious. Emoji suggestions available.
Context sensitive emoji suggestions based on the spoken context,
(User speaking) "I'm going to play the guitar in the music band."
(Voiceover speaking) I'm going to play the guitar in the music band.
Emojis suggestions available.
(Voiceover speaking) Emoji peace hand sign emoji Emoji horn Emoji guitar emoji.
Button.
Added emoji guitar emoji.
And emoji modifications commands.
(User speaking) "I'll call you later insert calling hand emoji"
(Voiceover speaking) I'll call you later Emoji calling hand gesture emoji
emoji suggestions available, (User speaking) "Change the emoji to dark skin".
(Voiceover speaking) Emoji changed to calling hand gesture
with dark skin tone emoji
Specifically to support natural language style emoji queries, we use search
engine API to retrieve related webpages.
And extracted emoji results from those websites.
And we also added emojis suggestions based on the spoken content.
Using keywords and neural network methods to suggest the semantically related emoji.
With the suggestions that users can then learn new emojis and explore
the way to use them, especially when they do not know what emoji to input.
We conducted an online user study with users from China, and the U.S.
In general, Voicemoji reduced 91% of the emoji entry time
compared to the iOS keyboard.
If you are interested in the implementation detail or other
facts, please read our paper.
Thank you.