Hello everyone! Today we will present VibEmoji: Exploring User-authoring Multi-modal Emoticons in Social Communication.
I'm Ziqi Zhou, and this is the collaborative work with Pingcheng An,
Qing Liu, Yifei Yin, Linghao Du, Da-Yuan Huang, and Jian Zhao from the University of Waterloo,
Huawei Human Machine Interaction Lab, and Southern University of Science and Technology.
The birth of the emoticon can be traced back to the 1970s
and it has expanded its usage from email to daily communications.
Emoticons enable people to convey feelings, emotions, and any nonverbal reactions concretely
and efficiently in different cultural backgrounds.
According to Emojipedia, the usage of emoticons has constantly been increasing from 2012.
In 2021, over 20% of tweets now included an emoji and five billion emojis are sent every day on Facebook Messenger.
Well, emoticons are more and more popular in social communications, static emojicons cannot
meet people's various demands for self-expression.
Users are seeking more customized and enriched form of emoticons.
Thus, the multi-modal emoticons that combine animation, haptic, and sound are emerging.
You can see there are some explorations in the industry.
Google Gboard enables users who use the emoji with recommended animation.
Apple IOS enables the customized memojis and animojis.
In Huawei Meetime, user can use pre-defined emojis with sound and vibration.
While, in Tecent Wechat, full screen animation and haptic effects will pop up for some emojis.
Thus, we formulated three design principles, which are encouraging users to freely combine multi-modal elements,
providing recommendations without limitation, and supporting on-the-fly authoring of multi-modal emoticons.
These considerations help us design features beyond the existing use cases and better support users communication.
We will introduce the design idea of our application.
The application looks very similar to a daily used communication app.
Pressing the left icon will open the interface
and there are three parts from bottom to up:
emoticon part, vibrotactile patterns part, and animation effects part.
User will select each of them and assemble a VibEmoji for communication.
Next, let's see a demo of our application.
With VibEmoji, users could freely combine emoticon stickers, vibrotactile patterns, and animation effects.
After selection, the orders of the elements in the unselected sections will change based on VibEmoji's recommendation.
Moreover, each vibrotactile element is displayed in its waveform,
when selected, it will be rendered by the haptic module of the phone in sync with other selected elements.
After selection the three elements, press the "Send" button, and the VibEmoji will be delivered.
In the other side, the communication partner will receive the VibEmoji.
Now, we will talk more details about each component, and let's start with the dataset.
The animation library comes from Kineticons, which contains a set of animations for icons and graphics.
We select 15 of them that are suitable for emojis.
The vibrotactile library is from Vibviz, which contains a set of vibrotactile patterns.
We filter out vibrations based on the harmony and relevancy with the emoticon data.
Finally, we selected 60 vibrations.
The emoticon dataset is from LEED, which is Lisbon Emoji and Emoticon Database.
We selected the emojis that appear in the IOS system.
We will recommend multi-modal elements to users based on their emotional relevancy to support the authoring process.
Thus, we collected valence-arousal data of the three elements in in order to find relationships.
For the emoticon dataset we have the valence-arousal data from LEED.
For vibrotactile data we did a mobile survey app with 26 participants.
And, for the animation data set we did a web-based survey with 52 participants.
Here is the web survey questionnaire in the left,
and the mobile-based survey questionnaire in the right.
I will talk about the recommendation system.
The recommendation system mainly considers the emotional model and user behaviors.
The emotional model is measured by the euclidean distance between elements in the valence-arousal 2d plane.
For example, once a sticker is selected, all vibrations that have lower euclidean distance will be in the front.
Another aspect is user behavior.
We use the TF-IDF method which is popular in the field of information retrieval.
TF means term frequency, which represents the frequency of the previous selection.
IFD, the inverse document frequency is used to reduce the bias of the two common elements.
For example, if there is an animation in every sticker,
the weight of that animation will be lower to increase the diversity of the recommendation system.
Every time when a new element is selected the order of all unselected elements will be renewed.
Next, we will introduce the system architecture.
The frontend is developed by React Native framework,
and the backend is developed by node.js.
The vibrotactile patterns are played using a native Apple haptic engine.
We use socket.io for real-time communication across the front-end and back-end.
The Apple Push Notification service is used for the message notification delivery,
in which the user could receive the notification on any Apple device.
We evaluated VibEmoji with 20 users in the wild for four weeks
and gathered rich empirical data about how they created, shared, and experienced multi-modal emoticons in everyday communication.
Overall, as you can see from the results, all participants sent 1824 messages and 581 multi-modal emoticons using VibEmoji in the field study.
The median interaction time is around 7.09 seconds.
The users gave strong and positive feedbacks about VibEmoji in different aspects, regarding the conversations
engagement, joy, expressiveness, accuracy, diversity, and mental feelings.
We want to point out four typical scenarios for multi-modal emoticons values.
The first scenario enhances the existing meaning that a static emoticon conveys.
An example is using a snore like vibration and slight scrolling motion to enhance the sleeping emoticon.
Another scenario is to create a brand new meaning.
For example, a shaking head motion with vibration in a zip emoticon
can express a completely opposite meaning of the original emoticon.
The third scenario is to set the atmosphere for a conversation session.
A participant reported he often used a group of multimodal emoticons to
set an atmosphere for making jokes.
The last scenarios construct attentive and empathetic responses.
Using the same emoticon with different combinations of vibrations and motion
can effectively convey the attentive response to the conversation partner without interruption.
said from a participant.
Apart from these four types of scenarios, we also discovered
more insights into how users construct and interpret multi-modal emoticons using VibEmoji.
Thank you for watching our presentation! For more information, please see our paper.