Hello, everyone.
My name is Toshiki Aoki, and I’m going to introduce EmoBalloon
a system to convey emotional arousal in text chats with speech
balloons by my colleagues and I from the University of Tokyo.
In recent years, computer mediated communication
CMC has become common in our daily lives.
However, the media richness theory suggests that there are limitations
to the transfer of emotional information in CMC, especially in text chats.
In text chat, only letters, symbols
and paralinguistic cues, such as emoticons can be used,
whereas the tone of voice, gestures and facial expressions that are used
to infer the other person's emotion
in face-to-face communication are often not available.
As a result, emotional communication
can be hampered, resulting in misunderstandings.
For example, in a situation like this in text chat,
"Good morning" and "Good Morning" are sent in the same way.
But as you can see from the tone of voice and facial expression,
the emotions of the speaker at this time are completely different.
Many methods to support the transmission of emotion
in text chats have been studied in the past.
Most commonly, emoticons and stamps are used in various text chat
applications and in recent research, speech balloon colors and fonts
have been also discussed as alternative emotional cues.
In this study, we focused on the shape of speech balloons,
which are always used in text chatting
but do not function as emotional cues in present applications
and confirm their effect on the emotional communication in text chat.
Since speech balloons are traditionally used
in Japanese manga to enhance the characters expressions, we began
our research by analyzing speech balloon shapes in the Japanese manga dataset,
Manga109, and confirmed the relationship between the shape
of speech balloons and the emotional arousal conveyed in dialogue text.
Next,
based on these findings, we used the neural network
ACGAN to construct a system that generates speech balloons
corresponding to an input emotional arousal value.
Then, to validate the system, we evaluated and analyzed
five images generated by ACGAN using crowdsourcing.
We found that the shape of speech balloons from a more "round" shape
to a more “explosion” -like shape has a linear
relationship with the degree of arousal.
Finally, we implemented EmoBalloon,
text chat system with ACGAN, and conducted a controlled experiment
to confirm the effect of generated speech balloons on the emotional communication
in text chat.
The experiment was conducted under three conditions:
a voice input EmoBalloon condition with automated speech balloon generation,
a manual selection EmoBalloon condition and emoticon condition used as a baseline.
In the condition where speech balloons
were automatically generated according to the voice input,
We developed a text chat application using voice input.
When the user speaks a message, the application detects
the speech content and the volume of the user's voice,
and then automatically generates a speech balloon from scaled volume.
Under the condition where the user manually selects speech balloons,
we developed a text chat application with traditional keyboard text input.
The user manually enters text and then selects the speech balloon
that they think is most appropriate to convey their emotion.
In the emoticon condition,
we developed a text-input text chat application
that allows users to freely input emoticons in their messages
similarly to current popular text chat applications.
Each participant role-played a dialogue script of an emotional situation
with a partner in a randomly assigned condition and rated the emotional arousal
and valence contained in each message on a 5-point scale
each time a message was sent or received.
As a result, when analyzing the difference in
arousal evaluations between a message sender and a receiver,
using either the voice-input or manual selection
EmoBalloon to convey emotional arousal
significantly increased agreement compared to text chat without any paralinguistic cues,
while emoticon did not show such a result.
From the results so far,
We found that Emoballoon was a more effective
method to convey emotional arousal in text chats than traditional emoticons.
In terms of emotional arousal evaluated by senders,
we expected that voice input EmoBalloon would cause senders
to produce messages with a higher level of arousal.
However, we found no evidence that voice input
would affect the sender's evaluation of the level of emotional arousal,
unlike the two text-input conditions.
one possible explanation is that
senders using the voice input EmoBalloon
may have been worried about a type of "negative emotional leak"
since the speech balloons were generated based on their voice
rather than selected manually, and the receivers were aware of this.
This finding should be investigated in more detail in future works.
Conversely, in terms of emotional valence, we found that
EmoBalloon had no significant effect on sender or receiver evaluations, unlike emoticons.
This indicated that speech balloons generated by EmoBalloon
can be characterized as paralinguistic cues that can appropriately
convey only arousal without affecting receivers' perceptions
regarding the emotional valence in text messages.
For more detailed results, please see our paper.
Since we were freshmen, we have continued this research
with the goal of communicating emotions over
CMC visually to people with hearing impairments.
Our research also highlights an opportunity to support
emotional communication over various barriers in novel ways,
including remote as well as cross-lingual communication.
We hope that our research inspires others as well
to search for ways to create a more inclusive society.
Thank you.