Hello everyone. My name is Youngwoo Yoon.
I’m presenting the paper “SGToolkit: an interactive gesture authoring toolkit for
embodied conversational agents.” This is a joint work with Keunwoo Park, Minsu Jang, Jaehong Kim
and Geehyuk Lee at ETRI and KAIST.
Embodied agents like social robots, game characters,
and virtual assistants are getting popular and being a very promising interaction medium. The
key difference from the other medium is that they have a human-like appearance,
so we naturally expect that they communicate as we do, using non-verbal behavior.
Non-verbal behavior might include facial expression, arm gesture, body posture,
and so on. Non-verbal behavior makes the agent look alive and human-like as
you can see this amazing robot from Engineered Arts.
In the present paper, we focus on generating upper-body arm gestures for given speech.
Generating speech gestures for artificial agents is not an easy problem. Typical methods are
keyframe animation and motion capture. These manual authoring methods give high-quality
gesture motion but there is a downside that they require high production cost. Another
solution is automatic generation. It has minimal production cost but low quality at the same time.
We aim to provide another option which is more efficient than manual authoring
and gives higher quality than the automatic method.
We had interviews with three experts working in different domains of
game, education, and robotics. They agreed that the automatic
gesture generation is necessary, but its quality is not satisfactory yet.
To improve the automatic method, they said the toolkit should allow a user to alter the
gesture motion by controlling some frames. And modifying abstract aspects such as style
is also necessary to give an easier way to control gestures. And the toolkit should be
platform independent so that users in different fields can use the toolkit.
Let me show you a demonstration video first. You can input a speech text
and click the generate button, then you will have automatically generated gesture motion.
Welcome to the mall. Our mall is the largest shopping mall
with over 250 shops. 8 screen cineplex, and plenty of gourmet cafes and restaurants.
Then you might want a metaphoric motion at the position of the word largest. Select
a specific time and craft the desired pose by moving or rotating joints of the 3D character.
Welcome to the mall. Our mall is the largest shopping mall with over 250 shops.
8 screen cineplex, and plenty of gourmet cafes and restaurants.
You see the generated motion is smooth overall as following the desired control.
We also have a collection of unit gestures. You can select one
and add to the motion control track.
And of course, you can move it along the timeline.
If you want more dynamic motion for some region. Then you would modify some style parameters.
Welcome to the mall. Our mall is the largest shopping mall with
over 250 shops. 8 screen cineplex, and plenty of gourmet cafes and restaurants.
Now, I’m explaining how we implemented this. Our model is based on the automatic
generation model which generates gesture motion from a given speech.
It is a neural generative model trained on a large corpus of speech gesture data.
More specifically, it encodes speech audio and speech text and generates a sequence of
human poses in 15 frames per second. On the base model, we added a pose control element that is
represented as desired human poses at specific time frames. The control vector is concatenated
to the speech feature vectors. Style controls were also concatenated similar to the pose
control. A style vector represents scalar values of desired degree of motion speed,
space, and handedness. Here, handedness is defined as a relative motion between right and left hands.
We trained the model on the TED dataset with the loss function here. The Huber loss
minimizes the difference between referential human motion and the generated motion. And
the GAN loss guides the model to generate realistic motion. The style term is to make
the model care about the input style vectors. Without this style term,
the model tends to ignore style control vectors which are somewhat weak signals.
We exposed the model to diverse pose and style controls by randomizing
control position and lengths. During the development, we used
some objective metrics of Frechet Gesture Distance to check the overall generation quality and pose
and style compliance scores to see whether the model follows the input controls or not.
We conducted a user study to compare the SGToolkit to automatic gesture generation and keyframe
animation. The user study consisted of two sessions. First one is the gesture design session
where the participants create gestures using the SGToolkit and using the manual keyframe animation.
And the quality assessment session is followed.
In this session, crowd workers evaluated the perceived quality of the gesture videos
from the previous design session and automatic gesture generation method.
In the gesture design session, we had two scenarios of
rushed and relaxed where the maximal authoring time is limited to 5 minutes and 20 minutes.
This is the survey result after the use of SGToolkit and manual keyframe animation method.
We asked participants whether they were able to create
gestures as intended and whether the result gestures were appropriate to the speech.
In the rushed scenario, SGT showed higher intention and appropriateness rating than MANUAL.
In the relaxed scenario where the participants had enough time to create
gestures, there were no significant differences between two conditions.
In the post-experimental interview, the participants said that it was easy to
create gestures with the SGToolkit because they could have an initial sketch
by using the automatic generation and they only had to add new gestures to some parts.
But one participant had a concern that our imagination can be limited by
an initial rough sketch.
In the quality assessment session. We compared four videos for the same speech.
There is the referential human motion that is a kind of topline.
And we had gestures generated automatically. And the gestures
from the previous gesture design session using the SGToolkit and Manual keyframe animation.
For each video, the participants rated appropriateness to speech and human-likeness of
motion in a 0-100 scale using a web-based video evaluation tool, called HEMVIP
shown on the right. We have collected over 4000 ratings from 120 participants.
These are box plots for appropriateness and human-likeness ratings. For the
human-likeness rating on the right, there is no significant difference between the conditions.
But we could find interesting differences for the appropriateness ratings on the left side.
In the rushed scenario where the authoring time is limited, the SGToolkit showed higher ratings
than the manual method. This indicates that users can create gestures more
efficiently using the SGToolkit. The SGToolkit did not outperform the automatic method,
but it still has benefits for containing a designer’s intention,
which is an important requirement but not possible only with the automatic generation model.
And in the relaxed scenario, SGToolkit showed higher ratings than the automatic method.
In summary, we introduced a new gesture authoring toolkit. It gives higher quality
output than automatic methods and is more efficient than manual authoring.
We accomplished this by proposing a neural generative model which accommodates
fine-level pose controls and coarse-level style controls.
The implemented SGToolkit is platform-agnostic and all the code for model, web frontend,
Blender animation is available. Feel free to utilize it in your domain.
This is the end of the presentation. Thank you for listening.