Hello. My name is Jeremy Chu, and today I will be discussing our paper
WordGesture-GAN: Modeling Word-Gesture Movement with Generative Adversarial Network
First, I would like to go over what word-gesture typing is
Word-gesture typing is a continuous stroke across a virtual keyboard in order to input an intended word
The video on the right depicts a user inputing a sentence using the word-gesture typing keyboard
This method has been widely adopted on consumer virtual keyboards
Such as iOS keyboard, the Microsoft Swift Keyboard, and Google's GBoard
In order to determine what the intended word is from a gesture
a decoder model is used, which combines a shape model and a language model to determine what is the likely word
Next I would like to go over a word gesture production model since that is what our model would be classified as
A word -gesture production model simulates how word-gestures are produced on a virtual keyboard
There are three key benefits to creating models such as these
These models help us to gain deeper insight into how users use a system
Second, they help to generalize how user interact with these systems
And third, these models allow us to create new unseen data that can help with the design and evaluation of interfaces
Here I would like to briefly go over the basic structure of a Generative Adversarial Network
In a Generative Adversarial Network two networks, the generator and the discriminator are put into an adversarial minimax game
The generator attempts to generate data that looks as realistic as possible in order to fool the discriminator
In the case of word-gesture production that would mean creating as realistic of word gestures as possible
While the discriminator is attempting to discern the generated data from the real data
In the case of word-gesture production that would mean it is attempting to figure out whether or not a gesture is generated or real
Generally, for the discriminator the output is a single value saying how real the input is
The generator generally draws from an easily sampleable latent space, such as a gaussian distribution, when generating outputs
However with this simple setup, there isn't any way to control what the output of the network is
However, the addition of a conditional input can allow for the control of outputs from the generator
In our paper, we present WordGesture-GAN
WordGesture-GAN is a Generative Adversarial Network that is able to generate word gestures
The model can use any arbitrary text as input and requires no reference gestures in order to generate a gesture
The diagram below shows a generalized view of how the entire network structure works
We summarize our contributions as follows
First, our model is able to generate full gesture for any arbitrary text
This includes spatially, that being the x and y coordinates of the points of the gestures
and temporally, that being the timing between the points in the gesture
Second, we use a variational auto encoder to control the latent space to ensure that it is easily sampleable
And third, we apply a two cycle training to ensure that the outputs have greater variety
Here I will go over how a word prototype is constructed for input to the generator
The first step is to get the centroids on the target keyboard for the given word
After acquiring the centroids, we then perform linear interpolation between the centroids to construct the full prototype
These points are normalized into the range of -1 to 1 relative to the keyboard width and height in order to align with the output of the generator
We then add a time dimension giving the time in seconds between the current and previous point
And the final resulting prototype is a sequence of points consisting of x coordinates, y coordinates, and the time between the current point and the previous point
Depicted here is a diagram of the generator structure
In training only, the variational encoder is used to encode user drawn gestures into a gaussian latent code
Like the Variational Autoencoder, the reparamaterization trick is used to ensure the output code is in a gaussian distribution
This Gaussian latent code is then concatenated with the word prototype as input into the generator
The final output of the generator is then a simulated gesture
With the simulated gesture, that simulated gesture is fed into the variational encoder in order to generate another gaussian latent code,
which is used in the two cycle training to compare against the original latent code
An important part of training WordGesture-GAN is the two cycle training method inspired by BicycleGAN
Different gaussian encodings z should result in different gestures when input into the generator
At the same time though, the encoder should be consistent in generating z values from input gestures
In order to enforce this, we train in two cycles
The first cycle is to take a reference gesture x and input it into the variational encoder to get a gaussian latent code z
This gaussian latent code z is then input into the generator to generate a simulated gesture and the two gestures,
x and x prime are compared against each other using a reconstruction loss
In a separate cycle, a random latent code z is sampled from a gaussian distribution
and is input into the generator to generate a new gesture x prime
This gesture is then fed back into the variational encoder to generate a new gaussian latent code
and this gaussian latent code is then compared against the original gaussian noise by applying the L1 loss
Here we present some visual results comparing user-drawn gestures against WordGesture-GAN and two other word gesture production models
The Minimum Jerk model, and a style transfer GAN
Important to note is that the word voice was not seen in training for any of these models
Gesture generated by WordGesture-GAN resemble the way that users draw gestures for the same words
Here we present further visual results of generated gestures from WordGesture-GAN
To reiterate, none of these words were seen in training
As you can see from the bottom row, WordGesture-GAN is able to generate a variety of gestures for a given word
Here we present empirical results comparing the simulated gesture distribution and the user drawn gesture distributions for each of the production models
The distance metric used is the Wasserstein distance, which can tell us the distance between the two distributions
The metrics used are the L2 distance and dynamic time warping
Important to note is in our calculation of the Wasserstein distance, the point order within each individual gesture is maintained
As shown in the results here, the WordGesture-GAN is more similar to the user-drawn gestures than either of the two other Word-Gesture production models
WordGesture-GAN can also simulate the velocity and acceleration curves for gestures
In evaluating the correlation for velocity and acceleration against the user drawn data
WordGesture-GAN is able to perform similarly to the Minimum Jerk model in predicting the velocity curve,
and similarly to the Style Transfer GAN in predicting the acceleration curve
Since our model can also predict the duration of gestures, we also compare our model against the curves, lines, and corners model, or the CLC model,
which is designed to predict the mean duration of gestures given a word
As can be seen from this bar graph, while the CLC model is able to better predict the duration of gestures for lengths of words that are shorter,
as the length of words get longer, the WordGesture-GAN is better able to predict the duration of gestures compared to the CLC model
Since WordGesture-GAN can model the timing of points, this also means that we can play back the gestures in real time
The top gesture in blue is a gesture generated by WordGesture-GAN
While the lower gesture in orange is a user drawn gesture
The WordGesture-GAN gesture has no reference to the user drawn gesture
Here's another example for the word place
And here are examples of word gestures generated for various words from WordGesture-GAN
Thank you for watching
For more details on the implementation of WordGesture-GAN
Please refer to the full paper