Hi everyone my name is Benedikte.
This is my first time at C&C and I'm very excited to present my work here today.
I train deep learning models on creative data like dance.
I'm interested in this partly because exploring
what computational mechanisms are needed to capture tacit information can garner insights into our own creative processes
however quantitative metrics used to train deep learning models often fall short of capturing important aspects, for this reason qualitative metrics are also needed.
The paper I'm here to present describes the results of a perceptual judgement experiment wherein respondents rate different aspects of ai-generated dance.
Dances is a very expressive thing and movement to music and convey of variety of information.
For example I'm sure most of you already have an impression of the music that was playing when these improvisations were recorded.
State-of-the-art generative AI may be able to create realistic looking body motion but is it able to capture these underlying correlations between sound and movement?
To examine this we trained a model on a set of dance improvisations set a different musical stimuli.
through an online survey we asked participants to rate the AI generated movement
Our question here is twofold: do respondents perceive the AI generated movement as realistic and having high quality and are these aspects related to how well the AI can model the sound motion mappings.
The model used is a mixture density recurrent neural network or MDRNN.
It takes as input a set of full-body motion capture recordings of 9 dances improvising to 6 different musical stimuli
We can then prompt the model with these audio features extracted from the music excerpts to produce the position of the body at the next frame.
We are not putting any constraints on the model for where each of these positions may be, the model needs to figure this out. so let's briefly see what that looks like.
In the early stages of training the model learns this strange shakey dance. It later learns to generate overall smoother movements but often breaks with what is possible for the human body to do like growing and shrinking at will.
The current model is able to generate relatively smooth and realistic movements, so let's start by comparing this to the human performances.
When looking at how the dancers moved to the 6 songs we found that two songs in particular stood out.
The dancers move the least two song B, a song with no rhythmical instruments and no clear beat and the most song F a pretty straightforward EDM track with a clear repetitive beat and high tempo.
We also looked at the fluidity of movement and found the opposite tendencies therefore song B and F.
Movements to song B were fluid while less so for song F.
To generate the stimuli for our survey we use the audio features for some F and B to prompt the mdrnn. We then present these examples once with their intended song and once with a different song in order to cause a mismatch between sound and movement.
We also included as a symetrical set of examples from the dataset for comparison. ie musically matched and mismatched animations of human dances using the same musical stimuli.
Participants are asked 4 questions, two related to the realism of the example and two related to their quality.
To measure realism we asked whether the movements are dance like and whether or not they fit together with the music.
To measure quality we ask whether the movements are expressive and whether participants like them.
It is worth noting that we do not expect the AI generated movements to score as high as the human movements.
The dancers are very good and even the most state-of-the-art movement models are yet to compare to human expressivity.
However, if this model is doing a good job we might expect to see the same tendencies across responses to human and AI examples.
In our results we discover some things that we expected. Firstly the AI generated movement score lower than their human counterparts, no surprises there.
The perceived musical fit of a given human example correlated directly to the sound-motion mappings, with mismatched audio examples scoring lower on musical fit.
The highest rated AI example was the animation of the high tempo song set to it's intended music.
As early results indicated the model generally performs better when generating movements for music with a clear beat.
This example gained the highest ratings across dance likeness, musical fit and preference but not expressivity -
which leads me to some of the more unexpected findings of the survey.
Animations to song B were rated as more expressive both for the human and AI examples.
This may imply that the model has successfully generated an expressive dance. However, this may also be due to certain aspects of the music itself.
One could argue that in general songs with slow tempo could be perceived as more emotional and thereby perhaps also more expressive.
The dance likeness ratings did not correlate to sound-mappings, neither for human or AI generated examples,
indicating that what we might perceive as dance does not always match up with what we feel suits the music.
We also noticed that when the human example of slower more fluid movements of example B are presented with mismatched upbeat music it is rated the lowest across all four questions.
Meaning respondents are less inclined to rate these movements as dance, less likely to say it fits the music, less likely to deam it expressive as well as perferring it less.
Moving slowly to fast paced music is apparently not good..
This study illuminated several areas where there is room for improvement for our current movement model.
Insights we would be hard pressed to discover using purely quantitative measures.
As such we hope to elaborate further on this in future work when comparing different AI models.
Thank you so much for listening.