Hi so I’m Ryan Louie, and this is joint work with Andy Coenen, Cheng Zhi Anna Huang,
Michael Terry, and Carrie Cai at Google Research. And the problem we were trying to understand
was how novices might co-create with a deep generative model. For novices interested in
music composition, the rise of generative AI holds promise. Anyone with who wants to
create music might use a generative model to enhance their ideas and capabilities, therefore
allowing them to co-create a full composition in partnership with AI.
Despite research advances in generative AI’s algorithmic performance, it remains unclear
what novices actually need when co-creating with these generative models.
To understand more clearly the needs of novice composers, we conducted a needfinding study.
Novices experienced 2 challenges while using a conventional interface
for a generative model. The first challenge was information overload.
Although the AI was fluent in creating a full piece of music, the AI also overwhelmed novices
when creating too many notes all at once. Managing all notes and voices at once was challenging
because parts that sounded off were hard to isolate and correct.
Second, novices became frustrated with the AI’s non-deterministic output. Novices had
musical goals for their composition; but when the generated notes did not match those goals,
they felt incapable of providing feedback to control the direction. All they could do
was request generated notes over and over, almost like rolling dice, hoping one of the
rolls matched their goals. Our needfinding demonstrated that the interactive
tools for co-creating with generative AI could be better designed.
Although the conventional interface used in our needfinding study does contain an interactive
tool called the infill mask, this wasn’t enough to fight these AI-induced challenges.
To better match co-creation needs, we built a system we call Cococo collaborative co-creation
which augments conventional generative music interfaces with a set of AI steering tools.
Let me show you how it works. So, voice lanes allow specifying for which
voices and for which time steps to generate music. With this capability, users can control
the amount of generated content they would like to work with. This was designed to address
information overload caused by Coconet’s default capabilities to infill all remaining
voices and sections at a time. We implemented two semantic sliders in Cococo
to constrain generated output along meaningful dimensions: a conventional versus surprising
slider, and a major versus minor slider. Third, our example based slider allowed people
to constrain generation to be more similar or different from an existing piece of music.
And we provided users multiple alternatives of generated output that they could audition
and choose the one closest aligned with their goals.
So let me show you how this works with a one minute video demo. Let me start it at the
beginning. So they are using voice lanes to generate
just the bass. With the three alternatives, they can select
and audition them. And then continue on with their process.
They can refine the generated output by adjusting those sliders.
After doing one voice, they can move on to doing other voices and continue setting the
initial direction. Finally they can use the voice lane again,
to complete all four voices. So, let me explain how the AI steering tools
were able to adjust the AI’s generated output. To teach you about the underlying technique,
I’ll explain how it works to power our example-based slider.
So in the interface, the user might use the example-based slider to request “music similar
to this part”. Behind the interface, the model represents
the user request to “generate notes similar to this example part” as higher probability
of sampling notes within the example. The desired qualities are encoded in a soft
prior distribution that has higher probabilities for notes in the example, which you can see
in dark orange. Soft priors modulate the generated output, without retraining the model.
By using soft priors to adjust the models’ sampling distribution, we get an adjusted
posterior sampling distribution that allows the output to adhere to both the original
context, encoded in the sampling distribution, and these additional user-desired qualities,
encoded in the soft prior distribution. So, having built Cococo with AI Steering Tools,
we wanted to understand in an evaluation study what impact AI steering tools had on novices
co-creation process. So, we conducted a within subject study with
21 music novices comparing the co-creation process with a conventional interface, which
had an infill mask and Cococo which had the AI Steering Tools.
Participants made two compositions using the two variants. As a prompt, users were shown
images and were asked to compose music that reflected the character and mood of one image
of their choosing. We measured the impact the tools on novice’s
composing experience, attitudes towards the AI, and their perceptions of the final composition.
we found that with Cococo better helped participants express their goals, they had higher self
efficacy, felt the experience was more engaging, and learned more. No significant difference
was found on effort when composing. Our qualitative findings provide insight on
why AI Steering Tools helped novices to better express their creative goals.
The tools supported users expressing their creative goals by allowing them to assign
different parts different musical qualities. For example, a user assigned the soprano voice
an alternating pitch pattern to get at this feeling of indecisiveness in a main character
while they added mysterious dinging sounds in the bass voice to give a nice harmonic
backdrop. We found that AI-steering tools increased
users’ control, comprehension, sense of collaboration with the AI, and trust of the
AI. Our qualitative findings provide insight on
why the tools increased novices sense of collaboration with the AI.
So, novices felt that with the AI Steering Tools, the AI could be adjusted to do what
they would like it to do. And they felt that the AI had a clear understanding of who was
in control. Comparatively, using the conventional interface,
users were still optimistic about the AI working like a brilliant composer whom work could
be outsourced to, quickly. But they were pessimistic about its “blackbox” persona when trying
to communicate with it and “take-it-or-leave-it” attitude when working together.
Finally, the tools also contributed to a greater sense of ownership of the composition relative
to the AI, participants felt the music was due more to their own contribution relative
to AI, felt the music was more complete. No significant difference was found between the
two tools when measured on uniqueness. Our qualitative findings indicated that novices
who used the AI Steering tools felt more useful as a composer, which made them feel that the
final composition was more due their contribution. In contrast, novices who generated a majority
of piece at once with the conventional interface felt that the machine was doing all the work.
Through our observations of users co-creation process, we unexpectedly found that the tools
helped users with building bit-by-bit, debugging the music, and overcoming AI limits.
So people used the AI Steering Tools to built up their composition bit-by-bit, by using
the AI to generate just a little bit at time. Generating each bit was a checkpoint to intervene
and adjust the AI and its generated output, before going on.
Let me show you an animation of what this process was like.
So they would select the first half of the soprano voice, and just generate that bit.
They would continue on, generating three other voices in the next measure. Whereas with the
conventional interface, novices composed using a sculpting strategy. They would refine the
fully auto-completed music by repeatedly using this infill mask. Generating lots of notes,
but selecting a region they might not like, and regenerating it and repeating that process.
So, you might think that these two strategies are just two different preferences between
users, but composing bit-by-bit had a surprising side effect.
By building from the ground up, a brick at a time, users developed familiarity with the
piece. That familiarity helped them to identify the
cause of musical glitches later on, allowing them to really debug the music.
In contrast when sculpting with the conventional interface, novices had less familiarity with
how all the voices were interacting. So it was hard to disentangle what causes what.
In addition to debugging generated output, users had to deal with limitations in the
AIs ability to generate desired musical qualities. You see, many users wanted to control a set
of musical primitives, or desired sub-qualities for expressing their musical goals.
The Steering tools again helped users to overcome these AI limits. For example, they might have
relied on the idea that long notes convey a sense of calmness, while the rising shape
in pitches could communicate triumph or escalation.
When participants who could not find an explicit tool that mapped to their envisioned primitive,
they repurposed the steering tools as proxy controls to these underlying primitives.
Set the slider to conventional slider to generate music that was “not super fast or not a
strong musical intensity.” Or they used knowledge that different voices
correlated to with lower and higher pitch ranges, to encourage the pitch shape of generated
notes. What’s so interesting about this work, is
that we found that partitioning and constraining AI capabilities through semantically-meaningful
tools was fundamental to the creative process. We didn't change the model. We just made these
additional these interactive tools. But the steering tools made a pivotal difference in
people’s creative ownership, self-efficacy, and collaboration.