Hi, my name is Hugo Scurto, I am a postdoc at Sorbonne Université, Paris.
In this video, I am going to present work done during my PhD thesis at IRCAM,
with colleagues Bavo Van Kerrebroeck, Baptiste Caramiaux, and Frédéric Bevilacqua.
Our paper is called "Designing Deep Reinforcement Learning for Human Parameter Exploration",
and was published earlier this year in ACM TOCHI.
Reinforcement learning defines a computational framework for the interaction between a learning agent and an environment,
where the agent incrementally learns by taking actions and receiving positive or negative feedback from the environment.
Our interest lied in investigating an interactive learning workflow,
where a human would communicate positive or negative feedback to an agent.
Specifically, we were interested in designing such interactive reinforcement learning workflow
for a creative task: human parameter exploration.
Parameter exploration consists in trying different parameters to explore different outcomes—in this case, sound.
Our hypothesis was that communicating positive or negative feedback to a learning agent
could help humans navigate in such high-dimensional spaces,
while also fostering their creative process.
Thus, our research question was:
How to design a generic reinforcement learning framework to support and foster human parameter exploration?
We opted for a participatory design approach to reinforcement learning with professional creatives,
using machine learning prototypes to rapidly test interactions before engineering,
and letting concepts develop through collective observations.
We focused on one creative domain, sound design,
and designed our first reinforcement learning prototype by implementing the "Sarsa" algorithm,
using human feedback as raw input data.
In our prototype, humans communicate feedback using an elementary two-button interface.
Behind the screen, an agent takes an action that modifies the parametric state of a sound interface,
thus generating a new sound, toward which humans can communicate their subjective preferences.
A pilot study, involving four expert sound designers,
let us conceptualise and develop a generic reinforcement learning agent,
that we named the "Co-Explorer".
The Co-Explorer implements a deep reinforcement learning algorithm, called "Deep TAMER",
engineered to learn rapidly in high-dimensional spaces.
The Co-Explorer also implements a curiosity-driven method
to steer its exploratory actions toward uncharted zones of the parameter space.
In addition to feedback, humans can interact with the Co-Explorer using keyboard and mouse controls
that we demonstrate in the following video.
We evaluated the Co-Explorer in a creative workshop involving twelve sound designers.
Sound designers used the Co-Explorer individually in two different tasks.
Each task was followed by a time of group discussion.
Overall, our process of participatory design of reinforcement learning with professional creatives led to several contributions.
On a technical level, we witnessed our switching from engineering the agent’s learning to designing its exploration behaviour.
This switching was made possible by working closely with our professional creatives.
On a conceptual level, we observed the switching from the task of human parameter exploration to a new workflow,
that we named "human-agent co-exploration".
This mixed-initiative workflow was shown to foster the creative process of sound designers.
Lastly, we extracted several insight and guidelines for designing with machine learning for creative applications.
We invite you to read our paper for detailed analysis of results,
including qualitative feedback from sound designers and quantitative analysis of user interaction.
Thank you very much for your attention!