Hi, my name is Hugo Scurto, I am a postdoc at Sorbonne Université, Paris.
In this video, I am going to present our paper, called "Prototyping Machine Learning Through Diffractive Art Practice",
co-authored with Baptiste Caramiaux and Frédéric Bevilacqua,
which assembles work done during my PhD thesis at IRCAM, and postdoc at École des Arts Décoratifs, Paris.
Machine learning has become one of the most successful computing techniques in the 21st century,
with engineering advances in deep learning and big data leading to innovative designs for HCI.
In recent years, design research has approached machine learning as design material,
with conceptual frameworks envisioning divergent and socially-situated designs for ML.
Yet, design researchers stated that "ML challenges the general idea of prototyping'',
due to the complexity of modern ML engineering, and the costly infrastructures required to build it.
What practices, then, could stand as alternative approaches for ML prototyping?
Our work seeks to explore code-based art practice as material approach to ML prototyping.
In code-based art practice, artists approach ML not only as a technique to engineer logical concepts.
They also approach ML as a computational material,
whose raw properties can be crafted and experienced within installations, performances, and other embodiments.
Many art works have shown how code-based art practices can interfere engineering with design,
and crucially, allow for transformative phenomena, such as artists becoming engineers,
algorithms living things, or humans computational material.
What research methods, then, could be used to attend and respond to such interferences within ML prototyping?
In this paper, we propose to explore diffraction to frame analysis and production of art-based ML prototypes within design research.
According to Karen Barad's material feminist theory, diffraction intends to displace reflection,
which assumes pre-existing subjects and objects interacting with each others,
as a dominant model of inquiry.
It does so by assuming that humans and non-humans are bound together within complex socio-material practices,
which are fluid and ever evolving;
and refers to their entanglement as intra-action, as opposed to interaction.
We believe that diffraction may help illuminate the entanglements of social and technical dimensions of ML,
living at the boundaries of engineering and design.
We propose to use the term "diffractive art practice", introduced by Jane Prophet and Helen Pritchard,
to describe our art practice with ML computations.
Our contributions are threefold.
First, we present two artistic works made by crafting ML materiality.
Second, we explore diffraction as conceptual and methodological framework for ML design.
Third, we identify five socio-technical conditions to prototype ML by interfering engineering with design.
Our first art project is called somasticks.
It is a new interface for musical expression that was crafted between 2017 and 2019 at IRCAM by the first author,
who acted as computer music performer and designer.
Contrary to many applications, somasticks do not employ ML to classify gestures.
Rather, it lets performers physically interact with a fluid ML model, which constantly learns to generate sound,
depending on both previous and current human movement.
In this project, we adopted a linear approach to ML prototyping.
First, eliciting embodied concepts related to drumming, then, crafting an ML prototype, and eventually,
using diffraction to analyse expert performers’ perspectives on somasticks.
We voluntarily skipped engineering to focus on ML prototyping.
We used an interactive unsupervised learning tool, "XMM",
to rapidly craft ML, based on small movement data, and a shallow Gaussian Mixture Model.
We refer to our paper for details on implementation.
Somasticks were experimented in a hands-on workshop involving six expert performers.
We used diffractive analysis to read through transcripts of performers' audio-visual data,
and specifically illuminate differences in performers' embodied perspectives on ML.
Globally, our results demonstrate how diffraction led to prototype ML in ways that valued computational properties of ML,
such as probabilistic uncertainty, real-time adaptation, and somaesthetic qualities,
over its optimal engineering.
Motivated by this project, we decided to leverage diffraction throughout the prototyping of "The Appprentices",
a collaborative robotic artwork started in 2020 at EnsadLab and Orange Xdlab.
In this project, the first author acted as computational and sound artist, working with ML as creative material.
The interdisciplinary team included one media artist, one UX designer, one product designer,
one anthropologist, one cognitive psychologist, and two engineers.
Here, ML prototyping relied on the first author's diffractive art practice,
who moved within and beyond human and non-human perspectives to produce novel ML configurations, as we will see.
Initial meetings aimed at producing a diffractive review of ML,
in which the first author introduced material elements of ML, along with alternative design concepts for ML, and ML-based art works.
This interdisciplinary space, designed to match other team members' interests,
was essential to provoke conversations on how ML could be designed.
Parallel to this, we defined a material-centered design space for The Appprentices.
We focused on a collective of three robotic objects, sharing a common organic shape.
We thought about exploring rhythm as an interspecific behavioural modality for the objets.
We eventually elicited three social concepts to guide the design of these rhythms and robot behaviours.
We then created Wizard of Oz ML prototypes, using Max/MSP to realise abstract graphical and sonic representations of objects,
and discuss embodied aspects of reinforcement learning and supervised learning collectively.
Team members reacted positively to these prototypes, generating ideas for objects' movements,
while learning about ML computational structures and properties.
We eventually produced material ML prototypes, using a modular robotics kit, MisB Kit.
We used an interactive reinforcement learning tool, the "Co-Explorer", and XMM,
to craft ML based on small robot movement data, and shallow models.
Team members engaged their bodily sensibilities to interpret the objects’ behaviours,
including the errors and strivings of their non-optimal machine learning,
and eventually validate these ML prototypes for The Appprentices.
Overall, our diffractive art practice with ML let us frame five socio-technical conditions to prototype ML.
First, we observed that art-based ML prototypes take their whole dimension when they are deployed "in the wild’’,
that is, in tight interaction with other humans and non-humans.
Such a situational whole should be understood as the entry point for ML prototypes to interfere engineering with design.
Second, we witnessed how small data approaches allowed for intuitive apprehension of ML by performers and artists,
who were all non-ML experts.
Contrary to big data, small data opens possibilities for designing ML systems that are scaled to their stakeholders,
and that can be collectively crafted by social and material entities under-represented within ML.
Third, our projects showed how shallow models, despite having elementary structures,
were able to adapt to humans and non-humans in ways that were valued by artists.
Shallow models may thus offer designers possibilities to prototype ML without relying
on large socio-material infrastructures required by modern deep learning.
Fourth, our ML prototypes all relied on learnable algorithms,
that is, algorithms whose actions can be grasped through experience of material embodiments.
Beyond notions of interpretability or black-box,
the notion of learnable algorithm enables to highlight appropriation of ML by non-ML experts as one crucial feature for ML.
And fifth, our work illuminates how somaesthetic behaviour,
that is, the temporal forms produced by uncertainties and errors of ML computations,
helped artists project themselves into ML using their bodies and perception in addition to their rational thinking.
Somaesthetic behaviour may thus open holistic approaches to ML design,
deconstructing stereotypes of intelligent behaviours,
and including the weaknesses and vulnerabilities that make us learn.
In our paper, these five interference conditions let us sketch contours of "intra-active machine learning",
a diffractive practice that attends and responds to differences in engineering and design
to produce novel ways of becoming ML practitioners.
By doing so, we do not want to impose a static account of what intra-active machine learning should be.
Rather, our wish is to start a dialogue about the generative potentials of diffraction for design,
and how diffractive practices may help resist normative framings of ML across disciplines.
We invite you to read our paper for more details on artworks, concepts, and techniques involved in diffraction and ML prototyping.
Thank you very much for your attention!