Good morning Everyone, we are Giorgia Petri and Pauline Vierne
and today, we are going to be presenting an overview of
the work in progress paper “An Interactive Garment
for Orchestra Conducting”.
We are respectively interaction designer and e-textiles designer
based at Berlin University of the Arts,
and we were part of a wider interdisciplinary research team
with universities related to Einstein Center Digital Future Berlin.
The project is a two-month collaboration between orchestra
conductors, hardware engineers, machine learning programmers,
a fashion designer and a jazz and electronic musician.
As a first development step, we set up a generic
wireless IoT wearable system architecture.
A micro-controller, with built-in Bluetooth Low-energy support,
receives data from e-textile sensors
and communicates these data to a Machine Learning workstation
based on Wekinator and a digital audio workstation.
We talk about an orchestra conductor's movements.
I suppose we may all imagine quiet and repetitive gestures.
The real picture though is that,
from our conductor's perspective,
some gestures were considered conventional
such as rhythm patterns and closing signal,
while others were used as a more personal expression
and with distinctive purposes,
for example preparation for entrance
or getting special attention from a musician.
We set up weekly rehearsal sessions
as part of our research methods.
Those rehearsals were key moments
to observe and record the conductor's gestures.
The observations in these sessions included video recordings,
drawings and sensor data recordings which were further analysed.
Iterative prototyping was used to assess
material functionality, aesthetics and interaction.
This slide shows our first session with a simple
stretch sensing layout based on foreseen joints motion.
We refined possible sensors' positions with paper tape placed live
where deformation was observed on the moving body.
The knitted tube was quickly left out as a garment shape
but was used here for exploratory motion and invisible conductance.
Regular rehearsals allowed us to experiment
with sensing materials and techniques
-here is a combination of stretch and pressure sensors-
while we kept looking for the best sensor layout
in order to improve recognition with ML.
As a first step we used machine learning as a creative tool
by investigating different approaches with supervised algorithms.
The first approach tested with the conductor was classification.
The aim was to define five movements that could be labelled
and therefore recognised by the system.
The algorithm we chose to use was the K-Nearest-Neighbor.
The conductor was then asked to select five key movements
while conducting a piece.
These five movements had to be well defined and distinguishable
allowing for stable and accurate model training.
The conductor then wore the jacket with the textile sensors
and assume static positions that were
representative of the movements.
These static positions were recorded and used
as a training data set to generate the model in Wekinator.
The conductor should be in the position before the recording starts
and stay in that position until the recording is stopped.
As we needed not only to recognise positions
but also to control continuous values with body movements,
such as filters or volume, we chose to implement
an additional part in the system that handles interaction.
In this case we used a linear regression method.
The advantages of this approach are the control
of a continuous target value,
so the output could be outside the training set,
and the absence of rigid, predefined categories.
We used the same five movements we recorded for
the classification method, and for each movement
we recorded data of the initial position,
one or two intermediate positions and the final position,
when the movement was at its maximum.
In this way we created a consistent
and reliable training data set
in order to have an accurate and precise control
of the values generated as output
controlling the different sound parameters.
While we worked from the beginning with available
stretch piezo-resistive materials for functional prototypes,
a parallel track during this project
was the use of In-Situ Polymerization.
This is a DIY method -documented and published
by Honnet et al. 2018- that allowed us to make our own
piezo resistive materials from scratch
in order to create custom sensors.
First experimentations were focused on stretch sensing.
While this wasn't completely convincing,
we also polymerized a wide range of other materials
such as felt, non woven, spacer fabrics or foam proving
the meaningfulness of such technique for pressure sensors.
The shoulder pads were developed for
and implemented in the final jacket.
For the final tailored interactive costume,
a see-through material was chosen from the fashion perspective
to display the sensing system located underneath.
As its rigidity acted against our initial work on stretch sensing,
we shifted our approach in order to use its stiffness
to apply pressure, notably on shoulders and back.
We hope that this presentation gave insights on the project's
methods and developments from our perspectives and roles.
Additional references regarding the state of the art are detailed
in sections of this paper that we didn't cover today.
Thank you for watching, we are here to answer your questions.