Hi there! My name is Emanuel Gollob, and today I'm gonna be talking about using robotics
and artificial intelligence to physically explore a space of aesthetic possibilities.
This is joint work with Magdalena Mayer and Johannes Braumann and we together did it at the
Creative Robotics research lab at the University of Art and Design in Linz. So let's get started.
Imagine two people walking into an exhibition and looking at the same sculpture. Even if
context, sculpture and time are the same, I guess both will very likely develop a very
different aesthetic judgment as well as aesthetic emotion. Empirical aesthetic research has
made a huge process in mapping all the components involved in this process. But what if we just
tackle the problem the other way around and think about a sculpture, which is reacting
on the aesthetic emotion of the perceiver or maybe even targeting a very specific neuronal outcome?
With our explorative research, we are building on a model of aesthetic experience
we are using generative adversarial networks. We are trying to implement the interaction
concept of an affect loop, and we're combining it with parametric robotic control. Each of
our artefacts, there are three, and I'm going to present them to you very soon, each of
them is consisting of an EEG device, a KUKA robotic arm, a generative adversarial network
as well as a database. So now let's get to the first device. For public interactive setups,
we are using a Muse EEG headband. It has four electrodes and is a consumer headband.
For more advanced settings, we are using an Enobio8 EEG cap with eight electrodes. On
my right side, you can see the electrode positions we are using. Also, in our paper, you can
find more details about how we process our EEG data. So let's continue and move forward
to our adapted GAN setup. For our adapted GAN setup, we are using a deep convolutional
generative adversarial network (DCGAN) combined with a physical execution. So the GAN is continuously
running in the background, and sequentially we're taking out samples and executing them
in the physical space. Whenever we are executing a sample in the physical space, we are using
a robot execution, and we start to measure the EEG feedback. Once the EEG feedback reaches
a certain threshold, defined by the targeted EEG feedback, we are gonna save the sample
into our database. Later on, the discriminator is trained on this database and is again training the
generator. As our aesthetic in the flux demands a near-real-time machinic reaction, we are
using a KUKA KR6 R900 robot. It has six axes and a maximum reach of 900 mm. We control
it through the real-time control software mxAutomation and the Grasshopper plugin KUKA|prc.
Each of our choreographies is defined in a parametric logic. So there are 256 mutable
parameters in a range from 0 to 255. And that also makes them translatable to a 16 by 16
grayscale image. Over the last year, we created three artefacts within this work. Each of
the artefacts providing different levels of interactivity. So first, I want to show you
"Doing Nothing with AI 1.0". It is targeting default mode network brain activity, as
Belfi et al. found correlations between default mode network activity and aesthetic perception,
but also it is in line with our conceptual interest in doing nothing brain activity. Here you see "Doing
nothing with AI 1.0". Whenever the receiver is putting on the EEG headband, the generator
generates the first sample, and the robot starts to execute the sample. From that point
on, we are collecting EEG data and interpreting the data compared to our defined targeted
EEG goal. Once a certain threshold is achieved, the sample is labelled and either we just
mutate it slightly, because the labelling was positive, or if the labelling was negative,
the generator is going to produce a new sample. Our second artefact is
"Doing nothing with AI 2.0". It builds on our experience from "Doing Nothing with AI 1.0". In the first
iteration, we found out that viewers untrained to movement nuances had a hard time to recognise
small parameter changes. So we extended the space of aesthetic possibilities with a parametric
soundscape as well as parametric visual scape. Here you can see a video of the experience,
and as you can see, we also tried to eliminate the context factor by creating an immersive experience.
Additionally, we let the perceiver walk freely in the space.
The next artefact we're going to present today is "Violin improvisation & Doing Nothing with AI 1.0".
For this iteration, we were working with a professional violin player. And he was improvising a violin play based
on his perception while being connected with "Doing Nothing with AI 1.0".
Someone preferring to define what arrives at the perceiver's side could take the step of defining an aesthetic
rather in a relative way than in a strictly defined way. Throughout our work in all our artefacts,
we did not save any EEG data. We only saved choreography data. And as several components
were not standardized, we distance ourselves from interpreting those data. But want to
share our observations and experiences. The outcome of the work we present here is not
an optimized aesthetic, but more an embedded interaction. So both Interactant's are acting,
learning and reacting on each other. In our work, we also realised the positive impact
of targeting unconscious brain networks. It was very beneficial for the interaction quality.
And we also realized that it was a negative impact to have technological layers visible.
Like for example, security devices or EEG devices.
Looking at future research we would be very interested to also try to target different EEG feedback.
As we mainly presented and set up our artefacts in an art and tech bubble, we would also be interested to try it with
a more diverse sample. For our work it was very beneficial to use a GAN setup as it was
very much straightforward. Still, it would also be interesting to try explainable learning
systems and have a look at how the learning process is happening in the back. Part of
our motivation was to research how brain activities are changing in parallel to digitalisation.
So it would also be exciting to do this as an exploratory study with a long-term perspective.
All of this couldn't have happened without a huge team. So we are very thankful to everyone
mentioned on this page. And it was only doable because several institutions supported us.
Thanks for that and I'm looking forward to our discussion.