Hello my name is Clarice Hilton and I'll be\h presenting InteractML making machine learning\h\h
accessible for creative practitioners working\h with movement interaction in immersive media.\h\h
This has come out of the 4i project which\h has been a joint research project between\h\h
Goldsmiths University, University of the Arts\h London, Coventry University, York University\h\h
and artist studio Gibson Martelli. Movement-based\h interaction is becoming increasingly popular in\h\h
performance and artwork which is in VR as movement\h sensor technologies such as hand tracking and\h\h
body tracking are becoming more accessible these\h sensors now come as standard with many VR headsets\h\h
The artwork behind me is Mutator VRs\h Vortex, through movement interaction\h\h
users can mix characteristics of the virtual\h environment including abstract shapes and sounds\h\h
This piece was created by William Lantham\h\h
alongside mathematicians and programmers\h Stephen Todd, Lance Putnam and Peter Todd
Movement interaction in VR can be incredibly\h beneficial as you are able to create intuitive\h\h
embodied movements in VR to effect the environment\h around you. This can cause increased presence\h\h
through these sensory motor contingencies that\h you experience and this reality-based interaction\h\h
style can create a more immersive experience. It's\h also been shown to increase focus and attention\h\h
as well as emotional engagement in the work,\h however, even the most simple interaction\h\h
can be incredibly difficult to implement and may\h need machine learning. This can be very difficult\h\h
and needs expert knowledge which not all artists\h or dancers or indie games developers may have
Furthermore compelling interaction is also\h really difficult to design and also many movement\h\h
interactions can be difficult to learn inside\h of VR if it's something you're not used to doing\h\h
this can have a negative effect on immersion This\h is where interactive machine learning can come in\h\h
interactive machine learning allows users to\h choose types of data for training set, record\h\h
or provide examples of the data and pair examples\h of a label to input into an algorithm, then test\h\h
to observe the output of the system iteratively,\h editing the setup to produce the ideal result.\h\h
In movement interaction this could allow creators\h to capture complex movements simply by performing\h\h
them to the system these performances provide\h the system with movement examples that form the\h\h
training set, the movement training set is used\h as an input to the machine learning algorithm\h\h
and the algorithm learns to recognize those\h performed movements. Each movement example\h\h
is paired with an output value by the creator so\h one movement for example if I start with my hands\h\h
to the side of my face and put them rotate them\h inwards so that their end with the tips of my\h\h
fingers facing each other I could label this blue\h or one. Then another movement so if my my hands\h\h
are starting facing with the fingers facing each\h other and the back of my hands facing outwards\h\h
and they rotate so then the fingertips are facing\h towards the sky this could be labeled two or black
The system can then recognize those movements\h through a live input into the system so\h\h
there's a live input of data showing that I'm\h doing the first action or the second action\h\h
the system can then recognize those\h movements and get the appropriate output,\h\h
this output can then be used to trigger\h anything in your scene or drive anything.\h\h
Interactive machine learning allows for\h designers to bridge the gap between ideation\h\h
and implementation stages whilst offering all\h the benefits of exploiting our tacit embodied\h\h
knowledge in the into in the design process\h this melds together ideation and implementation
So this is why we decided to create InteractML,\h InteractML is an interactive machine learning\h\h
tool for movement design specifically for\h dancers artists and indie games developers.\h\h
It has a node based design and you need as little\h as seven nodes to be able to make a working graph.\h\h
It's input agnostic so any sensor input\h could be used as input or any other values,\h\h
however, we've designed it specifically with VR\h in mind and therefore as well as the node-based\h\h
we also have a vr interface which\h allows you to record examples and\h\h
observe the output whilst inside of the headset
The idea behind the tool is that we\h want to create a working understanding\h\h
of machine learning so that users can use it to\h implement their movement interaction design but\h\h
it doesn't necessitate a deep understanding\h of algorithms behind it. Our design approach\h\h
is interactive and experimental\h and has a sense of play\h\h
we want there to the playful nature enables people\h to take risks and their movement interactions.\h\h
They can rapidly and easily refine their\h movement interaction designs based on\h\h
the feedback that the system is giving
Alongside the the tool we've also developed\h a movement ideation theory and methodology\h\h
on how you develop your movement interaction\h ideas. Here's a little bit of background to that.\h\h
Soma design approach and basis the effective\h and embodied dimensions in the design process\h\h
the approach suggests the framework for designing\h bodily based interaction based on some aesthetics\h\h
the approach encourages a slower process that is\h actively reflective of lived embodied experience\h\h
and the tacit understanding of movement\h and interacting the approach is also\h\h
inspired by Sheets-Johnstone's insight on the\h primacy of movement that movement and emotion\h\h
are expressively interdependent and draws from\h the ideas of the field of affective computing.
In our process we've been doing something that\h we called body-storming which is a practice of\h\h
designing movement and acting them out. Following\h from Caramiaux et al our methods uses an adapted\h\h
form of the critical incident technique a\h procedure that elicits designers to recall a\h\h
recently lived memory from their everyday lives\h and apply it as designed as input for design.\h\h
In our process we get people to think of a of\h something or movement that they've done in the\h\h
past 24 hours that is in some way interesting or\h different. We then get them to match this with a\h\h
superpower and then get them to change the\h movement in certain ways making it bigger\h\h
smaller faster slower go wrong.\h This play enables people to\h\h
really think like reflect on themselves and\h take risks without dwelling on the outcomes\h\h
we've used a user-centered design approach\h running workshops hackathons and residencies\h\h
this has involved the movement ideation workshop\h and the body storming followed by learning about\h\h
the tool through tutorials and exercises then\h onto prototyping and presenting their work back.\h\h
Insights from the interviews and observations\h are fed into the design of the tool\h\h
we've iterated on how the tool works and\h what features we've implemented in it through\h\h
through outcomes of these workshops. Originally\h these workshops were going to be done\h\h
all in person but of course it's been during\h the pandemic and we've moved them all online.\h\h
This has been a really interesting experience as\h it's meant that we've done these ideas ideation\h\h
sessions in VR in the in the image behind me\h you see a workshop that we ran in Rec Room
We've also done a longitudinal study which was\h the residencies where we had artists work with the\h\h
tool for three months and this gave us a greater\h understanding of how the use of the tool develops\h\h
and how the understanding of machine learning\h and how to use the tool develops over time\h\h
many of the artists continue to use InteractML\h in their work which has been a great outcome.\h\h
We've also observed a cross-pollination of ideas\h where as well as their ideas feeding into the\h\h
tool. Using the tool has also affected the design\h of movement that they've implemented into the\h\h
system, so as well as iterating over there how\h to design the setup of InteractML they've also\h\h
effected their iteration on how\h do they design their movements.
Thank you so much and I really\h look forward to your questions