Hi, my name is Chris Zimmerer
and in this talk i will present our case study
on the rapid development of
natural and synergistic multimodal interfaces
for XR use-cases.
First, let me give you an example
of how such multimodal interfaces look like
in the context of a virtual- and mixed reality application,
or XR for short.
Erstelle eine Pyramide.
Mache das Objekt rießig.
Färbe es rot.
Zerstöre die Falle.
Gehe dorthin.
Zerstöre die Falle.
In theory, these interfaces offer high expressive power
and are deemed highly suitable for XR applications.
However, the currently prevailing interfaces are unimodal,
as illustrated in this example.
They can be developed relatively quickly,
even by non-experts
since software platforms like Unity or Unreal
provide plugins with ready-to-use interaction techniques
such as menus and 3D point-and-click.
A comparable tool support for
multimodal solutions is yet missing.
Their development is very costly
and requires expert knowledge.
Consequently, there is very little user-centred research
with functional multimodal interfaces,
only few practical design guidelines,
and virtually no such interfaces in use today,
despite their potential benefits.
Here, better tool support is key
to making multimodal interfaces accessible to a wider group
of researchers and designers.
In this case study, we report on the toolchain
we develop and use to create
multimodal speech and gesture interfaces
for XR applications.
We also analyze our toolchain in terms of
its suitability for rapid development
based on years of application in research
and academic education.
At the center of the toolchain is our fusion method:
the concurrent Augmented Transition Network.
This fusion method is, on the one side,
able to analyze input from multiple concurrent modalities
and derive a joint meaning.
On the other side, it provides an integration concept
to communicate with XR applications.
It can identify the virtual object a user is referring to
and change its properties to specified values,
like the object's position to a target location.
In order to define interfaces,
our fusion method implements a declarative approach.
Interface developers can simply define multimodal commands
as transition networks
using a dedicated description language.
In contrast to machine learning based approaches,
it does not require any training data
or time costly optimization of learning parameters.
Designed interfaces can also be graphically visualized
for debugging purposes.
The analysis of our toolchain is based on three use cases:
the development of demo applications,
the development of functional interfaces
for user-centered research,
and its application in academic education.
Over the years we developed several demo applications.
These helped us to continuously develop our toolchain
in terms of supported features, performance, and usability.
The following video shows excerpts from some of them.
Create a planet.
Paint it red.
Rotate it around that sun.
You travel here.
You attack the archer.
What is this?
These are threshing clawers.
They would always appear in packs of at least 20 robots.
We also implemented functional
multimodal interface prototypes
for our user-centered research.
For example, we compared a multimodal
speech and gesture interface
with a state-of-the-art unimodal menu-based interface
regarding their impact on the user’s creative performance
in a VR design application.
These comparisons show that the interfaces
we created with our toolchain
were either on par or even outperformed
their respective unimodal comparison partners.
Finally, we use our toolchain in academic education.
In the following you will see a clip
showcasing short excerpts from a few of the applications
students were able to build with our toolchain.
Create a tomato there
and make it bigger.
Move the big tomato there.
Select this planet.
Make it this big.
Texture that cube with stars.
Select that firewood.
Move it there.
Create a campfire.
These applications were programmed as part of the
mandatory master course “Multimodal Interfaces”
in the Human-Computer-Interaction program
at the University of Würzburg here in Germany.
In the scope of this course,
students with vastly different backgrounds ranging from
psychology over media communication to computer science
use our toolchain to implement simple VR applications
with multimodal speech and gesture interfaces.
Based on these three use cases we analyzed our toolchain
in terms of its suitability for rapid interface prototyping,
especially in terms developer usability
and development time.
In addition, we identify remaining challenges
that need to be addressed.
Our declarative approach was well received by developers.
The description language is suitable to design
multimodal interfaces for XR
and the graph visualization helped developers
to debug their interfaces.
We estimate that non-experts can implement
multimodal speech and gesture interfaces for VR in about
45 hours when using our toolchain for the first time,
which is consistent with
iterative interface design processes.
Therefore, we conclude that our toolchain is suitable
for rapid prototyping.
But of course, there are still challenges to tackle.
The integration concept for applications
was perceived as rather complex.
A possible simplification would be
to directly integrate the fusion method
as a plugin for a concrete XR platform like Unity.
Another promising extension of our description language
is a graphical editor that allows developers
to create interfaces by simply
dragging and dropping predefined interaction blocks.
So as a takeaway:
our experiences in developing and applying our toolchain
show that a good tool support is key to making
the development of multimodal interfaces more accessible.
Enabling a wider group of researchers and designers
to rapidly develop multimodal interfaces
will hopefully support the creation
of even better interfaces for XR in the future.
Thank you for your attention and have a good day!