Hello and welcome.
So I'm going to present our work "Losing its touch: Understanding user
perception of multimodal interaction and smart assistance".
My name is Margarita
and I'm going to present this work also on behalf of my colleagues.
Veronika Krauß and Dennis Lawo and Gunnar Stevens, we are all part of the research
group "Verbraucherinformatik" and are researching at the University
of Siegen and Institute for "Verbraucherinformatik".
Some of you or many of you might already have experience with Alexa as an IPA,
either embedded on a standalone device or other IPAs embedded on your smartphone.
And similar to super Apps, IPAs are evolving from a set of main functions
to an expansion by third party applications.
In Amazon's case, this is called skills and for Google assistants,
they are usually call actions.
Nonetheless, as prior research shows, there are still many NLP issues
and due to high expectations of speech as a natural language modality.
There is much disappointment from side of the users.
And this leads to only a limited use or even a nonuse of the scope of functions.
So this brings us to our first question
"What are users expectations and mental models?".
towards the orchestration of skills
regarding its operability and functionality?
Further, we wanted to investigate how Multimodality
might be and solution to support voice based problems and enhance the experience
as Oviatt et al.
already in 2004 described the advantages of multimodal applications
could improve usability and increase transparency, flexibility and efficiency.
Besides the work on mobile IPA and chatbots it is still unclear
how the potential of multi componental experience and multimodal
experience might be realized by designers and appropriated by users so far.
So our second question was how does a multimodal interface
design support users, multi componental experience and interaction?
Therefore, we did an observatory study
conducted in a laboratory setting to really observe
and listen to the interactions between users and the echo Show.
As the Echo Show has
the possibility also to visualize
some of the functions and maybe to help users better interact.
All the sample was resembling the demographic distribution
of Germany in 2019 and we had 20 participants equal
in genders, and each session lasted for 60 minutes on average.
Afterwards we thematically analyzed and iteratively coded all our results.
As you see, the participants completed each ten tasks
that were distributed upon five selected skills.
The amazon skill was somehow always in the middle.
I would say, as Alexa is the main orchestrator of all skills
and sometimes some of the Amazon own skills are triggered.
And also as you see, there are brand specific skills
and some of the modalities and most of the skills
on the modality of both visual and vocal interface.
But one skill was a voice- only.
So based on our lense of an IPA as an orchestrator of skills.
We want now to focus on our new observations and what features
designers of IPAs and skills could learn by that.
Based on our insights were derived four design
implications that I'm going to present now.
So towards a voice-first design approach for skills
was based on that that participants really tried past interaction patterns
they learned from the original source of application or website.
That means, the structures, ithe signifiers, as they were familiar
with from the, for example, chefkoch.de recipe website.
Providers of atomic data and unified recipe structures
can easily reuse them and adapt them to and device
like a Echo show, whereas user generated content was much more difficult
since it has to be explicitly cleaned and formatted for those devices.
And we assume that power users of distinct services who have a strong connection
to the brand experience, will need longer
to adapt to voice- first services.
Therefore, we really have to rethink the content and functions
from previous digital services as components.
So the same as with mobile first design.
We have to rethink how this fits
on the screen and how interaction is used to.
And in this case, we have also to be aware of that.
The IPA is responsible to guide users through
to infrastructure and orchestrate between all components.
Further, most participants were really curious about the potential
of having more advanced features, really supporting the daily routines
and exceed the functions they already know from the voice-
only devices such as the Echo Dot.
Therefore skills are understood as an inherent part of Alexa
and are not really recognized as third party add ons,
although of course some participants
encountered some of those add ons already with the Echo Dot.
The skill scope was compared to daily routines and also offline experiences.
That means like browsing through promotions or creating shopping lists.
This was always the match and in a sense of efficiency and convenience.
So super apps already showed that it's possible
to exceed the capabilities in an integrated solution
by offering more skills and components that can interact together.
So they really have to be based on specific use cases,
routines, context that are familiar to the users, and a clear communication
of the skills scope, what is possible and what not, and
ensure the interoperability
of all components to really match users expectations, practices and preferences.
Switching context was a massive issue for all of our participants.
So it's really important to differentiate
between the global agent wide and local skill-wide contextual hierarchy.
Because this was invisible to participants.
they were really insecure about activating and switching skills.
Also activated unexpected skills
like the aroma designer that no one heard before of it.
And also participants had to familiarize themselves
with English words like skills.
Did not recognize that some vocabulary
was chosen to trigger specific functions and also skill names.
were sometimes hard to access because of the use of brand names
or spelling out website name.
So one solution might also be to support users to customize their IPA
and also just to simplify the access to settings.
Because some of the participants
tried to find out the settings on the device but
did not think of using the smartphone to do so.
And also
we have to ensure a conceptual and infrastructural consistency
to reduce the risks to mix up commands
and to signify the position of the user.
Our last implication concerns the appropriateness of modalities,
so most of the participants were really critical of the relevance
of the information presented on the device, since they didn't
want to see some mood pictures.
But really wanted to have shown the information
they need to browse recipes or to use their respective skills.
This resulted in interactions that were perceived as bothersome,
and they criticized the lack of the visual feedback channel.
Also, some of the visual presentations sometimes violated
the basic design principles for graphical user interfaces,
and all in all, this strengthened the burden of interpretations
and establishing mutual
understanding since users really had to make an effort
to understand what the device was telling them.
So in future designs, modalities require
respective signifiers to allow for intuitive interaction.
The same goes for a visual as auditive signifiers.
And all in all, the design has this great potential to compensate
for the lack of an IPA's intelligence,
to interpret all the information right by having the right
dialogs, the visual support, trying to communicate emotions with the
channels that we have.
Our findings show that normal interaction issues
with IPAs like NLP issues still exist,
but also that conceptual issues emerge from a range of skills
which lack the appropriate data models, and do not make proper use
of the additional screen to interact with or depict information.
So to ensure those multi-componental and multimodal experience,
we have to balance all elements to each other.
And in general, all participants expected Alexa to orchestrate all skills
and support their activities as well as switching between skills.
Thank you very much for listening and watching,
and if you're interested in more details, take a deeper look into the paper.
Or if you have any questions, just let me know.
I am happy to discuss and I hope you found
some interesting insights that you can use in your further research.