Hello everyone. I'm Marius Hoggenmueller and I'm presenting our work on
"Context-Based Interface Prototyping: Understanding the Effect of Prototype Representation on User Feedback"
The rise of autonomous systems in cities requires new approaches for prototyping and evaluating how people interact with those systems.
For autonomous vehicles, HCI-researchers began to explore the use of external human-machine interfaces (eHMIs).
However, the evaluation with prospective users is challenging: it’s associated with high costs of real-world prototypes and potential risks to participants.
and potential risks to participants.
Therefore, researchers fall back on a variety of simulation platforms and prototype representations.
These include video-based prototypes or immersive prototypes, for example using VR headsets.
Recently, researchers have also used 360-degree cameras to create immersive prototypes, which offer high perceived fidelity and sense of presence.
In this work we investigated the question: What is the effect of prototype representation on user feedback.
We conducted a comparative, mixed-methods study of three prototype representations of an eHMI with 42 participants.
We compared a real-world VR, a computer-generated VR, and a real-world video representation.
For real-world VR, we recorded a fully functional AV with an eHMI and staged various interactions.
The chosen scenario was a shared AV service pretending to pick up the study participant.
For computer-generated VR, we replicated the scenes in Unity.
For real-world video, we presented the 360-degree recording on a computer screen.
After experiencing the first prototype, we asked participants to fill out questionnaires on trust, user experience and sense of presence.
We conducted interviews after participants experienced the first prototype and after having experienced all three prototypes.
For sense of presence, the sub-scales spatial presence and engagement were significantly higher for both VR representations.
Ecological validity was significantly higher for the real-world VR representation.
No significant difference for negative effects was found.
The interviews revealed that a majority of participants favoured real-world VR, often linked to the visual realism.
One participant referred to the reduced cognitive load so that you can actually focus on the aspects of the car.
For trust towards the AV, the ratings were slightly higher for the VR representations, however no significant differences were found.
The interviews revealed that participants assessed trust towards the vehicle based on the displayed behaviour of the vehicle.
Interestingly, in the post-hoc interviews a third of the participants stated that their overall trust was higher in the real-world VR representation.
We identified three factors: spatial awareness, realism of vehicle behaviour and realism of people and the environment.
Here, the depictions of people in computer-generated VR influenced the trust of participants towards the situation.
One participant for example said: `I was wondering what my relationship to that man was meant - it took a lot of my attention.'
Another, even asked herself: `What if I punch him.'
Later after experiencing the real-world VR representation, she mentioned:
`I didn’t feel like I wanted to hit the people because in the real-world VR environment they were understandable and they made sense.'
For user experience of the eHMI, no significant differences were found.
However, the interviews revealed that participants focused on various aspects in each of the prototypes.
For the video, participants were able to report on more of the the eHMI light patterns.
One participant explained that in VR she was just trying to concentrate on which vehicle was her's and didn’t think to look for any other information.
In RW-VR, participants further reported on having more issues with distinguishing the light colours and therefore also brought up more concerns,
for example using colour as a distinguisher for a high concentration of AV’s.
Also, contextual factors beyond the eHM and the AV system, were brought up more often after having experienced the VR representations.
Based on the findings, our paper offers guidelines for evaluating context-based interface through simulations.
We recommend the use of non-immersive prototypes for focused interface evaluations, while as immersive prototypes for holistic assessment.
Real-world representations can increase familiarity and overall trust and also uncover interface anomalies under more natural conditions.
There are several aspects to consider for the composition of scenes, for example staging interactions and consider the effect of environment and people.
For computer-generated VR prototypes, we recommend to avoid virtual characters in close proxemic zones of participants
and also avoid unnecessary details to prevent distraction by imperfections.