Hey everyone My name is Aneesh Tarun. I am a postdoctoral researcher at the Synaesthetic Media Lab, Ryerson University in Toronto. I am here to present our work:
"Playing by ear: designing for the physical in a sound-based virtual reality narrative". This project was done at Ryerson University as well. In this paper we present two proof of concept narrative VR experiences...
...where we focused on sound-based physical interactions. We outline our design process, and we also talk about the findings from our evaluation that we did with one of our vr experiences.
I'll be giving you a brief overview of what the paper is about, but I strongly encourage you to read the full paper for the all the details.
Talking about the motivation for our work, Ivan Sutherland in his seminal work, the Ultimate Display, envisioned a display that would engage as many senses as possible.
And one of his first virtual reality headsets, that is, the "head-mounted 3D display" was his first step towards this ultimate display. Even with 60 years of research and development,
consumer VR, we can argue, is still primarily a visual display. There are efforts to add other senses into it. Marshall and Tennent critique...
...VR designs in an interesting way where they argue that the designs are constrained by the popular conceptualization that we see in VR MR and XR.
The way these constraints affect VR in two ways: one is that there's a visual bias...
...in these designs and the other is that there's an inherent assumption that the goal of all these designs is a complete simulation of reality.
So they question this assumption and identify a design space where we can have a deliberate consideration for the sensory world that is outside of the VR, outside of the headset.
And in this paper we explore this design space where we bring other senses into consideration when we are designing for vr.
We drew inspiration from sound-based tangible user interfaces, some of which, the examples I'm showing on my screen. Such as the MusicBottles, the sound of touch project, or the...
...cloakroom project.
The design of these sound-making objects is interesting because it involves a direct consideration for not just the digital possibilities of the space...
...but also the tangible and embodied possibilities. So we wanted to see if there are lessons within that space that we can bring into VR design.
We set out (with) two research questions, that is, (1) how might we employ lessons learned across these sound-based interactions for VR design? and (2) how might the physicality of these interactions contribute to new conceptualizations of narrative VR design?
To answer these questions, we followed a research through design approach...
...where we examined the contours of this design space and we put out our conceptual considerations out into two interaction scenarios that we created for VR.
As Gaverr points out, the purpose of research through design is not to provide concrete generalizations but rather to produce design objects that embody our conceptualization and that's exactly what we did.
We started off with an informal workshop where we invited 20 members of our lab to participate.
In this workshop, we kept aside the virtual and the digital aspects of design and focused only on the physical. We brought in everyday objects: a picture, a stuffed animal, a wooden toy.
And the participants in the workshop discussed these objects' physical characteristics and how they might be used for sound-based interactions.
These discussions led us to design two narrative experiences in VR where we incorporate tangible and embodied interactions with sound.
In our first narrative experience a person wearing the headset would listen to instructions and follow a linear narrative experience.
As you can see on screen, we drew from diegetic design and situated our objects...
...and our environment in the story world. A person engaging in the narrative was able to touch, reach, and grasp everyday objects. They could listen to audio and song and different sounds and well, they could also dance.
We created another narrative experience. This second one was a lot more open-ended nonlinear experience. Here, a person again could grab different unique physical objects in their physical space which had their own sounds. They could walk around....
...different virtual...
...zones that we had created. This led to them being able to create a new soundscape as they brought in different objects with sounds into different zones. Here, a person could walk, kneel, stand, sit, or grasp, or listen to the soundscape. We used...
...HTC Vive headset and controllers and we use the unity for creating the virtual worlds. We took the second narrative experience, which was a lot more detailed and we ran a...
...study with the eight participants. We gathered both quantitative and qualitative data. We focused on the qualitative data in our analysis and we grouped our findings into four different themes.
So first...
When we asked participants about their favorite parts of the experience...
...all the participants described some sort of a sensory experience.
With varied descriptions such as "it was calming", "soothing", "relaxing". Whereas, when we asked about their experience with the visual aspect, that was their least favorite part of the overall narrative. Which is natural, given that we...
...emphasize less on the visual aspects of the virtual world.
and
With bringing physicality to the center, that is, we ran a workshop, a VR workshop without VR. We were actually able to decenter the visual by starting off with the physical...and...this strongly informed our narrative designs...and...there was a clear link between our findings...
...observations, and what we saw with the tangible user interfaces.
Something interesting that came out in our analysis also was... when we compared the experience of a novice with more experienced VR participants...
It appeared that the previous experience of the participant with VR defined their overarching expectations about what VR is all about.
As an example, participant 7 (P7F), and I quote them here, "having used VR before I'm not used to being able to physically grab things with my hands, it's always with the remotes." So for this participant, due to their previous experience with VR, the controller had already a limited...
...limited the expectations of physicality and tactility of the interactions. Whereas participant 2 (P2F)...
...who had no prior VR experience before this...
...before they tried this out. They said, and I quote "It's very different from other games, because of all the senses that you have involved, you can touch things you can interact with them."
So here participant 2 thought that this is how all VR experiences are...that is very sensory.
So it's something interesting to, I guess, reflect upon as a designer is... the way we set a tone about the VR experience going forward, whether it's sensory or controller-based, could have implications on how VR is accepted broadly.
To wrap up, I guess for us... prioritizing the physical helped us to draw links between the current VR design and prior work that conceptualizes physical digital interactions and spaces.
With the caveat that research through design isn't about generalizability, we still think that we have some ideas for future work. There's definitely a need for more analytical work where we can examine differences of physicality across VR design.
Such as, for example, examining gestures movement and touch in VR. And when we reflect on how we use tangible user interfaces to design a VR experience, it was quite successful.
We think that there may be an opportunity to look at how the physical is considered in the VR, and the physical is considered in other domains, for example, say Internet of Things.
What if we think of VR as an IoT device -as always integrated into the particularities of the physical space. What kind of design outcomes can that have?
So i'd like to end with that.
And thank you for listening to my presentation.
And if you have any questions you can contact any one of us. Thank you.