I’m presenting EmbER: A System for Transfer of Interoceptive Sensations to Improve Social
Perception.
I’m Caitlin Morris, from the MIT Media Lab, and my co-authors included Valdemar Danry
and Pattie Maes, also from the MIT Media Lab.
The experience of social connection is essential to wellbeing.
A sense of social connectedness depends on interpreting communication cues from conversation
partners.
In face-to-face interactions, we typically have access to a wide range of nonverbal cues
from various communication channels, including environmental elements like haptics and olfactics,
physical positioning cues like kinesics and proxemics, and many others.
In remote interactions, even in rich media environments such as AR and VR, these channels
are largely removed or reduced, leading to decreased feelings of social connection.
Precise replication of the body in virtual environments suffers from challenges such
as uncanny valley effects, often creating further emotional dissonance.
Rather than restoring existing communication channels, we explore whether adding novel
communication channels can increase a sense of social connectedness.
In particular, we evaluate the impact of transferring interoceptive sensations (breath and heartbeat)
to another person through haptic and audio feedback.
EmbER, or Embodied Experience Remotely, is designed as a multi-channel system fusing
a video channel of a person speaking, with augmented channels of heart rate or breathing
rate reflecting that person’s physiological state in real time.
In our experiment methodology, scripted data designed to simulate a narrator’s physiological
arousal state, is fed to the control software to drive outputs.
In the Haptic mode, a microcontroller is driven by the control software to vary haptic pulse
speed, conveyed through a custom wearable wristband with a vibrating motor.
In the Audio mode, an added layer of Brownian noise is volume-enveloped to create a rhythmic
breathing pattern, overlaid on the voice audio from the speaker.
Both methods were calibrated to sit just below the level of conscious perception, not requiring
or demanding attentive thought.
In a calibration phase, vibration and auditory noise were played through the wristband and
headphones, respectively, to determine a feedback level which would be just below the threshold
of perception for each participant.
Participants were told that they would experience sounds or vibrations similar to a breath or
heartbeat, but were not told that this specifically belonged to the narrator.
The video sequence used in our study consisted of three pairs of videos, each pair followed
by a short survey.
In each pair, a short video of a nature scene played first, to act as an emotional neutralizer
and allow the capture of an affective baseline state.
Following the nature video, the participant watched a 5 minute video of a filmed narrator
describing an emotional experience.
After all the video cycles were complete, the participant engaged in a phenomenological
interview to understand how the participants experienced the narrators, and how the experience
was influenced by EmbER.
To evaluate the impact of EmbER on social perception, we deployed a mixed-methods approach
which incorporated: self-report measures, physiological data capture, and the post-study
interviews.
In our assessment of participants’ post-video survey results, we saw the most notable results
on the item related to feeling of Connectedness.
We measured significant effects of the Breath condition on participants’ feeling of connectedness
toward the narrator, measured on a Likert scale.
We also saw an increase in the number of physiological arousal events, or a significant change in
heart rate in a short time interval, as compared to the control video condition.
We organized the phenomenological interviews into a series of themes which arose in discussion
with the participants.
Participants noted that their level of conscious perception of the feedback device was related
to their feeling of connectedness toward the narrator.
Many participants reported hearing no augmented feedback in the audio channel, despite the
increased effectiveness of this modality on reported connectedness, indicating that the
effect was primarily unconscious.
Similarly, many participants reported being consciously aware of the haptic feedback,
and that their awareness created a sense of disconnect, as they did not fully interpret
the sensation to be their own.
This is likely related to the lack of significant improvement in the haptic condition, and can
be addressed in future work.
We also observed participants engage in reflective processes to determine the self-other relationship
to the device.
Most participants intuitively understood that the sensory information belonged to the narrator,
despite not being specifically told whether the sensation was meant to mimic the narrator
or their own body.
We began to see trends towards increased physiological synchrony between the participant and the
simulated narrator data, especially in participants who reported a stronger feeling of connection
via survey.
Here, in a cluster of the three participants who reported the highest sense of connection,
we begin to see patterns of synchrony between the simulated heart rate (in orange) and the
participant’s real heart rate (in blue).
We compare this to participants who did not report a strong sense of connection, where
few markers of synchrony are evident.
We look forward to investigating this trend further in future work.
In this work, we presented EmbER, a method for augmenting virtual interactions with a
remote partner’s interoceptive sensations to improve emotional connection.
We showed that the sensation of a narrator’s breath, added as an audio channel, caused
participants watching the narrator to feel an increased sense of connectedness as compared
to those without the sensory augmentation channel.
In future work, we would like to extend this platform to realtime interactions between
remote partners.
We also hope to further develop the sensory feedback modalities, including the haptic
feedback modality, for a more intuitive sensory experience.
We hope to highlight the potential for augmenting digital interactions with shared interoceptive
data, and explore possibilities to improve one’s sense of social connection, which
we find a critical component of wellbeing.
Thank you.