Hello, I’m Jianing Zheng
I am PhD candidate at Media and Arts Technology Centre at Queen Mary University of London.
I will be presenting the work
"Material Matters: Exploring Materiality in Digital Musical Instruments Design"
This work was completed together with my co-authors Nick Bryan-Kinns and Andrew McPherson.
Digital Musical Instruments (DMIs) are musical instruments consisting of a physical interface and computer-based synthesis of sound and other feedback modalities.
As Human-Computer Interaction (HCI) and computational technology in general become more tangible,
research on the materiality of DMI physical interface design has also gained more attention.
Previous research shows musicians' expectations of the gestural and sonic interaction of an instrument
are fundamentally linked to its material properties.
DMI designers and researchers have explored the use of non-rigid materials in musical instruments and interfaces.
Previous research indicated musicians' affirmations and expectations on deformable DMI,
but the material properties of deformable DMIs are underexplored.
Similarly, exploring the evoked meaning of those materials in DMI design apart from their digital interactivity has been underexplored.
This paper presents a study investigating materiality in digital musical instruments design,
focusing specifically on deformable materials as they offer opportunities for physical interface design and materiality research.
We take a step back from pre-made objects to focus on the materials themselves and ask:
How do DMI designers explore and question possible and preferable futures of deformable DMIs through material probes?
To explore this question, we present the method and results of a study with three aims:
1) To understand how DMI designers perceive material qualities in general and in the DMI context,
2) to investigate how they interpret the properties of the materials and interact with deformable materials in the musical context, and
3) to analyse their understanding and expectation of materials in DMI design.
There were 15 participants participated our study.
As the study took place during a period of lockdown in which in-person social interaction was heavily restricted,
we structured our study to take place remotely,
with participants taking part on their own and in their homes.
Participants were connected to the facilitator online.
Before the study, 12 material samples were sent to participants by post.
We made five types of non-rigid materials for participants to use in this study,
which include fabric, foam in low density (softer), foam in high density (more rigid), rubber, and copper.
The materials samples were made in two categories:
raw material (i.e. on its own) and constructed material combinations.
We loosely structured our study following Material Probe approach and questions.
We guided participants to focus on the wholeness of the material in the first part of the study
to explore how they "perceive, value, and articulate material qualities of objects from their memories".
We then invited participants to generate speculative or hypothetical instruments based on the materials after playing with the materials.
Participants were then encouraged to talk about their previous DMI design experience
in relation to the materials and views on deformable DMIs in general.
After each part, a semi-structured interview was followed.
All participants were recruited using academic mailing lists and the researchers’ social networks.
Participants were recruited in two groups,
the first group (N=7) was recruited in China, and the second group (N=8) was recruited in the UK.
Both groups followed the same study methodology as described before.
The interviews were conducted in Chinese for group 1 and in English for group 2,
which allowed participants to express themselves fluently.
Five themes were identified in the thematic analysis of the interview data and the video observations.
A list of themes and codes is provided in this Table for the reference of coding process.
These themes were common to both participant groups.
When participants were asked to generate speculative or hypothetical instruments,
the most frequently mentioned design thinking approach was based on the material properties.
Nine participants commented that the texture and haptic feedback of the material gave them ideas of its control mechanism in music.
As expected, different types of material afforded different types of gestures that corresponded with the material properties.
The three most common patterns of interaction with different materials appeared:
finger-based gestures, hand-based gestures, and body gestures.
Here, we also find the different gestures people used with the same material sample
and what mapping strategies they expected to have in DMIs.
This theme emerged in the second part of the study — playing with the material samples.
Eight participants suggested that the interaction implication of the object was mainly based on their previous experience with existing interfaces.
These evidence suggest the materials implied their interaction techniques and functionalities from two aspects:
i) the properties of the material, such as foam and fabric, are soft to be squeezed and stretched,
ii) the prior expectations of construction of the object, such as the smaller sized square foam suggested a button.
Participants talked about the flexibility of using the material samples in specific musical contexts and their concerns with its use.
As each deformable material could allow multiple gestures,
participants discussed that designing intuitive and efficient musical gestures with such materials might be problematic
as they need to consider the difficulty of playing the instrument in performance when designing it.
Participants mentioned that the precise control of deformable DMIs was challenging for performing on the stage.
To generate a good quality of music with DMIs,
musicians need to prepare, rehearse and practice before the performance.
However, some deformable materials were not very stable and robust.
Overall, this study presents an approach to investigating materiality in DMI design through the material lens.
Participants' initial perceptions of the materials influence their interpretation of the meaning of materials,
thus influencing their design ideas developed with the materials.
Our findings indicate that the unfamiliar materials might help DMI designers to focus their thinking on the materials themselves
because they do not fall back on previous experiences or focus on technical constraints.
We found that participants from engineering and technical backgrounds were less concerned with
how a material would look like in the final piece and more concerned with the technical implementation.
When DMI designers consider the actual musical activity when interpreting the materials,
there is a potential change in their thinking.
One limitation of the study design is that
in part 1 participants were invited to talk about their initial ideas of
how they imagined the material samples could be used in a musical context
- which might put people back into a technical mindset.
Due to the technical challenges of designing with deformable materials,
discarding technology constraints may help DMI designers focus on the possibilities of deformable materials in music.
Future research needs to explore how DMI designers might take such design decisions and approaches to the implementation phase.
This paper reports on a study with fifteen digital musical instruments (DMIs) designers
investigating the evoked meaning of material properties in a musical context beyond their digital interactivity.
Our results suggest there is a strong connection between the tactility of an instrument and the imagined sound.
We believe there is great potential for further refining and applying the material exploration approach in digital musical instrument design.
Thanks for listening.