Welcome to my talk about StringTouch -- From Sting Instruments towards new Interface Morphologies
My name is Beat Rossmy and I am giving this talk on behalf of my co-authors Sonja Rümelin
and Alexander Wiethoff.
When we talk about tangible user interfaces and the design of user interfaces in the HCI
context.
Most researcher start with familiar mental models and metaphors to create understandable
user interfaces.
If you think about understandable user interfaces and familiar mental models you could think
of string instruments for example.
But if they are really a part of many cultures and a cross cultural source and inspiration,
they are manly unconsidered in the HCI context.
And if they are considered, they are not considered as a whole, but they are rather considered
as the string itself or it is the cord which is taken into account.
So for example user interfaces can attach string to persons, or we explore how people
can manipulate the strings and the cords to perform actions, but if the whole instrument
is considered it is rather about how to improve the instrument, how to make it more expressive.
But chordophones, string instruments, are a rich resource.
We have a lot of different designs, we have lute type of instruments, zithers or harps.
And the shape, the morphology of the instrument, is defining how we interact with them, how
we use them, how we play them, how we position them towards our body.
We investigated further and found that some of these, and especially the zither types,
have a huge correspondence to modern technology.
Namely the interaction with touchscreens or touch surfaces, or like tabletop devices.
We took that as a starting point and asked ourselves where do we actually interact with
touch screen but it is not the best idea or context for doing so.
We found the car where more and more touch screens are implemented.
But touch screens have a big problem, we lose the passive haptic feedback of the normal
user interface.
So, for example an old car interior would have a dedicated knob for volume and a dedicated
knob for temperature of the car.
And I could learn where it is, I could feel it.
But on the touch screen I have to look, I have to see, I have to search the element
before I can use it.
So obviously when I am in a car and I am driving I should probably watch the road rather than
the interface.
We tried to adapt the string context to this shape of the dashboard and we created a flexible
dashboard surface, including and incorporating string like shapes along the whole dimension.
Via interacting with these strings you could control the media system.
After that we were interested, probably there is more to it, there is more stuff you could
do with strings, we should probably learn more from string instruments.
So we did a research about interactions typically performed on string instruments and we collected
an interaction vocabulary and we clustered it and found different interaction types.
For example gestures corresponding to the excitation, the stimulation, of the string,
as well as gestures which are defining the pitch.
And there is as also an overlap like actions which are defining.
Pitch as well as excite the strings.
So, with this knowledge we were now interest to explore these gestures to a certain extent
in a generic peripheral device.
So, we created a new USB generic peripheral device where we had a similar approach, we
had these ridges along the shape and we were interested in what can I do with it, what
are the possibilities of the morphology.
I want to show you a quick video demonstration, where we actually show you what it can do.
VIDEO IS PLAYING
As we demonstrated, there are different types of interactions which we implemented.
One idea we took from string instruments is that most of the interactions are not really
based on an absolute position.
So there is not one specific location on a string instrument, but rather it is more about
the context.
So for example playing a chord and moving the chord.
It stays the same chord shape.
You change the pitch but it is still the same configuration of your fingers.
So we took the concept of pie-menus.
You touch a string and around your finger position you have generic UI elements popping
up, which you later can interact with.
You can push and pull.
This is a gesture which is taken from the Ghuzeng, the Chinese zither instrument, to
for example select or deselect or increase or decrease values.
We also explored with strumming gestures, strumming along the ridges, the string or
sliding along them.
To actually track these different kinds of interactions, we had to differentiate touch
from deformation.
To detect the touch we captured the capacitive touch via capacitive touch areas in our PCB,
and to track the deformation we included magnets in the silicone ridge and if you push into
the material you change the distance from the magnet towards the hall-effect sensor
and you increase the voltage which you can measure.
And the same goes for pulling, so if you pull you increase the distance between magnet and
sensor and you have a lower value you can measure.
We designed our PCBs in a way that we can easily make a larger or smaller interface.
It was a modular design and you could daisy chain multiple of these and then in the silicone
layer we had all these magnets for each hall effect sensor and a teensy lc board to handle
all the data and generate USB messages which we then process on the computer.
We were also interested in evaluating the haptic properties of the device.
Could the design, the affordance of the silicone ridges actually influence what kind of movements
and interactions we could perform with it and also could we differentiate different
shapes and textures, just by feeling them.
We did a user study where we had blind test with different shapes and textures and we
tested different widths and heights and strength of the material.
And we saw that there is an influence of design on the interactions which are possible.
So, for example a really low design is hard to pull, because it is hard to grab.
Whereas a really thick design is hard to bend and so on.
So the design is influencing the interactions we can perform and this is really interesting
because we then can in theory design our interfaces that they support the interactions we are
intending the users to do.
So currently we are exploring different things.
We explore flexible PCB designs, breakable PCBs directly casted into the silicone to
enable free form, free shape interfaces.
Another thing which we are interested in is going back to specific application scenarios.
One would be example to make use of the capability of the users to differentiate the strings
and to perform interactions with the strings without direct visual contact.
We could for example include them in the back of devices to enable discreet interactions.
Or we could for example use cylindric shapes and wrap around our silicone surface to enable
multiple users from different sides to interact with the interface.
As a conclusion.
String instruments are a rich resource of inspiration.
We found different types of interactions, different types of morphologies we could use
or reuse.
We also found that the technology we developed, the capacitive touch in combination with the
magnets and hall-effect sensors, could provide a way to avoid the Midas touch problem.
Because we can differentiate touch from deformation.
So, we could have a hover state and an interaction state, which is a big problem on touch surfaces.
Further, we saw that people can feel differences in the different designs and this could be
sued to reduce visual attention, which is needed to operate the interface.
So thank you very much for you attention and if you want to get in touch feel free to contact
me.
So please enjoy the rest of the TEI conference, have a nice time.