- This work was done together
with Giovanni Toano and Lana Claus,
Jacob Simmonsen and Kasper Hornbeck.
Shape-changing interfaces has been a field
that's been maturing over the past five years.
We're seeing increasing examples
of shape-changing interfaces,
frameworks, we're seeing shape-changing benches and phones,
and data visualizations.
We're seeing use of studies becoming recurring
in this field, so we're seeing a lot of stuff happening.
There's still a lot of areas
that remains largely unexplored,
and we have two perspectives that we want to highlight
with this work, and that's we haven't really seen
any accounts of the design process
of shape-changing interfaces, and we haven't really seen
any very systematic analyses of existing HCI concepts,
such as affordances or metaphors.
That's what we aim to do in this work.
We have this three approach, we want to analyze examples
of shape-changing interfaces using existing frameworks,
or one framework.
We investigate which kind of high level strategies
that people use when they design,
whether intentionally or unintentionally,
such as affordance and metaphors.
What we did, rather than look at existing examples
of shape-changing interfaces, we wanted to explore,
well, what could shape-changing interfaces become
when we don't have to worry about technical limitations,
we don't have to worry about building this stuff,
which for those of you who've tried it,
it's really darn hard to do, so what we did,
we tried to recruit 264 researchers
with published papers
at CHI or TEI over the past five years
on shape-changing interfaces, or the related fields
of organic use interfaces
or actuated tangible user interfaces.
Out of those 264, we managed to recruit 21.
Some of you might be sitting here today,
I know there's a couple,
so hopefully you'll agree on our analyses of your designs.
What we did about these 21 participants
was we sent them two design tasks,
and we asked them to spend about one hour completing it,
and they could use any means they felt comfortable with
to communicating these ideas on shape-changing interfaces.
There was no demand on how many ideas you had to make,
but just sketch for that amount of time,
use whatever medium, whether drawing, models, 3D,
video or photos.
We had both sketches, we had photos, we had 3D models,
we had somebody use images that they found
to convey the experience, so a lot of different styles
of communication.
The two tasks we made, we made two sets of them.
At first, we want to see what happened
if we asked them to design.
The one group of people got a sort of very familiar
shape-changing artifact category, which is mobile phones,
is about the most explored artifact
within shape-changing interfaces.
We wanted to try a different category,
which is an artifact that hasn't been explored
within shape change, which was the radio.
For each of these two artifacts, we created two tasks,
a pragmatic, which could be setting the mode
on your mobile phone, and indicating it,
so whether it was flight mode, or it was silent,
or it was normal,
or on the radio, that could be setting the volume
from high to low.
The hedonic task was how can you perhaps use shape
to convey an emotion and a message on a mobile phone,
or indicate the genre of the music playing.
Is it jazz, is it rock, is it hip-hop, whatever is it?
Each participant got either of the two tasks
for the mobile or for the radio,
and they were fairly equal distributed.
I think one of the interesting things that came out of this
was how people actually manage to convey the dynamic nature
of shape-changing interfaces,
so what they had was all a static medium,
so they had sketches, or something then ...
They implored a range of different strategies
to actually convey this ability to change
from one state to another.
I'm just going to touch upon a few of them,
but I want to say most of them used
sketching the end points, so sketching one shape
to change from and then another one.
The second most used was to use written descriptions
to add to our understanding of in what manner did it change
from one shape to another.
I want to highlight two more that are a bit different
and didn't occur as much, but actually gave quite a lot
of information about how the thing moved,
apart from this static input,
and one of them was actually using a timeline,
so showing how much time actually the time spent
of that set of events going around on this mobile phone,
and the other was actually sketching small characters,
like this small bug to say,
"Well, it was going to curl up,"
but they found it hard to sketch that curling up motion,
and then they used to sort of draw a small bug,
and sort of like a bug, how it would curl up.
There's lots of different interesting elements
in how you actually manage to sketch these dynamics.
In our analyses, we chose these three perspectives,
and a few different theoretical framings to understand,
and I'm going to go through each of them.
I'm going to pass over likely with the vocabulary,
spend a bit more time on the metaphors and the affordance.
The vocabulary, we used, well one of the frameworks
that I have helped developed just to check up,
it's been five years almost since we made it,
and a lot of things have occurred.
It's a still useful framework for analyzing
shape-changing interfaces,
and as you can see in these three illustrations,
we saw a lot of different types of shape-changing.
We saw textures, we saw shapes, we saw very organic,
we saw more geometric shape-changes,
a very wide spectrum and nearly all of them,
sort of types of, eight types of shape-changes
that was mentioned in the vocabulary,
was used, except for viscosity.
I'm not going to touch upon much of this,
I'm just going to highlight a few of the contributions
from the area, and I think that it actually showed
a much more varied use of the types of shape-changes used
than we see within the existing work of shape-changes,
and a lot more extreme cases of shape-changing interfaces,
as you can see down here.
Changing from a piano shape to a DJ console,
perhaps beyond what we can build today,
but it shows the minds where we might want to be going.
If you have issues radical atoms materials,
I think it's most interesting just to see
that compared to the sounds we see today
in shape-changing interfaces,
it showed a really, really tight coupling
between the interaction and the shape-change,
so we have a lot of pinching, and rotating and twisting,
a lot of gestures related to the shape change.
Perhaps that's where people want to go,
but maybe are not able to today.
Some of the challenges adults have highlighted is of course
sketching these dynamics, which is hard,
but I think also more poignant is that
it was actually categorizing some of the more complicated
shape-changes the framework wasn't really very good at,
so it's good at saying, well it goes from a round shape
to a pointy shape, a simple example at the top.
But, is it really sufficient to describe it
as a change in form when you go from
a piano shape to this DJ console?
It has a lot more connotation from the user
than simply changing shape, so I think we need to evolve
those frameworks to capture more about the interaction,
as well as the types of changes in shape.
If you go to the metaphor side,
then if we look at the examples of shape-changing interfaces
in the papers that's been written about it,
then it's actually mentioned metaphors quite a lot.
We have people mentioning anthropomorphic metaphors,
animal metaphors,
or that just sort of visual and tactile metaphors
that's highlighted by it,
however when you look more critically at it,
it's more used to describe, rather than analyze,
and there's no examples of actually relating
back to the categories of metaphors that we know from HCI.
We wanted to try in this case to be a bit analytical
and see did they actually use metaphors and in what way?
So, we used Lakoff and Johnson's three types of metaphors,
we also looked at the metaphoric means,
which is how the cues from this artifact you used
in the metaphor transforms to the shape-changing interface,
as well as the metaphoric entailment,
so if what we have of these signifies what they apply
about the signified.
If you go through these three metaphoric categories,
then you start at this orientational metaphors,
and they're the sort of metaphors
where we use spacial orientations,
such as up and down, left and right,
to leverage our everyday understanding of spaciology
to convey this information.
Especially in the radio volume case, we saw a lot
of this type of metaphor,
and that's perhaps quite natural,
since we're normally used to turning a volume up
by rotating a dial, maybe pulling pushing a slider up,
but here rather just simply a slider,
it might be the whole radio that you actually squish down
to have a lower volume, or you push it up.
We also saw it to convey emotions,
so saying sad is down, happy is up,
as well as more unconventional uses of it,
saying that twisting a shape, rotating it, is louder,
where we see that rotation in the shape,
and straight is then more quiet.
Then, we had the structural metaphors,
and they are using real world concepts
to describe something that's abstract,
so you know it from your trash can icon
on your desktop, on your computer.
So, here we saw various literal metaphors,
so if you have put your phone in flight mode,
well the phone gets wings.
We might see if you want to say something that's happy,
then you try to make the phone into a smiley shape.
Interesting, we actually also saw elements where we transfer
like this radio, where we transfer the volume icon
that's 2D into a 3D shape and try to use that as a metaphor,
so it is a metaphor for something else,
and then we now transfer it in the physical shape.
The last one, and I think there's some really interesting
elements in this one,
is trying to use a better basic category of existence
in the physical world, such as substance coffee container,
to explain a concept.
In this case,
if you want to enter flight mode on your phone,
then that means you break a connection with the outside
and you simply break that shape of the phone,
and you have that as a metaphor for what's going on,
or you might see
angry as having a very pointy shape, or sort of,
so that has a material, so anger has a material,
or if it's happy, it might be a softer shape.
I think some of the things that we all,
those of us that are really interested
in shape-changing interfaces, are the dynamics,
and the question is then,
well, how does dynamics actually come into play
when we talk about metaphors?
In the cases, we didn't only see metaphors being used
in their static shape,
so I change from this shape,
and then that shape means something else.
It was also seen in the movement,
so you have in the middle here this mobile phone,
that tries to convey sadness through hanging the head
and doing a sobbing pose,
so there the metaphor for what it means to imply
only becomes evident in the motion.
The last case was a way of setting,
turning this phone and its shape into this
night mode or flight mode, where you disconnect it,
and you do it by rocking it like a baby,
so that's the gesture, so it's true the interaction
that the metaphor becomes apparent.
Here I'm really relying on using the dynamics
to use the metaphors and not only the static shapes.
If you look at this area and what other contributions
that this paper brings in,
well, whether the participants really thought
about metaphors, or whether they unintentional,
then when we analyzed it,
then in 37 out of 42 of the sketch instances,
answers we had, we saw that they used some kind of metaphor,
which is quite a lot.
Perhaps there is a really huge potential for us
to become more specific about how we use metaphors,
and actually use them in this sense,
in the different manners, and if you think about it,
maybe it makes sense, because metaphors is a way of making
abstract concepts physical, which is, well,
we're taking interaction into the physical world.
I think that's also sort of this interesting part about,
well, we saw some familiar metaphors in a physical form,
sort of the up is more as we know
from turning up and down the volume,
we also saw it, but just in a physical form,
so maybe we can become more clear about which metaphors
we can actually really transfer really well
from what we know from our latent phase,
and where we need to diverge and take our own approach,
which means that we need to think about
how we think about conventions, and how we establish them,
and whether we as a community should really take part
in thinking about this, or if we just let it go,
and let it happen as this turns into real world.
Something that I haven't touched upon
is this metaphoric entailment,
so when you make these metaphors really physical,
then as you see in the top case,
well, one of the people chose to use adjusting the volume,
thinking about an accordion, you can use that shape,
turning it on its eyes, because normally interaction,
it's easy to push down, but it's actually hard to pull some
of these interfaces up,
but that would entail I might be thinking it's an accordion,
I can play along with the music,
which I couldn't in this case,
where as in the other example, it turned in the radio,
when you play classical music,
it turned into this piano shape,
but the design allowed us to play along with the music,
and thereby transferring more of the cues,
and not having this misunderstanding about what's going on.
It's a really, really neat thing
about this metaphoric entailment.
Lastly, affordances, and shape-changing interfaces
has really fused a lot on claiming dynamic affordances,
it can solve some of the affordance problems of the world.
We can have one shape, and then could have a dance
of something else,
but they don't really become very specific
about what they mean, it's often mentioned,
but sometimes, it's a bit weird,
and it's not very clear how it actually mention affordances.
Here, we took Kaptelnin and Nardi's approach
of instrumental affordances, which divides them
into handling affordance and effector affordances.
The handling affordance is this notion
of how does it communicate what I can do with it,
and you see on the left,
well, I can maybe see I can bend a corner,
but the effector affordance of entering silent mode
is not very clear.
On the right, they're more well integrated,
so I turn the dial to change the volume up and down,
and the shape changes alongside,
a better support and more well-integrated
handling and effector affordances.
So, there's a lot of potential I think for exploring
what it means to have affordances
when you have something that's dynamic.
Still, really a lot of work to be done on this.
How does motion deal with it?
How can we use shape-changes and affordances
alongside traditionally UI mechanisms?
Examples show there is a lack of this well-integrated
handling and effector affordances,
and you don't necessarily get good affordances,
whatever that means,
by making shape-changing interfaces.
Lastly, I just want to highlight a really interesting
design pattern that we saw a couple of times,
which was this hiding and revealing functionalities.
The radio on the left, volume was low when it's enclosed,
and it's opening up when the volume is higher,
and you see something of the same with the amount
of dots on the side.
Just to remind you of the conclusion,
I'm just going to leave it here and open up for questions.
(audience clapping)
- Any questions?
Okay, I've got a question.
Did you, was there a design that was really surprising,
something that didn't fit your, the views you had,
or what you had expected to receive?
- This one, on the left, that curls around your ear,
and takes up as the radio, and turns it into something else
that becomes bigger so you can see the volume,
and adapts too ...
I was definitely not expecting that,
but I must say I was really excited about that radio,
and I really want to build that one (laughing),
because I think it's really cool,
hiding and revealing this sort of part of the functionality,
and making something clear through the shape.
- [Audience Member] Hi, Hiroshishi, from MIT Media.
I really appreciate the attendance
about affordance and the metaphor.
First, I was mopping over the meaning, it's so complex,
like a black hole or an art.
Second, existing affordances, you could be dependent
on proximity of the physical world,
or they make you sit on mechanical steps,
like a door knob, but the last via-rotta-gotos
was shaped bunny ears, changing bunny ears enabled,
it's beta can't anticipate how it really transform,
so I think this direction, this scene is so important,
so we need a strong wand in PhD.
If anybody wants, come there-metty.
- Anybody willing?
- [Audience Member] Yeah, thank you.
- [Host] Okay, well, let's thank our speaker again.
(audience clapping)