Hi, my name is Pedro Lopes from the University of Chicago, and today I'd like to talk about
a reflection paper that, this very group of people that stand here in this slide, my co-authors
and I did about a paradigm shift in our thinking about interfaces.
We are used to think of HCI as the study of the interaction between humans and computers.
This perspective of "interaction" makes us consider users and devices as two separate
entities, the user, and the device.
That I represent here with these bubbles.
Now this separation has proven very valuable since the early days of interactive computing
and this is what allowed us to move from mainframes to personal computers by improving the interaction
between the user and the device, by, for instance, optimizing the input device's bandwidth and
its ergonomics.
Then, trying to make these interactions happen everywhere and anytime, led to a flipping
of the form factor of the interactive device, with now the device being smaller than its
user.
As we know this led to mobile devices, and to a proliferation in our analysis of interactions
with these devices, fueled by the need of solutions to new modalities, to smaller devices,
things like touch and this pretty much ended up putting an interactive device into everyone's
pocket.
As this need for interactions everywhere grew and grew it moved the devices from the user's
pocket closer to the user's body, in the form of wearable devices, which as we know can
be attached to the user's body or embedded in your clothing.
This of course also brought a new set of challenges, such as how to interact eyes-free when moving
or in really quick bursts of interaction.
Now this way of thinking of the letter "I" in "HCI" as in "interaction" is very powerful.
It draws on the very definition of the word "communication with someone or something".
It is precisely that that we, as researchers, are often optimizing.
We are optimizing the speed of the communication, the bandwidth of the communication, between
the user and the device.
But today I want to emphasize that there is a new interface paradigm at the horizon, just
after wearable devices, and that these devices "overlap" with the human body to a much larger
degree than ever before.
Let me exemplify this using something that is very close to my heart: interactive devices
based on electrical muscle stimulation.
These have the capability of moving the user's muscles involuntarily.
As such, by just activating these electrodes, one can turn your muscles into a display by
controlling how you sketch.
Now, here, what is striking is that the user’s muscles are an integral part of the hardware
of the device, and that’s why we say there’s a large overlap between user and device.
As a result of this overlap we really think that is will be productive to think of the
letter "I" in "HCI" not just of "interaction" but also as "integration".
This paradigm shift is not limited to devices that overlap with the user’s senses or muscles,
but even with implanted devices.
These are devices that are completely inside the user's body, usually devices from the
medical realm, such as pacemakers and RFIDs, but also, as we will see, some interactive
devices.
So we think this perspective of integration can be very useful to explore the fusion of
the boundaries between users and devices, and maybe even uncover challenges that only
arise if you would consider these two entities, user and device, as one integrated entity.
So that’s precisely we what we did, we immersed 29 researchers for five days at Dagstuhl to
discuss the grand challenges of integration between devices and users.
We wanted to find which issues are exacerbated when looking at HCI from the perspective of
integration, and we cluster these issues, what we think
are, grand challenges for the field; and, these are organized into four categories:
(1) human compatible technology, (2) identity and behavior
(3) integration with society and (4) designing for integrated interfaces.
Now I can’t dive into each of these challenges here, in this format, so let me highlight
two key challenges that I think are especially relevant for future interfaces that integrate
directly with the body.
And let me start with "biocompatibility" and I truly mean it in the broader possible sense
of the word.
Here we are departing from rigid wearable devices that just sit loosely on the user’s
skin and diving directly into attaching or sticking devices, such as films, electrodes,
and so forth directly into the user’s epidermis.
What really sets these epidermal devices apart from wearables is that they are made to adapt
to the user’s body shape, by engineering them using flexible or stretchable materials.
Now, sometimes parts of our research devices might accomplish this goal, but they still
hide away pieces of electronics that are ultimately comprised of non stretchable or non conformable
materials.
The reason why we call this a grand challenge, rather than an ordinary challenge, is that
we don’t mean to just conform the sensor or actuators to the skin, as it is the case
here, but to go further and conform the complete device to the body.
We are seeing a range of new ways to approach this still open challenge from devices that
do not have a rigid PCB to stretchable transistors.
Now the next type of devices are place under the user’s epidermis, and are therefore
literally under the skin.
These are fingertip magnets, RFID implants and even some more complex interactive devices.
Lastly, as we move past the skin and into the fat and tissue, we have truly implanted
devices, such as cochlear implants, insulin pumps and pacemakers.
Now if we look at each of these layers of this stack, we see different key challenges
for HCI researchers.
Devices attached to the skin directly do not require any surgery or complex procedure because
they based on stickers, like sticky electrodes of muscle stimulation.
On the contrary, this means these stickers obstruct the skin, interfere heavily with
tactile acuity and diminish your dexterity, and of course are not permanent.
So need to be constantly reapplied.
This is a key limitation because while these are the safest devices in this list, they
actually the only ones that only have applications in research right now.
Now subdermal devices are of a different class, they require small surgical procedures to
go under your epidermis.
These devices allow for permanent application of simple actuators such as magnets in your
fingertips or RFIDs in your hand, but as they grow in complexity, such as the this one here
with LEDs and batteries by Tim Cannon, well, there's a whole new range of open challenges:
how do you upgrade?
How do you communicate without cables and open wounds that might get infected?
And of course, due to their deeper level of implantation, these truly implanted devices,
such as pacemakers, inherit the challenges from all the layers above with an extra level
of critical care.
That’s biocompatibility of the materials to avoid rejection and toxicity, and, of course,
the need for serious surgery.
They of course have their benefits, such as lifelong permanence, access to organs and
not just to sensory apparatus, that’s how a pacemaker regulates heart function.
And they usually don’t need any explicit user involvement, most of them just do their
think automatically.
So while these are certainly implanted devices they are really interactive implanted device,
as they do not expose an interface to the user.
So it is an open challenge how you might allow your users to control and interact with such
devices.
Now I’ve talked about challenges down at the intersection of hardware and the user’s
biology in their skin, but how do these new integrated devices might affect user’s identify
and behavior?
The big open question here is, if you’ve integrated with a device, to a point that,
as I just told you, the device maybe sits well inside your body and is accessed trough
a very directly means of interaction, such as muscle movement, what does this do to your
sense of self?
In other works if there is an interface paradigm shift (towards integrated devices), is there
also a shift in perception of self?
Technologies for changing one’s perception of self are out there, in fact, virtual reality
(VR) has been a rather popular means to achieve many of these illusions, such as to shift
one’s perspective into that of a child with a smaller body.
And certainly this is possible outside of the black box of VR.
Here’s a user that sees from the viewpoint of a camera attached to their waist and, as
my postdoc Jun Nishida found out, users not only see from the height of a child, but their
behavior is somewhat transformed into that of an actual child.
Now, these devices do this by tricking our optical sense, but, as this integration moves
from the user’s senses… to the user’ actuators, like their muscles, self-perception
changes even more because we can now manipulate also self-action.
For example, in one of my early projects using muscle stimulation, I created a variation
of a popular children’s game called "red hands" or "slapsies", in which two people
are playing and one tries to slap the hands of the other and the other tries to avoid,
here played by one player.
Here the trick is that your voluntary hand is controlled by means of muscle stimulation
and you are trying to avoid with the hand that is under your voluntary control.
When my participants tried this out, what happened is that they reported a sense of
blurred ownership.
Sometimes referring to their computer controlled hand as par of their own body and sometimes
as an interactive device that they just played a game with, almost outside of themselves.
This is even more evident as we start to give user’s abilities beyond their biological
capacity, for example, here we used muscle stimulation to accelerate someone’s reaction
time so much they are able to take a picture of this fast moving baseball.
This counters your intuition, because my intuition would have been to say that if I would feel
the complete sense of agency (the sense of "I did this") when I'm doing a voluntary action.
So if I am taking the picture by myself I would feel maximum agency.
But if anything, or any device, helps me to do that faster than humanly possible, for
instance by assistance by means of electrical muscle stimulation, I would feel absolutely
no agency.
But our findings were actually quite different.
We saw that users actually felt completely in control of actions that were still performed
faster and could only be performed with that speed by means of muscle stimulation.
In fact, there are many other devices out there that can achieve this kind of assisted
control, such as extra limbs that hold tools for you, or muscle stimulation that makes
you operate a tool that you've never used before, or even a tail that automatically
balances your body.
Now, these exhibit very varying levels of agency, from user driven, to mixed initiative,
to things that just automatically do their thing.
As such, different balances between user-device agency might yield different perceptions in
their users, such as: “I did this” or “this made me do it” or “this helped
me”
We think this is really a grand open challenge for devices that have the ability to control
the user’s body.
How are you going to design your interfaces to have this mixed initiative and mixed agency?
What’s also more striking about the perception of self, is that is influenced by how others
see us.
And as such, the open challenge here is also to examine the perception of these integrated
devices on others, for example, how will others perceive you when you have this robotic tail
like this one, or this interactive tail by our coauthor Dag.
And, in fact, some researchers, such as Christian Holz and his colleagues, have explicitly looked
at how users of mockup implanted devices felt as they walked around their surroundings using
these devices around bystanders.
And so we think this is a realy key challenge to deepen the understanding of the future
of these devices, to understand how they impact the user and the users around them.
Now, let me summarize.
I presented the grand challenges we derived by looking at HCI as "integration" rather
than "interaction".
These were clustered in four categories that you see right here.
And I also want to emphasize that, while I did not get into all of them in this talk,
this is really the kind of paper that you should read, dissect and disagree with.
As a paper co-authored by 26 people, I also don't agree with all the perspectives in it
and we really meant this as a conversation starter.
Now if you find yourself saying "HCI" but meaning something else than our existing HCI
acronym and you are getting confused with "integration" and "interaction", don't worry.
We are not necessarily trying to overload our very useful term.
In fact, most of us refer to human computer integration as "HInt" (or hint).
Now, of course, some of us went all in with it, such as myself, when I decided to call
my HCI lab, the "human computer integration lab".
So we also trying to provoke more researchers to consider this perspective when they are
just doing their research at every moment, and sometimes thinking and considering users
and devices as one integrated entity might be very very useful for your own research.
Lastly, I really want to thank all my amazing co authors and, especially, I want to thank
Flloyd [Mueller], Jonathan [Grudin] and Pattie [Maes] for bringing us all together and writing
up this paper.
I'd love to hear more of your questions, please send me an email.