- So we are happy to be here to share with you
our experience in using trigger-action programming
for personalizing the behavior of humanoid robot,
and this is joint work with Nicola Leonardi,
who is a master's student in the digital humanities degree
at the University of Pisa,
and Marco Manca and Carmen Santoro,
who are here somewhere in the room,
so let's say that at some point we had some monies,
and we decided to buy a humanoid robot.
It was expensive.
Was 16,000 euros, at least.
For us it was an expensive study,
but, I mean, it was really interesting
to see how much interest this kind of technology
can raise from any type of users,
so ranging from children to older adults,
so we said, "Oh, this is really nice and interesting,"
and we all see that robots are increasingly used
in many types of contexts.
Robot is quite a general term,
so we probably need to distinguish a little bit.
For example, there are industrial robots,
which usually are used in specific contexts
for doing specific tasks
such as picking an object and move somewhere,
and such humanoid robots, which have this feature
to be able to interact
by using human-like modalities,
and they can really be deployed
in any type of context:
for housework, supporting elderly,
disabled people, hospital, and so on.
One interesting issue that comes at this point is that,
I mean, since they can really be used
in many possible contexts,
it's very hard for developers to predict
all such possible use, to predict all the possible needs,
so we think that a key challenge with such robots
is to allow the customization
in order to address how the specific the situated,
often evolving user needs.
So we think that if we even allow the end users
to directly decide how they should be able,
for their specific needs,
this solution would be very, very high impact.
So let's say that, generally speaking,
we are interested in end-user development,
which concerns some metatools
that allow people without programming experience
to be able to directly create or modify their application,
and on the other hand, also,
robots can be programmed,
but they usually are languages
that are really targeting engineers.
There are some effort,
there have been some efforts
in order to simplify the development of a robot application,
so for example,
Code Graph is a icon data flow visual language
where we have some icons
associated with some set of functionalities
that people can compose.
I remember last CHI, Weintrop presented a paper
which was about using a block-based language
for supporting industrial robots,
but as I said, that this kind of robots,
usually they have a specific task to perform,
so they like a limited set of options
that should be supported,
while modern humanoid robots,
they really have a wide variety of possible interactions,
possible action that they can perform.
So let's say
we thought about trigger-action programming: Why?
Because we say, okay, these complex structure
where you just to indicate the specific situation
that are available for you,
and the desired feedback is something
that even people without an algorithmic ability
can think of,
or can really use for specifying their needs,
and so they don't need to learn
a complex programming structure.
It actually mean some trigger-action programming
is already used for the automation of web services,
so probably the most common environment is IFTTT,
where you can connect existing services,
such as Facebook, Dropbox, and so on,
so you can say if someone tags me on Facebook
then please
turn my light on the office a specific color,
for example, and so we wanted to investigate
can this paradigm be applied
to support personalization of humanoid robots,
and another in this challenges: Can we exploit this paradigm
to connect the behavior of the robot
with what happens in the surrounding environment?
So let's say that in order to address such challenges
we have a language
where we can distinguish between events and condition,
because events in when something really changes,
and condition specific states
that can be through for some time,
and then, of course, it's possible
to define various types of action,
so we can say when I look like sad,
then the robot should tell me some joke,
or when someone arrives at the door,
the robot should go there and say, "Oh, who are you?
"What do you want?" things like that.
It's easy to specify.
So let's say that we have such humanoid robots,
we have a lot of sensors, marked objects, and so on,
and we have already some software
that developers have created and made available
for controlling the behavior of the robot,
and then arrives the end user.
The end user has specific needs: I want to change something,
and so we provide this tailoring environment
where it can indicate the specific personalization rules,
and such rules are sent to a rule manager,
which subscribes to a middleware,
we call it a context manager,
that is able to receive the events from the robot
in the surrounding environment.
So it has what we call a context delegator
associated with the robot,
which is a kind of intermediate level
between the low-level events
and the events that can be specified
by a person without programming experience.
If so, when those event triggers occur,
then it can activate the desired behaviors,
the desired actions, and so on.
So this is the robot that we have used
for our research,
so it's by SoftBank Robotics, so 1.2 meters tall.
It has really rich of a sensor: tactile sensors,
laser sensors, has four microphone, two RGB cameras.
It can really perform
the typical human-like modalities of interaction:
speech, gesture,
can follow with the gaze someone who is nearby,
detect proxemics, and so on,
and so this is a simple environment
that we propose to people without programming experience
in order to specify the trigger.
I mean, so they can,
they are classified according to different categories,
and the selected elements are elected,
and when they select the events associated with the robot,
they are classified depending on the type of event,
whether they are related to touching something,
or they are related to recognizing
some aspect that happens nearby,
and we also give the possibility, of course,
to compose such trigger and such action,
so for example, if you look at IFTTT,
it only support single-trigger rules,
while for robots it becomes important
to be able to specify some more structured behavior,
so it's possible to indicate logical operators
for composing triggers,
and to indicate whether actions should be executed
in parallel or in sequence,
and another interesting point is that we can have, also,
a kind of a connection,
a kind of continuum between what happens in the environment
and what happens with the robot,
so it's possible so to specify triggers
that are related to events
of objects that are nearby,
for example lights, a heating system,
oven, fridge, and so on,
so we can connect these two world,
so we can say
when a person enter a room and the light is off,
then please, the robot should greet the person,
and then the light should change,
should be on and to a specific color.
So just to give an idea about how it works,
oh, pause,
and so this is the environment,
so it's really target people
who don't any programming language
so they can start
follow these categorization of possible trigger,
then, of course, in this case,
we are interested in the robot, the purpose,
so they say, well, if there is a, recognize a face,
and it's someone that the robot does not know,
and this person says, "Oh, what is this lab about?"
then the possible action, okay, play an excited animation,
synthesize a welcome message,
show on the screen some information
concerning the laboratory,
or say something about such a laboratory,
so I mean, it really takes a very short time
to create this specific rule so people then,
they can save, provide a name,
and they can share the rule with other people,
and then, of course, they can execute it.
- [Woman] Hi.
- [Robot] Hi.
- [Woman] What is lab about?
- [Robot] Welcome, visitor.
This is the Human-Computer Interaction Lab.
Our research activity is in methods and tools
to support user interface designers,
software developers, and end users
in obtaining systems that can be accessed
from different contexts of use
in such a way to improve usability, accessibility,
and user experience.
- [Woman] Fine, thanks.
Bye-bye.
- So you see it's really takes a short time to create it,
and it can be immediately executed,
so we can have another example.
In this case the person is known,
and ask, "Oh, what's my agenda today?"
- [Alex] Hi, Pepper.
- [Pepper] Hi.
- [Alex] What's my agenda today?
- [Pepper] Hi, Alex.
Here is your agenda.
At 12 o'clock you have to attend to a seminar
about website usability in room A27.
The speaker is Jacob Nielson.
At 3:00 p.m. there is the Smart Kids project
general meeting in room C9.
- So, I mean, as you can see, it's easy to create.
It's easy to execute,
and to see what the resulting behavior should be,
so we say, okay, let's try first a user test.
Of course, we involve people
who are not professional developers,
people who don't have any computer science degree,
and then we found in the administration offices
people who were really useful
for this type of task and test,
and we got 17 people.
They are the various educational levels,
and so the test was done in the laboratory,
and it was organized in various parts,
so first we asked them to indicate,
what rules would you like, in general,
to specify to a similar robot?
Then we asked them to specify rules
with increasing complexity,
so which means one trigger one action,
two triggers one action, two triggers two actions,
and then we gave them
some natural language description of some rules,
and asked them, "Okay, please use the tool
"to specify such rules, and execute them,
"and see whether the resulting behavior
"is really what you expected."
So let's say that they were able to do it.
I mean, they did not perform
any particular significant error,
or sometimes of time, I mean,
it did not take a particularly long time
to create such rules.
I mean, there was some,
there was a bit longer to the first time
because it was also the first time
that they used this rule-aided tool,
so they also took that opportunity
in order to familiarize a little bit with this environment,
and they filled in a SUS questionnaire, which was positive,
and then, of course, we asked them about,
"Did the robot perform as you expected?"
In general, yes.
There was some, let's say, minor issue
with speech recognition
because sometimes we know
that it always not work as it is expected,
and also, in terms of performance,
in general it was okay, but in some cases
they were expecting some more prompt reaction
from the robot,
while of course the robot is a complex system.
In some case it takes some time
before performing the desired behavior,
but in general, I mean,
the experience seemed to be positive.
So let's say that if we want to just comment a little bit
in more general terms from this experience,
what we can say is that one key issue
is to offer the most appropriate abstraction level
because programming a robot can be done
at many, many levels.
It can range from low-level hardware primitives
to a high-level search of the area,
and of course on one end
we wanted to provide the possibility
to specify rules using concepts that people understand.
On the other hand, we want, also,
to allow them to construct,
to have more control on what can happen,
so this implies some more technical description.
Then, I mean,
let's say that this type of approach for humanoid robot
is different with respect to what happens with IoT services,
where usually we have smart objects,
which have quite simple behavior,
a few properties, and so on.
The robot is really a system that can do many things,
so it becomes important
to provide a little bit more structure
in specifying the possible action.
We think that this opportunity to create a continuum
between robot behavior and what happens nearby
is really opening up a lot of new services
that can be exploited
in many application domains.
Of course, I mean, that this a issue that sometimes,
so what people specify
is not exactly what they would like to get,
I mean, would like to obtain,
and so maybe some more support
for simulating the rule execution,
or to understand when some rules are in conflict,
would be useful.
There are case in which the robot does not exactly behave
as it's expected
because, for example, there is low connectivity and so on,
so in order to avoid frustration from the user side,
it's possible to make them aware
that there is some temporary issue,
and so the robot performance
is not as optimal as it would be expected,
and of course, I mean, it's important
not only to facilitate this creation
of personalization rules
to allow people to share rules,
to use previous rules, to create new ones.
This is something that can be particularly useful
in this context.
So let's say that, on one hand,
I mean, our study demonstrated that this is protophysical,
I mean, it's something that can work, really,
with people without programming experience,
but we are aware that this is just a first step
along an interesting direction
because how our test was in a laboratory,
we gave them specific tasks to perform,
so what we do like in the near future
is to carry out some more,
let's say, longitudinal validation,
where in the wild people can use this environment
so that we can investigate to what extent it can be useful,
how such personalization needs can be expressed,
and whether or not they evolve over time,
so that would be something interesting
that we plan to investigate soon.
Thank you.
(audience applauding)
- [Emilia] Well, of course,
Emilia Barakova, Eindhoven University of Technology.
You know me.
I want to ask
what was the cyclomatic complexity
of the programs they developed.
- The what?
What type of--
- [Emilia] Cyclomatic complexity
means how many loops they implemented.
- Okay, well, (laughs), well, we gave them
these three exercises.
So let's say that, I mean,
so they did not specify many, many rules,
so in that case I don't remember
any particular case of a loop.
Of course, I mean,
when people create the rules in an extended manner,
there are some issues that can occur,
and one is the possibility to create loops,
but, I mean,
I don't remember any particular loop when we did this.
Also, probably
because they did not specify many, many rules,
I mean, so it was a limited set
that targeted the different types of behaviors,
and so there was not this particular issue.
- [Emilia] Maybe, well, our version one
of our end-user developmental system
was also something like,
well, if something happens, something else will follow,
but finally we converge it with dynamical system approach
when there is,
the whole behavioral space is a dynamical system
is triggered of all possible sensory information,
so you can get more realistic behavior.
For example, if something happen,
for example, person interacts with the robot,
but pushes down and the robot falls,
or robot is not able to perform some action,
so the robot have to go back to the state
when he can perform some interaction,
so maybe--
- No, I see the point.
Let's say that in this case we don't target the possibility
to specify all the possible here,
so we really want to allow them to say,
I would like this something more, because such a robot,
they arrive with some basic behavior available and so on,
but in some cases
they don't capture the specific needs that people have,
I mean, so what we would like to give the possibility
is to say, oh, let me, I mean, I have this specific need.
Let me add the, and so maybe,
but I mean, I can see that it would be interesting
to do some study to compare different matter
for different approaches to see what,
and probably when they are most effective,
because I don't see that one solution is always the best,
of course, I mean, so we cannot--
- [Emilia] It's not about best solution,
but possibility
to allow a little bit more complex interaction.
- Yeah, that's, depends also on, then, the user,
and their knowledge of technical aspect, of course, yeah.
- [Emilia] Well, we want to work with agencies
who don't have technical knowledge,
but how to allow these people
to create more complex behavior.
This is the question. - That's
an interesting challenge, (laughs) yeah.
- [Moderator] We have time for one last question
if it's a short one.
Not seeing any hands, we might wrap up.
The next thing coming up on the schedule is the morning.