Hello, my name is Lawrence and today I'll be presenting on the work I did with Dan,
Veronica, and Sean titled "User-defined Swarm Robot Control".
Robots are increasingly being deployed in large quantities across
personal, commercial, and industrial sectors.
Its application ranges from transportation to parcel sorting to firefighting.
Fleets of robots are especially effective in tasks that
either require a distributed network or collective effort to combat difficult tasks.
So we're moving towards a society where people will often
encounter and will need to interact, collaborate, or supervise swarms of robots.
In such society, one important question that needs to be addressed is
how do we interact with the robots?
One challenge that arise for a swarm of robots is that
it's unclear how to best command so many robots in situ. While one could command
each robot individually, that is a very inefficient method and so how can we
enable users to effectively command a swarm of robots?
Prior work in control of multi robot systems have mostly explored leveraging gesture.
Researchers have used EMG and IMU sensors to detect muscle signals and acceleration of user's
forearm to derive a set of hand gestures that could be used to command multi-robot systems.
Computer vision also has been used to detect whole body posture to
control multi robot systems. However, one problem with both of these is that the
resulting command sets are somewhat arbitrary and are not very intuitive.
This is because they're primarily based on the limitations of the sensor rather
than really taking into account how users would naturally interact.
Thus, instead of choosing the sensor first and then figuring out a set of
input that the sensor can distinguish, we decide to flip this
process around and first focus on finding this user-defined
set of inputs for multi robot systems.
To do this, we conducted an elicitation study.
Elicitations studies have been used in both HCI and HRI fields.
For applications like surface computing and human drone interaction, researchers were
able to compile an interaction set that is intuitive to users. Research have
shown that gestures authored by more people regardless of whether there are
researchers or participants were preferred to those authored by fewer people.
In our case, we use this elicitation study to answer the following research questions.
First, how do users prefer to command a swarm of robots?
Is it actually different from expert to fine or sensor based command set?
and what types of modality to people use for different commands?
Another question that emerges for multi robot systems is
how does the number of robots affect this?
Literature in HRI have shown that people's perception changes based on how many robots they interact with.
What is its effect on how people command?
Lastly, what role does proximity have in how people command robots?
Again, literature from HRI have shown that proximity to robots affect
how human perceive them.
To answer these questions, we ran an elicitation study with the following independent variables.
For the number of robots, we had one, five, or 20 robots.
And for proximity zones, we either use intimate or personal.
We also compiled a set of commands for multi robot systems from prior literature.
Broadly, there are four categories.
First is robot selection where users are asked to select different number of robots.
Second is inter-robot interaction where participants have to command the relationship between these robots.
So for example, for split and merge tasks,
participants had to either split up a group into two groups or merge them into one.
Another category is user-robot interaction where the user either tries to get attention of the robot
or decides on the spatial relationship between the robots and the user.
Lastly, there is robot-environment interaction. Here, participants either decide how the
robots navigate in the environment or the speed at which they navigate.
They also command robots to manipulate objects in the environment.
For example, you may ask them to grab an object.
To run the study, we had the following study set up.
A TV monitor displayed a pictorial prompt and robots were placed
on a table with the user standing next to them.
For the robots, we used the Zooids platform from Le Goc at al.
as they are small enough to allow manipulation of
many robots with one hand and also fast enough to support real-time interaction.
The study procedure was as follows.
First, we show the participants a pictorial flashcard containing
the initial and final state of the robots. We then ask them to
perform whatever action they think is appropriate to command the robots to go
from the initial to the final state. We told them there were no constraints and
we use a Wizard of Oz study design where the study operator initiated the
robot behaviors in response to the participant action.
After each trial, we ask the participants to recall and explain their actions to get a better
understanding of what they did and why they did it.
We look at the results for the first question.
We compiled a user-defined interaction set by combining
responses from the users. Some of them are shown here.
To command a robot to move to a specific point, participants pointed at
the desired goal location. When there are many robots, participants physically move
one of them towards the goal target. To get attention of the robots, participants
either wave at them or snap their finger.
Combining all of these, we generated a comprehensive set
and the interactions with the highest agreements are shown here.
For instance, to merge or split a group of robots,
participants use their two hands to either virtually
push together the robots, or separate the robots.
As seen from the previous video,
to command a single robot to move to a specific point, users were most likely to
point at the goal location.
When controlling the speed of the robots,
participants use a calming gesture with both of their hands.
Finally for a grab an object task,
participants physically manipulated two of the robots and move
them toward the object and expected the rest to follow.
For the full interaction set, please refer to our paper.
Here is the breakdown of modality across all the
user-generated interaction. We found that gesture is the most common modality and
that supports prior work that have mostly leveraged gesture for
a multi-robot system control.
However, the interaction set is still multimodal
in that a significant portion still uses touch or verbal communication.
One interesting trend is that verbal commands were almost always
accompanied by gesture
Even for participants whose primary modality was verbal,
they still subconsciously use gesture on their side to complement their verbal command.
The second question we had was on the number of robots and its effect.
We did observe significant effects in that even for similar task,
participants use different number of hands based on how
many robots they interact with.
This effect was significant for both gesture as well as touch interactions.
For gesture, we saw an increase in two-handed gesture
when there are more robots
and decrease in use of one or two fingers.
Similar trends were observed for touch interaction as well.
Just to clarify, one or two finger gesture or touch interaction refer to instances where
users either point with their finger or grab a robot with two fingers.
These types of inaction occurred abundantly
and thus we labeled them in a separate category
than a one-handed interaction.
Regard to the effects of number of robots,
participants like p5 commented after the study that "I often
wanted to use both hands when interacting with a group of robots even
though I knew already a single hand could work the same".
We also saw an effect of number of robots on how users control manual robots to a touch.
With more robots, instead of trying to directly touch all of the robots like done here,
participants were more likely to use a different strategy where they
only control a subset of the robots like here or only just one of the robots and
expected the rest to follow.
For touch interactions, we observe this shift
in control paradigm as the number of robots is increased.
Instead of trying to control all of the robots,
participants were more likely to only control a subset of them and expected the rest to follow.
Participants like p2 noted that
"When there are too many, I wasn't sure how to grab them all so it
led me to think other ways to direct them".
Lastly, we look at the effects of proximity
Indeed, there was a significant effect of proximity.
In particular, we saw that proximity affected
how many hands or fingers people use for gesture.
When robots were far away, people tend to use more one-handed gesture and while we
didn't investigate the exact reason behind this, we suspect it had to do it
the fact that people tended to lean forward when the robots are further away
and it would have been unstable to use both hands while leaning forward.
Further investigation will be needed to confirm this.
From the elicitation study, we compiled the following design insights.
First, the interaction vocabulary is dynamic
in that it changes based on the state of the robots.
Future solution must be able to recognize use of both arms or hands.
It should also understand that users tend to touch only up to a limited number of robots.
Second, the user defined command set is multimodal but with an emphasis on gesture.
An ideal interface should first focus on gesture recognition but
should still detect multimodal inputs as it would provide flexibility to users
in different scenarios.
Lastly, future interfaces should provide back channeling responses
We observed numerous instances where participants were unsure
if their commands were being properly understood by the users and so
you can see them repeating their commands until they're satisfied.
It's important to provide this feedback in real time to the user either by
overlaying visual output through an AR-like systems or through a haptic feedback.
There are some limitations to our results.
As we use a relatively small tabletop robots within a limited workspace,
the results may not generalize to other significant different systems.
Additional studies will be needed to see if the trend is consistent across
different scale and size.
We also observed instances where users fall back
on their early or first commands even when the parameter or tasks is significantly altered.
A larger participant pool could help mitigate the effect of this "legacy effect".
During our study, participants were solely focused on the robots and
the task. It'll be interesting to see that the command set is different
when the user is paying attention to some other task.
Finally, our work has shown that culture background plays an important role.
So before deployment of any interface, it's important to investigate the effect
of different cultures.
In conclusion, we're beginning to see teams of robots
entering our daily lives. As these robots need to interact with not only developers
but also everyone else, it's important that these robots have this vocabulary
that is accessible to everyone.
Thank you for listening to my talk