Thank you very much for that introduction.
I am super excited to actually talk
about something related to ghosts here, especially
in New Orleans.
So my name is [? Xun ?] [? Qian. ?] I'm a PhD student
from the [INAUDIBLE] University.
Today I am going to present GhostAR--
A Time-space Editor for Embodied Authoring
of Human-Robot Collaborative Task with Augmented Reality.
This work is in collaboration with [INAUDIBLE],, [INAUDIBLE],,
Manav, [? Ke, ?] and Professor Karthik Ramani.
So what is GhostAR?
In a nutshell, GhostAR is an authoring system
for human-robot collaboration task.
The key feature of GhostAR is that it realizes
human's physical demonstration with in situ AR avatars, which
we call it a ghost.
And then using these ghosts as time-space reference,
user can then program for collaborative robot behavior.
So before we move into the details of GhostAR,
I would like to talk a little bit
about the background of robot programming, or task planning,
OK.
Through all these years, we have been figuring out
a lot of different ways that allow users to intuitively
program tasks for robots, through graphical user
interface, tangible user interface,
and physical demonstration.
However, for most of these methods for programming robot,
after the robot has been programmed,
they are most likely operating by themselves in isolation.
That is not the case for human-robot collaboration.
In human-robot collaboration, or HRC,
humans and robots are two equally important parts
during the operation.
And therefore, if you want to plan
a task for HRC application, you not only
need to plan for the robot.
You also need to plan for the human at the same time.
And more importantly, since humans are actually
working with the robot collaboratively
and they are working towards a common goal,
they're not working in isolation, right?
And that means during the operation,
they must be constantly adjusting and adapting
to each other's uncertainty in order
to work in a collaborative manner.
For example, if I noticed my partner is working at a higher
speed than I am, I might want to consider
picking up my own speed in order to keep up.
OK, so you see all of these little things
needs to actually be taken into consideration when
you are planning for HRC tasks.
And that is why traditional methods for robot programming
cannot be successfully adopted for HRC applications.
So prior work on human-robot collaboration
predominantly used this method to program a task,
involving using two humans demonstrating the task,
and with one of them actually acting as a robot.
This method automatically established the collaboration,
the correspondence between the human and the robot
through a physical demonstration.
And it's actually establishing adopting
a philosophy of programming by demonstration,
enabling users to program robot behaviors
with natural human emotions.
However, since this method does not have a digital interface,
it supports very little sophisticated editing
after the robot has been planned.
And therefore, resulting in a relatively simple task.
So inspired by the prior work as well as our own knowledge,
we decided to adopt an augmented reality interface on top
of programming by demonstration method
for to create our own system.
So the workflow of GhostAR can be
illustrated in these pictures.
To program a human-robot collaborative task,
the user first demonstrates the human's action
by physically demonstrating the human part
of the collaboration.
And then his motion is captured and recorded and displayed
as a series of human ghosts.
Using these human ghosts as time-space reference,
the user can then program for the collaborative robot
behavior.
At this point, the task planning for the HRC
has already been finished, and then the user
can proceed to actually carry out
the collaboration with the physical robot
by simply repeating what he has done before.
And our system is capturing this real-time motion,
compared to his authoring content,
and then find out the corresponding robot behavior.
Then issues the command to drive the physical robot,
then completing the human-robot collaboration.
So essentially, this is how GhostAR works.
And then our system actually supports two types
of human-robot collaboration, including
synchronized and trigger.
For synchronized collaboration, the human and the robot action
takes place at the same pace, while for that trigger action,
the robot action is actually triggered
and takes place after the human action.
It is important to mention that our task adopts
the type of human-robot collaboration
as human lead, robot assist, meaning that the action is
always initiated by the human user,
while the robot always tries to adapt and assist
the human user.
There are a few different types of human-robot collaboration,
which are not the focus of this work.
So now, let me actually describe and explain
the actual system design of GhostAR using an example.
This is the example.
The example involves one human simultaneously collaborating
with two robots, and the human, in the example,
delivers the object onto the table.
And the arm bot then picks it up and puts it into the basket.
Meanwhile, the camera bot follows
the human and videotapes the entire process.
OK, so in order to complete this human-robot collaboration using
GhostAR system, the user first physically
placed out the human part of the human-robot collaboration
by acting it out.
And then our system captures his motion
and represent it with an animating AR ghost.
And using these AR ghosts as a time-space reference,
the user can then define the part
of the human motion he want the robot to be collaborating with,
using the grouping function.
And then assign robot collaboration,
choosing from synchronized or trigger.
To program a synchronized robot collaboration,
the user can simply just manipulate a virtual robot
while having the human ghost as the time-space reference,
and therefore, simultaneously establishing
the correspondence between the human and the robot.
So in this case, if the human moves faster,
the robot will move faster in order to keep up.
In terms of the trigger human-robot collaboration,
the robot is literally triggered after the human action.
In our case, this robot arm will start picking up the object
after the human put the object onto the table.
OK, so when the user finished the authoring,
he had the option to actually go into the preview
mode and preview the entire task using AR simulation to make
sure that everything is working as intended before he moved
into the action mode and actually carry out
the collaboration with the physical robot.
Well, he simply needs to repeat what he has done before,
and our system makes it easier for user
by providing a guiding AR ghost, and then completing
the human-robot collaboration.
So essentially, what is happening during the action
mode when the user is carrying out the collaboration
with a physical robot is that there is an underlying
collaboration model in our system
that takes in the human's real-time motion as input
and compare with what he has authored based on the authoring
human ghost using the motion-mapping algorithm based
on dynamic time warping, and then based on this result,
we'll be able to find out the collaborative robot
behavior, then issue that to drive the physical robot
and then completing the human-robot collaboration.
So it is worth mentioning that our system is based on--
our motion-mapping algorithm is based on dynamic time warping.
So essentially what it does is that it find out the similarity
between two time series.
And in our case, the first time series
is when the user is authoring the human part of the motion,
and then the second time series is
when he's actually acting out during the action mode.
So our system, the entire AR authoring system,
is developed using Unity engine.
And you also notice that during the--
if you want to author a robot behavior,
you actually manipulate through a virtual robot.
So in order to provide realistic robot simulation
that best mimics the real dynamics
and physics of the physical robot,
we adopt Ross as the back end robot simulation engine.
Here we demonstrate a few more examples
using GhostAR system, including human-robot joint assembly
task, light providing flying drone in a dark room,
and also a handshake example with a [INAUDIBLE]..
To evaluate our system, we have conducted a three-session user
study and invited 12 users.
And the first session of the user study
is to evaluate the motion-mapping algorithm
with our system where we ask the users to perform
a series of human motions.
And according to this result, we can
see that our system is capable of achieving
fairly high synchronized action progress estimation results.
And in terms of trigger task detection,
according to our result, the difference
between the system detection time and the ground
choose can be achieved within 300 milliseconds, which
is pretty good.
So the second session of the user study
is to evaluate the robot authoring user activity, where
we ask the users to manipulate a flying drone
as well as a ground robot to try to follow
a pre-programmed human ghost as close as possible while trying
to then calculate the distance arrow as a result.
OK, so as you can see from this figure,
the drone is a little bit harder to manipulate than the ground
robot, obviously due to the added degree of freedom.
But more interestingly, we can see
that an experienced user which has been practicing
this task for five times is capable of achieving
much, much higher accuracy than [INAUDIBLE] user,
indicating our system is pretty easy to use and master.
The third session of the user study
is to evaluate the overall usability of our system, where
we ask the users to perform a comprehensive joint assembly
task with a [INAUDIBLE] bot.
And according to the user feedback,
we find out that users generally appreciate and enjoy
using our system, and they gave relatively high ratings
for the user experience ratings, especially
in terms of the ghost.
They particularly like the ghost,
using the ghost idea to simulate and to visualize
the robot authoring before anything physical takes place.
OK, so the user does like the ghost.
However, we find something interesting and [INAUDIBLE]..
Some users mentioned that the ghost can actually
be obtrusive or destructive at some time.
So whenever there is too many ghosts in front of their face,
they feel like they can't see whatever they want to,
they actually wanted to see.
So this actually brings out a very, very interesting
question, probably a foreign research question
for future endeavor is that how can we
balance between clear authoring real and instructional AR
content while providing both to the user?
Well, I believe that the answer to this question
might open potential new directions
for future endeavors.
So if there's one thing I would like you to take away
from my presentation, it's that GhostAR is essentially
an authoring system for human-robot collaborative
task through embodied demonstration with AR ghost
as contextual time-space reference.
Thank you very much for your attention.
I will now be ready for any questions.
[APPLAUSE]
All right.
Questions for the speaker?
Maybe while we get the next person to go set up as well.
All right, so I've got one to start us off.
So this seems to work really well for sequential tasks,
so you had the example where somebody
has to bring an object over, place it,
and the robot picks it up and does something with it.
Is there anything that comes up as a challenge,
or maybe how do you handle the case where
I need to be working with the robot back and forth,
so maybe it needs to carry something with me to a goal,
and then I need to place it, and then it takes it from there?
That back and forth seems to add complexity
because the simulation and the prototyping stage
is going to not actually have that physical systems, right?
Well, in that case, actually we have
concluded that most of the complex behavior back and forth
between human and robot can actually
be dissected into the unit synchronized or triggered task.
And most of the local robot behavior
is actually handled by the robot itself,
such as like detecting the obstacles
and ultimately do some simple decision makings.
But yeah, in terms of the--
to answer your question, we think most of the questions
can be dissected into trigger task and synchronized task.
OK.
Great.
Other questions?
All right.
Let's thank--