[Music]
you
[Music]
a lot of inspiration for my work comes
from nature and I really wanted to show
you these examples where this Eagle is
able to see through water spot the fish
and plan its path such that it can grasp
it while they're both in motion or this
example here where you have a flock of
birds a flock of starlings navigating in
space in seemingly perfect coordination
creating these beautiful formations
without actually crashing into each
other so the question is what kind of
capabilities is it that we need in terms
of perception and situational awareness
to equip our robots such that they can
navigate in the world around us
autonomously and also interact with the
environment so I've taken this problem
and I've broken it down into three
distinct challenges that we are aiming
to address firstly we need to provide
our robots with a sense of where they
are in space and what amount of their
surroundings looks like then building on
top of that a level of intelligence we
want to have richer scene
representations such that our robots are
able to interact with the scene to reach
out and grasp an object for example and
finally we want to exploit the potential
presence of multiple intelligent agents
in the scene and have them collaborating
in a team which rings me to the vision
of what we would like to do in the lab
we would like to have a team of robots
let's say drones that they're equipped
with sensors let's say cameras and we
want to develop their visual perception
and intelligence that will enable them
to navigate autonomously in space as
individuals first of all before
collaborating in a team to perform a
common task together for example to
build a map of their surroundings but in
order to achieve this vision we actually
have to address these challenges which
also correspond to the main research
directions in my lab so I'm gonna go
through them and I'm gonna show you some
of our recent results and current
challenges that were faced with so
firstly we need to talk about how we can
track the motion of a robot while it is
moving
so you might say you know I know how to
do that I'm gonna just strap a phone on
it switch my GPS localization on and let
the Navigator track the position of the
phone and hence the robot inside the map
so it requires that we have a good GPS
signal and map of the environment and
then that sums up your solution now are
we relying on any unrealistic
assumptions well indoors here GPS signal
doesn't work right
and also outdoors is not as reliable in
narrow streets so that's out then how
about the map well new buildings get
erected old buildings hopefully old
buildings get demolished new streets get
built so we cannot really rely on a
pre-built map so in computer vision and
robotics we do this we try and estimate
the position of the robot and what the
environment looks like at the same time
using onboard sensors like a camera so
we humans are pretty good at
understanding roughly how the camera has
been moving in order to capture a video
like this so the question is how can we
make the robot understand how the camera
has been moving just by looking at a
video well the way we do it is that we
pick natural scene features to serve as
landmarks like the square patches that
you see here and judging how these
features move from one image to the next
we start reasoning about how the camera
has been moving in order to capture a
video like this so this video was
captured with handheld camera but it's
exactly these same principles but we use
later on to showcase that it is possible
to automate the flight of a small
aircraft this big using one onboard
camera and an inertial sensor so you'll
see in a second that the kind of maps
we're building are very different from
Google Maps there are actually point
cloud maps and the system is also
estimating the camera pose while
forgetting past camera poses why because
the onboard capabilities the onboard
resources of the drone are limited so in
order to be able to process new data in
a timely manner sometimes we need to
forget about past experiences so this
system
it was open source so we released it to
the public and it was also the basis of
many systems that came after it but I
have to say that in order to capture a
video like this a pretty video like this
there needed to be a lot of hand tuning
in advance which means that we haven't
actually achieved true autonomy yet why
well one of the problems is that when
you're perceiving the world through a
camera lens you're prone to illumination
changes and seasonal changes so we've
been working on algorithms to robust if
I plays recognition to really teach our
robots to understand when they're coming
back to a location where they have been
to before and here are results from a
system like this which when it's given a
query image is trying to find a matching
image from a database return and
matching image from a database if it
exists despite the fact that it might
have been captured at a different time
of day or a different season so how do
we achieve this well we train our system
with lots and lots and lots of examples
of pairs of images of the same location
taken in February and in June so the
system really learns what makes a place
distinctive with respect to other places
but also what changes in the scene are
transient such that the system can
suppress them when deciding whether the
query image corresponds to something
that the robot has seen before so once
we have a basic point cloud map of the
environment the question then is how can
we build a richer scene representations
such a terrible can interact with the
scene to reach out and grasp an object
or to avoid obstacles while flying so
how useful would it be if we had drones
that are intelligent enough to override
the pilots commands in cases where he or
she is driving it into a wall something
like semi autonomous driving but for
drones so here is an example where drone
is densifying the very local
surroundings of the camera which can
later on be used to really avoid
obstacles and really complete the loop
of navigation such as since in this
example here so the drone takes off and
it knows nothing about the scene and we
give it a goal position
outside the slide on the right-hand side
outside the slide so that goal position
acts like a magnet so the drone really
just wants to arrive at the goal but as
it perceives new parts of the scene it
starts understanding that there are
obstacles in the way so it needs to
constantly replan its path such that I
can avoid these obstacles and still
reach the goal which brings me to the
final challenge of how can we have
multiple robots collaborating with each
other now I mentioned before that drones
have limited resources onboard and
sometimes when they are exploring for a
long time some scene then they need to
forget about past experiences so in
order to combat this and also help
robots collaborate with each other in
one of our works we are introducing a
central server that can be a laptop as
in this case or it can be the cloud if
you'd like so as a drones take off they
know nothing about the scene and they
know nothing about the rest of the fleet
each other so they start building a map
of their surroundings trying to estimate
where their current pose is with respect
to that map but before forgetting any of
their experiences they send them down to
the server so the server acts like a
bookkeeping entity taking information
from everyone and because it's also a
central server it can sit on the ground
it can have a stronger computational
capability so it can do things like
identifying overlap between maps merge
maps and also send an information back
to the agents for example send portions
of the map that are relevant to this
drone here but another drone has mapped
in the past so this system really brings
the basis for any collaboration that can
happen on top of that for example to
share the load of an object to be
transported so with these three
challenges we are aiming to teach robots
to see and collaborate because I believe
these are key capabilities and pushing
for further progress in robotic autonomy
today and you might ask so what are the
applications of this technology
well the possibilities are endless
whether ones want to digitize and
logical structure to have a robotic team
to help in search-and-rescue or crop
monitoring then computer vision and
robotics have a lot to offer and can
have a great impact I did spend most of
my talk talking about drones because
they pose some of the most difficult
challenges for robotic perception
because they have limited resources on
where they move in 3d
they're very agile they're very fast so
we argue that it is the same principle
so if you manage to do this for drones
we can use the same principles on other
kinds of platforms for example to
automate train navigation or to bring
robots in the construction-site or even
to have multiplayer games using our
phones and I'll leave you with this
thought as robots become more and more
autonomous they promise to transform our
lives this transformation can be
marvelous but we need to continue
understanding it and continue guiding
its path to this end thank you very much
[Applause]
[Music]
you
[Music]