how do autonomous vehicles see you've
probably heard of lidar sensors or other
weird cameras they are using but how do
they work how can they see the world and
what do they see compared to us
understanding how they work is essential
if we want to put them on the road
especially if you work in the government
or build the next regulations but also
as a client of these services we
previously covered how tesla autopilot
sees and works but they are different
from conventional autonomous vehicles
tesla only uses cameras to understand
the world while most of them like waymo
use regular cameras and 3d lidar sensors
these lidar sensors are pretty simple to
understand they won't produce images
like regular cameras but 3d point clouds
lidar cameras measure the distance
between objects calculating the pulse
lasers traveling time that they project
to the object this way they will produce
a very few data points with valuable and
exact distance information as you can
see here these data points are called
point clouds and it just means that what
we will see are just many points at the
right positions creating some sort of 3d
model of the world here you can see how
lidar on the right isn't that precise to
understand what it sees but it's pretty
good to understand that with very little
information which is perfect for
efficiently computing the data in real
time an essential criteria for
autonomous vehicles this minimal amount
of data and high spatial precision is
perfect because coupled with rgb images
as shown on the left we have both
accurate distance information and the
accurate object information we lack with
lidar data alone especially from far
away objects or people
this is why waymo and other autonomous
vehicles companies use both kinds of
sensors to understand the world still
how can we efficiently combine this
information and have the vehicle
understand it and what does the vehicle
end up seeing only points everywhere is
it enough for driving on the roads we
will look into this with a new research
paper by waymo and google research
to release many advancements like this
one the researchers needed to run many
many experiments and be super organized
plus their code needs to be easily
reproducible and near perfection as many
people will depend on it in life or
death situations fortunately these are
two strong points of this episode
sponsor weights and biases if you want
to publish papers in big conferences or
create the future of autonomous vehicles
i think using weights and biases will
certainly help it changed my life as a
researcher and for my work at design
stripe weights and biases will
automatically track each run the hyper
parameters the github version hardware
and os used then you can easily create
your own workspace using filters groups
and your own panels to display anything
you need to analyze how can you not be
well organized with such a tool it
basically contains everything you need
for your code to be reproducible without
you even trying for your future
colleagues and people that will
implement your amazing work please take
the time to ensure your work is
reproducible and if you want help with
that try out weights and biases with the
first thing we do
i think i couldn't summarize the paper
better than the sentence they used in
their article we present fortinet which
learns how to combine 3d point clouds in
time and rgb camera images in time for
the widespread application of 3d object
detection in autonomous driving i hope
you enjoyed the video please subscribe
and i'm just kidding let's dive a little
deeper into this sentence this is what
the 3d object detection we are talking
about looks like and it's also what the
car will end up seeing it's a very
accurate representation of the world
around the vehicle with all objects
appearing and precisely identified how
cool does that look and more interesting
how did they end up with this result
they produced this view using lidar data
called point clouds in time or pcit and
regular cameras are here called rgb
videos these are both four dimensional
inputs just like we humans see and
understand the world the four dimensions
come from the video being taken in time
so the vehicle has access to past frames
to help understand context and objects
to guess future behaviors just like we
do creating the fourth dimension the
three others are the 3d space we are
familiar with we call this task scene
understanding and has been widely
studied in computer vision and has seen
many advancements with the recent
progress of the field and machine
learning algorithms it's also crucial in
self-driving vehicles where we want to
have a near perfect comprehension of the
scenes here you can see that the two
networks always talk to each other with
connections this is mainly because when
we take images we have objects at
various ranges in the shot and with
different proportions the car in front
will look much bigger than the car far
away but we still need to consider both
like us when we see someone far away and
feel like it's our friend but still wait
closer to be sure before calling his
name the car will lack details for such
far away objects to patch for that we
will extract and share information from
different levels in the network sharing
information throughout the network is a
powerful solution because neural
networks use small detectors of fixed
size to condense the image the deeper we
get into the network meaning that early
layers will be able to detect small
objects and on the edges or parts of the
bigger objects deeper layers will lose
the small objects but be able to detect
large objects with great precision the
main challenge with this approach is
combining these two very different kinds
of information through these connections
the lidar 3d space data and more regular
rgb frames using both information at all
network steps as described earlier is
best to understand the whole scene
better but how can we merge two
different streams of information and use
the time dimension efficiently this data
translation between the two branches is
what the network learns during training
in a supervised way with a similar
process as in the self-attention
mechanisms i covered in previous videos
by trying to recreate the real model of
the world but to facilitate this data
translation they use a model called
point pilers which takes point clouds
and gives a two-dimensional
representation you can see this as a
pseudo image of the point cloud as they
call it creating somewhat of a regular
image representing the point cloud with
the same properties as the rgb images we
have in the other branch instead of the
pixels being red green blue colors they
simply represent the depth and positions
of the object x y z coordinates this
pseudo image is also really sparse
meaning that the information on this
representation is only dense around
imported objects and most probably
useful for the model regarding time as i
said we simply have the fourth dimension
in the input image to keep track of the
frames these two branches we see are
convolutional neural networks that
encode the images as described in
multiple of my videos and then decode
this encoded information to recreate the
3d representation we have here so it
uses a very similar encoder for both
branches shares information with each
other and reconstructs a 3d model of the
world using a decoder and voila this is
how the weimo vehicles see our world it
can process 32 point clouds in time and
16 rgb frames within
164 milliseconds producing better
results than other methods this might
not ring any bell so we can compare it
with the next best approach that is less
accurate and takes 300 milliseconds
almost double the time to process
of course this was just an overview of
this new paper by google research and
weymouth i'd recommend reading the paper
to learn more about their models
architecture and other features i didn't
dive into like time information's
efficiency problem it's linked in the
description below i hope you enjoyed the
video and if you did please consider
subscribing to the channel and
commenting what you think of this
summary i'd love to read what you think
thank you for watching and i will see
you next week with another amazing paper
[Music]
you