sensor fusion is an integral part of the design of autonomous systems things like
self-driving cars radar tracking stations and the Internet of Things all
rely on sensor fusion of one sort or another so the questions I would like to
answer in this video are what is sensor fusion and how does it help in the
design of autonomous systems and this will be a good first video if you're new
to the topic because we're gonna go over a broad definition of what it is and
then show a few scenarios that illustrate the various ways sensor
fusion can be used so I hope you stick around I'm Brian and welcome to a MATLAB
Tech Talk the high-level definition is that sensor fusion is combining two or
more data sources in a way that generates a better understanding of the
system better here refers to the solution being more consistent over time
more accurate and more dependable than it would be with a single data source
for the most part we can think of data is coming from sensors and what they're
measuring provides the understanding of the system for example things like how
fast it's accelerating or the distance to some object but a data source could
also be a mathematical model because as designers we have some knowledge of the
physical world and we can encode that knowledge into the fusion algorithm to
improve the measurements from the sensors to understand why let's start
with the big picture autonomous systems need to interact with the world around
them and in order to be successful there are certain capabilities that the system
needs to have we can divide these into four main areas since perceive plan and
act since refers to directly measuring the environment with sensors it's
collecting information from the system and the external world for a
self-driving car for example this sensor suite might include radar lidar visible
cameras and a whole bunch more but simply gathering data with sensors isn't
good enough because the system needs to be able to interpret the data and turn
it into something that can be understood and acted on by the autonomous system
this is the role of the perceive step to make sense of well the sensed data
for example let's say that this is an image from a vehicle camera sensor the
car has to ultimately interpret the blob of pixels as a road with lane lines and
that there's something off to the side that could be a pedestrian about to
cross the street or it's may be a stationary mailbox this level of
understanding is critical in order for the system to determine what to do next
this is the planning step where it figures out what it would like to do and
finds a path to get there and lastly the system calculates the best actions that
get the system to follow that path this last step is what the controller and the
control system is doing alright let's go back to the perceive step because I want
to go into a little more detail here this step has two different but equally
important responsibilities it's responsible for self-awareness which is
referred to as localization or positioning you know answering questions
like where am i what am i doing and what state am I in but it's also responsible
for situational awareness things like detecting other objects out in the
environment and tracking them so where does sensor fusion come in well it sort
of straddles sense and perceive as it has a hand in both of these capabilities
it's the process of taking multiple sensor measurements combining them in
mixing in additional information from mathematical models with the goal of
having a better understanding of the world with which the system can use to
plan and act so with that in mind let's walk through four different ways that
sensor fusion can help us do a better job at localization and positioning of
our own system as well as detecting and tracking other objects all right for the
first scenario let's talk about possibly one of the more common reasons for
sensor fusion and that's to increase the quality of the data we always want to
work with data that has less noise and less uncertainty and fewer deviations
from the truth just overall nice clean data and as a simple example let's take
a single accelerometer and place it on a table so that it's only measuring the
acceleration due to gravity if this was a perfect sensor the output
would breed a constant 9.81 m/s^2 however the actual measurement will be
noisy how noisy depends on the quality of the sensor and this is unpredictable
noise so we can't get rid of it through calibration but we can reduce the
overall noise in the signal if we add a second accelerometer and average the two
readings as long as the noise isn't correlated across the sensors fusing
them together like this reduces the combined noise by a factor of the square
root of the number of sensors so for identical sensors fuse together we'll
have half the noise of a single one so in this case all that makes up this very
simple fusion algorithm is an averaging function now we can also reduce noise by
combining measurements from two or more different types of sensors and this can
help if we have to deal with correlated noise sources for example let's say
we're trying to measure the direction your phone is facing relative to north
we could use the phone magnetometer to measure the angle from magnetic north so
that's pretty easy however just like with the accelerometer this sensor
measurement will be noisy and if we want to reduce that noise then we may be
tempted to add a second magnetometer however at least some contribution of
noises coming from the moving magnetic fields created by the electronics within
the phone itself this means that every magnetometer will be affected by this
correlated noise source and so averaging the sensors won't remove it now two ways
to solve this problem are to simply move the sensors away from the corrupting
magnetic fields which is hard to do with a phone or you can just filter the
measurement through some form of a low-pass filter which would add lag and
make the measurement less responsive but another option is to fuse the
magnetometer with an angular rate sensor a gyro the gyro will be noisy as well
but by using two different sensor types we're reducing the likelihood that that
noise is correlated and so they can be used to calibrate each other the basic
gist is that if the magnetometer measures a change in the magnetic field
the gyro can be used to confirm if that rotation came from the phone physically
moving or if it's just from noise there's several different fusion
algorithms that can accomplish this blending but a common filter is
probably one of the more common methods and the interesting thing about common
filters is that a mathematical model of the system is already built into the
filter so you're getting the benefit of fusing together sensor measurements and
your knowledge of the physical world if you want to learn more about common
filters check out the MATLAB Tech Talk video that I've linked to in the
description alright the second benefit of sensor fusion is that it can increase
reliability if we have two identical sensors fused together like we have with
the averaged accelerometers then we have a backup in case one fails of course
with this scenario we lose quality if one sensor fails but at least we don't
lose the whole measurement we can also add a third sensor into the mix and the
fusion algorithm could throw out the data of any single sensor that's
producing a measurement that differs from the other two an example here could
be using three pitot tubes to have a reliable measure of an aircraft's air
speed if one breaks or reads incorrectly then the air speed is still known using
the other two so duplicating sensors is an effective way to increase reliability
however we have to be careful of single failure modes that can affect all of the
sensors at the same time an aircraft that flies through freezing rain might
find that all three pitot tubes freeze up and no amount of voting or sensor
fusion will save the measurement again this is where fusing together sensors
that measure different quantities can help the situation the aircraft can be
set up to supplement the airspeed measurements from the pitot tubes with
an airspeed estimate using GPS and atmospheric wind models in this case
airspeed can still be estimated when the primary sensor suite is unavailable
again quality may be reduced but the airspeed can still be determined which
is important for the safety of the aircraft now losing a sensor doesn't
always mean that the sensor failed it could mean that the quantity that
they're measuring drops out momentarily for example take a radar system that's
tracking the location of a small boat on the ocean the radar station is sending
out a radio signal which reflects off the boat and back again
round-trip travel time the Doppler shift of the signal and the azimuth and
elevation of the tracking station are all combined to estimate the location
and range rate of the boat however if a larger cargo ship gets between the radar
station and the smaller boat then the measurement will shift instantly to the
location and range rate of that blocking object so in this case we don't need an
alternate sensor type or a secondary radar tracking station to help when the
measurement drops out because we can use a model of the physical world an
algorithm could develop a speed and heading model of the object that's being
tracked and when the object is out of the radar line-of-sight the model can
take over and make predictions this of course only works when the object you're
tracking is relatively predictable and you don't have to rely on your
predictions long term which is pretty much the case for slow-moving ships okay
the third benefit of sensor fusion is that it can be used to estimate
unmeasured States now it's important to recognize that unmeasured doesn't mean
unmeasurable it just means that the system doesn't have a sensor that can
directly measure the state were interested in for example a visible
camera can't measure the distance to an object in its field of view a large
object far away can have the same number of pixels as a small but close object
however we can add a second optical sensor and through sensor fusion extract
three-dimensional information the fusion algorithm would compare the scene from
the two different angles and measure the relative distances between the objects
in the two images so in this way these two sensors can't measure distance
individually but they can when combined now in the next video we're going to
expand on this concept of using sensors to estimate unmeasured States by showing
how we can estimate position using an accelerometer and a gyro for now though
I want to move on to the last benefit that I'm going to cover in this video
sensor fusion can be used to increase the coverage area let's imagine the
short-range ultrasonic sensors on a car which are used for parking assist these
are the sensors that are measuring the distance to nearby objects like other
parked cars and the curb to let you know when you're close to impact each
individual sensor may only have a range a few feet and a narrow field of view
therefore if the car needs to have full coverage on all four sides additional
sensors need to be added and the measurements fuse together to produce a
larger total field of view now more than likely these measurements won't be
averaged or combined mathematically in any way since it's usually helpful to
know which sensor is registering an object so that you have an idea of where
that object is relative to the car but the algorithm that pulls all of these
sensors together into one coherent system is still a form of sensor fusion
so hopefully you can start to see that there's a lot of different ways to do
sensor fusion and even though the methods don't necessarily share a common
algorithm or even have the same design objective the general idea behind them
is ubiquitous use multiple data sources to improve measurement quality
reliability and coverage as well as be able to estimate states that aren't
measured directly the fact that sensor fusion has this broad appeal across
completely different types of autonomous systems is what makes it an interesting
and rewarding topic to learn in the next two videos we're going to go into more
detail on sensor fusion for localization and for multi object tracking in the
next video in particular we're going to show how we can combine an accelerometer
magnetometer and a gyro to estimate orientation so if you don't want to miss
that and future Tech Talk videos don't forget to subscribe to this channel also
if you want to check out my channel control system lectures I cover more
control theory topics there as well I'll see you next time