in this video i want to introduce you to
track level fusion
or track to track fusion you may hear
these two terms used interchangeably
now we're not going to go into any
particular algorithm rather i want to
explain what track level fusion is
and how it's different from the tracking
algorithms we've already covered in this
series
and hopefully along the way this will
provide some motivation
and intuition into why track level
fusion is necessary in some situations
and then also we're going to go through
some of the challenges associated with
it
so i hope you stick around i'm brian and
welcome to a matlab tech talk
to begin let's compare two different
tracking architectures
we'll start with an architecture that
we'll call central level tracking
you'll probably be used to this if
you've been following along with this
series
in this architecture the sensor
detections are fed
into a tracking algorithm that assigns
the detections to tracks
and updates the states and covariances
of the tracked objects
exactly what we saw in the past few
videos the key here
is that all of the sensor data is fused
or blended together at the same level
and within the same tracker
sort of a mother ship approach where
there's this one centralized unit that
takes in all of the information that's
available
and performs the calculations necessary
to estimate the tracks
now with this architecture we could have
just a single sensor
or multiple sensors that are blended
together and we could be tracking just a
single target
or multiple targets and the targets
themselves can be point
objects or extended objects as long as
sensor measurements feed into a central
tracker
no matter what it's tracking we can
think of it as a central
level tracker now let's compare this
architecture to one that uses
so-called sensor level tracking and
track level fusion
the idea here is that one or more
sensors
feed into a central level tracker just
like the other architecture
but now we have several of these
trackers each fusing together their own
set of sensors
these are the sensor level trackers in
this architecture
each produces its own track estimates
but now we can combine or
fuse together all of these estimated
tracks into a single track set
which we'll call the central tracks
using a track
level fuser as a quick example
one sensor level tracker might say that
there's an object
here with this probability distribution
another sensor level tracker might say
that an object is here with a different
probability distribution
the track level fuser would need to
determine if these two tracks
are two different objects or associate
them
as the same object and if it's the same
object then combine the two estimates
to create a new state and probability
distribution
that is more accurate than either source
track on its own
and we can use this type of architecture
with multiple sensors
and to track single or multiple objects
and for both point
and extended objects so similar
capabilities as central level tracking
but with a slightly different approach a
distributed
approach to solving the problem so
the question at this point might be why
not just use a single tracker
why go through the extra steps of having
multiple trackers and then having to
fuse their tracks together
well to answer that let's look at some
of the benefits of doing track to track
fusion
and some of the scenarios where it's a
more appealing option
and then after that we'll look at some
of the challenges with it and hopefully
you'll be able to form an opinion as to
when you may choose one architecture
over another
track level fusion can be beneficial if
you're worried about access to data
bandwidth compute capabilities and
specialization
let's start with access to data you may
be
forced to use a track level fuser if you
don't have access to the raw sensor data
and this might be the case if you buy a
sensor that has a fusion and tracking
algorithm
built into it for example you might have
a lidar system
that doesn't return a point cloud but
instead is capable of tracking some
number of objects in the scene
and returning the tracks for each of
them
and a track here might be a state vector
with position velocity orientation and
shape
as well as a set of covariance matrices
that indicates how much confidence there
is
in each state in this case in order to
blend
this track information with other
sensors say with a visible camera system
on your autonomous vehicle
you're going to need a track level fuser
now even if you buy or build sensors
that give you access to the detections
it may not be possible to transfer real
time all of that information from the
sensor to the compute element that's
running the tracker
some sensors can generate data rates
that are far too great for the bandwidth
of the communication buses
especially lidar invisible cameras that
are sampling dozens of times per second
and if you have limited bandwidth within
your communication system
that's a limited number of bits per
second that you can send
then you may be interested in reducing
the size of the data that each
sensor is sending track information is
small
compared to say a camera image so if a
local computer
can process the sensor information and
distill it down to a best estimate for
the objects it's tracking
then there's a lot less information that
needs to be sent to the main computer
that's running the track level fuser
now even if bandwidth isn't a problem
and you can send all the data you want
there might still be an issue with
compute capabilities
again imagine there are dozens of
visible cameras and lidar sensors on a
vehicle
a central level tracker would need to be
able to ingest
and process all of that data in one
giant tracking algorithm and this may
take too much processing time to produce
estimates
at the desired sample rate however
if a local computer is processing its
own sensor data
all of that initial processing is being
done in parallel
distributed among many computers and
then the track fusion algorithm
only needs to be able to process the
much smaller track information
which can speed up the entire processing
time considerably
okay so maybe none of this matters to
you because
let's say you have a powerful computer
that's able to handle all of the data at
once
it may still be beneficial to have a
track level fuser
because it allows for the sensor level
trackers to be specialized
to their particular sensor type remember
in a tracker we have to set up motion
models
and sensor models and associate
detections to objects
and existing tracks and we have to tune
it to a particular set of hardware and
expected environment conditions and so
on
and all of that can be much easier if we
build a tracker
that focuses on fusing together say just
the camera data and then we can fuse
those tracks
with ones that are generated by say the
lidar base tracker
in this way we don't have a single
massive tracking algorithm
but many smaller algorithms that can be
easier to set up
tune and test
so each of these are reasons why you may
want to or
need to implement a track level fusion
algorithm on your vehicle
and these all seem like really good
benefits so the question now is
why don't we just do track level fusion
all the time and to answer that
let's look at two of its challenges
reduced accuracy
and correlated noise let's start with
reduced accuracy
we've already said that sensors can
produce a lot of data and the tracker
distills that data down to a state
vector with a lot less
information and as a byproduct of that
process
we may be removing some information that
is useful
information that the track level fuser
no longer has access to
in this way when we fuse tracks together
it can produce a result that is less
accurate than we could get by
fusing together all of the information
at the sensor level
as a quick example of this imagine we're
tracking the same two objects
with two different sensors sensor a has
the following detections
its tracker has generated a track for
the left object
but not for the right it's treating
those detections as spurious noise and
ignoring it
or maybe it just hasn't established a
track yet for it
sensor b is doing the opposite it has a
track for the right object but it's
ignoring the left
detections now fusing these two tracks
together
results in both of them being included
in the central track list
unmodified since there's no other
information to update them with
however if we had grouped all of the
sensor detections together
the single detections would have been
grouped with the others and
could be used to help improve the
estimates
but there is another issue with track
level fusion that arguably is more
important
and that's the problem with correlated
noise if the tracks that were fusing
together are correlated in some way
then we can't just multiply their
probabilities together like we can
in a standard kalman filter as an
extreme example of this
imagine two trackers that are using the
same process model to predict a future
state
each model is initialized with perfect
knowledge of the state of the object
which it got from its own respective
perfect sensor
as the models propagate this perfect
state forward in time our
uncertainty grows due to the process
noise or
errors in the model we now have two
uncertain track estimates
we may try to fuse these two tracks
together so that we have a central
estimate
that we have more confidence in i mean
if two different models predict the same
state
then it seems like we should have more
confidence in the fused solution
than we would in either solution on its
own however
these track probabilities are highly
correlated since they were generated
using the same model
so we shouldn't actually have any more
confidence in the fuse solution
than we do the individual ones because
we essentially just ran the same model
twice
so why would combining the result give
us any more information
so if there's no correlation between
tracks then we want our tracker to take
advantage of this
and increase our confidence in the
solution
but if there is a lot of correlation
then we want the tracker to fuse the
solutions in a way
that doesn't increase our confidence but
here's the real issue
we may not necessarily know how or
if two tracks are correlated and
handling this
unknown correlation is the idea behind
some of the existing fusion algorithms
like covariance intersection and i've
left a link to a resource that goes into
way more depth on this
but let me give you a visualization of
approximately
how it works assume a tracker produces a
probability distribution of where an
object is located on a 2d plane
represented with this oval and a second
tracker produces this distribution
of the same object to fuse these two
probabilities together
we can create a third probability that
bounds the intersection of these two
that is we're looking at where these two
ovals intersect and creating a
distribution that completely includes
that intersection
you can see that this distribution is
smaller than
either of the previous ones indicating
that we have more confidence in the
solution
however as the process noise or the
sensor noise becomes more correlated
and these two probabilities start to
line up you can see how the intersection
grows
until the two probabilities perfectly
align and it becomes the exact
same distribution now this is a
conservative approach to fusing
probabilities
because as you can imagine we may have
distributions
that line up purely by chance and with
no correlation at all
and when that happens the method will
still treat them as though they are
correlated
okay there's one last thing i want to
talk about in this video
so far we've looked at this architecture
where the source tracks into the fuser
come from sensor level trackers however
in some situations
the source tracks may come from other
track level fusers
and this can create some interesting
phenomenon
to illustrate this problem let's imagine
that there are two autonomous vehicles
each with their own set of sensor level
trackers that feed
into a track level fuser the rear
vehicle is positioned
such that it can't see a pedestrian up
ahead but the front vehicle can see it
you could imagine that it would benefit
the rear vehicle to know that there's a
pedestrian up ahead
so that when it does come into view it
doesn't need to waste precious time
establishing a new track it'll already
exist
so to accomplish this both vehicles can
share their central tracks
with each other and fused them along
with their own
estimated tracks but now we've
introduced
a possible problem called rumors and it
happens like this
the front vehicle tells the rear vehicle
about the pedestrian by sharing its
track with it
so now both vehicles know about the
object and are propagating its state
with their own process models each time
step
the front vehicle senses the object and
updates its track
and says hey the object is still here
and the rear vehicle says yep i've been
tracking it as well with my process
model we're good
but let's imagine now that the object
disappears or
moves out of frame of the sensor the
front vehicle
then may drop the track and say hey that
object's gone
but the rear vehicle who is propagating
the state still
may not have dropped the track by then
and tells the front vehicle
hey don't worry about it somebody told
me about this object and i'm still
tracking it
here's its state information even though
all it's doing is propagating the state
that the front vehicle gave it
well now the front vehicle would say
okay i'll keep this track since you're
telling me it's still there
and now a rumor has started that track
persists even though no vehicle is
actually sensing it anymore
so track fusion algorithms need to be
set up in a way that discourages rumor
propagation
which can actually be pretty tricky to
do without discouraging
actual non-rumored tracks
okay so there are more benefits and
challenges with central level
and track level fusion than what i
covered here but these are some of the
major ones
as you might see now there isn't one
tracking approach that's best
suited for every situation hopefully you
can start to piece together which
tracking scenarios could benefit from
the speed
and efficiency of track level fusion and
which scenarios would be better suited
for the less complex
and possibly more accurate central level
tracking
as always if you want to explore further
than what i've covered here i've linked
to several resources that go into more
detail on everything i've talked about
but i hope this short video has helped
if you don't want to miss any other
future tech talk videos don't forget to
subscribe to this channel
and if you want to check out my channel
control system lectures i cover more
control theory topics there as well
thanks for watching and i'll see you
next time