I’m at the University of Western Australia. That is one of the world’s most advanced self-driving
vehicles, and this pattern should confuse it enough into thinking that I’m a banana and
running me over. Please, don’t try this at home.
Before I recuperate my university fees by committing insurance fraud,
we first need to try to understand what’s going on inside our autonomous bus’ brain.
We’ve been trying to get autonomous vehicles working for the better part of 80 years. In the
early days we had some really exciting successes by embedding metal magnetised road spikes into
the roads which our vehicle could navigate along. While this was surprisingly successful
especially given the rudimentary level of our computing technology at the time, it's not a
particularly scalable technology. Australia alone has about 800 thousand kilometres of roadways,
only 40% of which are actually paved over. While it might work to get you from A to B, if you want
to get to any other location it's really not going to work out. For that we need something that is
infinitely more scalable. Maybe GPS could work. While it’s great at letting you know wherever you
are on the planet, it's not particularly precise: you know your car is on the road but you don’t
know which bit of the road it's on. It might crash into a bus going through the other lane.
We already use sight-based navigation to get around the roads today. We use it to stay
within a lane, to stop ourselves from crashing into the bus ahead, and importantly for knowing
what hazards are up by looking at street signs. On top of that, cameras are getting pretty cheap.
Because of this most of the serious attempts at getting self-driving vehicles
commercially available use visible light and high definition cameras.
However, this means that we really need to understand image recognition.
Given how easy it is for us to spot and identify objects we often forget that this is actually
quite an impressive feat. Even within the animal kingdom we have especially well-developed vision.
There is a compelling theory that our unusually complex brains came about in order to facilitate
vision processing. Individuals which could tell the subtle differences between the shape and hue
of safe and unsafe berries, or could spot a lion hiding in wait in the long grasses,
had a competitive advantage and were able to pass on these successful vision genes onto future
generations. We think, according to this theory, that science, literature, art and engineering,
while all being very useful, are actually just unexpected byproducts of these complex brains
developed for vision processing. When we try to replicate how we humans identify objects within
a computer, especially a computer with limited processing power, we run into a few problems.
To see why that's the case, let's take the basic example of street signs.
Fortunately for us, they are nice, regular, standardised shapes with high contrast and bold
patterns. They’re also pretty universal so once you’ve programmed the street signs for one country
it should mostly carry over to the next. This standardisation is as a result of the
1924 National Conference on Street and Highway Safety which laid out the ground rules for traffic
signs which still influence the streetscapes around us today. Apparently, the committee was
really afraid of pointy objects because when they elected to use a shape-based system of
identifying hazards they reserved shapes with more corners to denote more dangerous things.
Pedestrian warnings – not that dangerous, let’s make it a triangle with three points. Speed limit,
a bit more dangerous lets go four. Stop sign, oh boy, you get eight sides! The committee
thought that train level crossings were the most dangerous of all, so those signs didn’t get ten,
twelve, or every 50 corners – they got infinity corners, or what we commonly refer to as a circle.
I’m in Australia so some of the most important street signs for us to recognise
are those that say ‘Warning! Kangaroos, cows, bobtails and snakes crossing!’.
Which of those animals do you think is most dangerous to hit going at 110 kilometres per hour?
Regardless of which one you really don’t want to hit, ultimately we need to be sure that our car
is able to recognise the street sign when it's quite far away and respond accordingly.
How are we going to get a computer to recognise our sign?
Here is one approach which may be an option. Ok. It’s a yellow square with a nice thick
black boundary, rotated 45 degrees and a silhouette kangaroo in the middle.
Let's start out with that boundary; four lines meeting at 90 degree angles. But what’s a line?
Well, it's straight? Fantastic! But that doesn’t really correspond to computer code
so we’re going to need to come up with some sort of algorithm.
Let's look at the pixels and just consider the ones that are dark. Pick one and now consider
the pixels which surround it. Are any of those also dark? Nope, ok let's find a different one.
This time are any of those dark? Yes, cool, ok now have a look at its adjacent tiles,
are any of those also dark? Great. Now look around its adjacent tiles. And that one’s adjacent tiles.
And another. And another. Now plot the coordinates together. Is the gradient roughly consistent?
Perfect - that means we’ve found ourselves a line. Repeat this a bunch more times and if
the plotted lines match up in a roughly diamond shape then we’ve found ourselves a street sign.
Now that we’ve identified our square, we need to check if it's a warning sign or just a random box
on the side of the road. We do this by taking the average RGB value from inside our rectangle and
corresponding it to either yellow or not yellow: telling us if it's a warning sign or just a random
box on the side of the road. Now we need to work out what the caution sign is warning us against.
If it’s kangaroos then our response should be to slow down and be a bit more cautious. If, on the
other hand, its warning us about drop bears then the only appropriate response is to wind up the
window and get out of there as fast as possible. For this we need an even more advanced
line-plotter algorithm, and maybe include a few matrix transformation
routines to take into account if you’re looking at our kangaroo from an angle.
But not all kangaroo drawings look the same - so we need a database of potential designs.
What if there are branches in the way? Well, that would mess up our shape so we need to accept a
range of kangaroo-like objects and maybe include a function that removes branches from the image.
Now do that for every single street sign, zebra crossing, set of traffic lights,
and road marking that you can think of. And a few which you might have forgotten. You
can start to appreciate how this quickly becomes a very complex coding nightmare.
Rather than getting an underpaid, overworked, sleep-deprived, caffeine-addicted,
undergraduate student to manually program every single object that your car might come across,
another option is to get the computer to do that for you. This is what's broadly defined as machine
learning. One approach is what is called a neural net; similar to the neurons within a human brain.
Rather than taking a set of instructions and just going through it one-by-one,
instead we have a complex web of relations between inputs,
analysis, and outputs. Here is how you might apply it to a 16-pixel image.
We start out with 16 neurons each of which look at the specific brightness value of a single pixel.
They then send this value to a set of 20 nodes which each compare the
output of these signals and give their own output based on some rule. For example:
in this node it might be to find the average brightness between these four adjacent pixels,
ignoring all of the other inputs. The results for all 20 of these comparisons
are then sent through to the next layer of the network which does another round of comparisons.
Subsequent layers do more and more analysis, eventually leading to the final output layer,
which by this point ‘knows’ that the input image was of the letter A and gives ‘A’ as the output
result. Putting in a different image into the exact same neural net would give the letter “B”.
While this may look like magic, this complex behaviour derives entirely from simple relations.
Unlike with normal programming, no one is manually going through the net and manually changing how
the nodes do their weighting. Instead, a special training program is doing that for itself.
Our neural net training regime is basically pattern recognition turned up to 11. We start out
with a few hundred randomly assigned networks, each with slightly different node weightings.
Obviously when we chuck in some sample images the results we’re going to get are going to be pretty
much gibberish. However, a few of them are going to be slightly less gibberish than the other ones.
These are the most fit of their generation and, in a process similar to evolution,
are allowed to combine with each other - giving us the next generation. We also include a little
bit of random mutation which means that the next generation can be even better than their parents…
or alternatively completely garbage. We take the ones that are really good
and combine those together again; giving us the next generation. Which we combine the best ones of
again, giving us another generation, and another and another and another. Eventually after maybe
hundreds or even thousands of generations, and a huge amount of processing power,
eventually we get a network that works pretty-much all the time with pretty-much all of the samples.
What happens during this training is that the net is slowly identifying key characteristics
which correspond to each of the objects being analysed. If you or I were describing a bus
we’d probably say something about it being bright yellow, with lots of windows,
and big wing mirror. The difference being that the net will occasionally identify characteristics
which, while true in the training data, don’t necessarily correspond to the real world.
Here is one particularly dangerous attack created by an American research team.
The team trained up their own neural net using some publicly available street sign data.
After a couple of weeks of training they could reliably identify stop signs, such as this one.
Next, they attached some white and black bars. You and I can still tell that it's a stop sign,
but the team’s neural net could not. Instead, it was certain that it was a speed-limit 45.
If the neural net was in control of an autonomous vehicle then it would
have driven straight through the intersection. Instead of recognising the shape or letters S-T-O
and P that you and I might recognise in a stop sign, instead I think the neural net has latched
onto something different. I think that because the number four always has white patch in the middle
and the number 5 some sort of line at the bottom, it’s seen these characteristics - knowing that
it only ever sees them in the context of a speed limit 45 sign - and just run with that.
This is always going to be an issue when we have limited processing power and storage. If
we have a complex way of identifying a sign and a simple way, and both are equally successful with
our training set, then the evolutionary pressures of this algorithm will prefer the simple option.
So what makes my T-shirt so dangerous? Here is a demonstration come up with the folks at Google,
working on the Google Vision AI. Here they are identifying the subject of an image,
and yep they’ve correctly identified that it’s a banana. And now they’re adding in a picture
of a toaster. But no simple toaster picture is going to make Google confuse what the subject
of the image is. No sir, they are much smarter than that. Oh dear! Pretty much 100% certain
that it's a toaster. And this thing doesn't even look like a toaster. What’s going on?
Here is what the classifier algorithm thinks is the pure essence of toaster. When the neural net
sees it, it ignores everything else and just looks for the toaster. This is what the Google
team dubbed as adversarial patches, and they’re surprisingly simple to create. Here is the one
that I have on my t-shirt. It’s what the algorithm thinks is a banana. As long as it can get a clear
shot of the patch then it should ignore me entirely, and potentially run me over. All it
thinks its doing is running over a banana. Let's head to the real world to test it out for real.
So what’s going to happen with the UWA bus. I have the right t-shirt on, it has the wrong sensors,
in a few metres I’m going to become a banana-flavoured pancake. Except… it stops.
That's because although we may have been able to confuse the neural net, ultimately there was a
backup system: there was some infrared proximity sensors which stopped when I came too close.
No autonomous system should rely on just one level of redundancy. In addition to our
proximity sensors, we could also cross-reference the government databases which store the location
and types of all the countries’ street signs. We could then compare this with what we’re observing
on the streets around us. On top of that, Google appears to be improving their vision AI.
Here is a Python program I’ve written using the Google Vision AI. We take a photograph and then
the Google AI identifies the most likely candidates for the subject of that image.
Here we can see that we’re correctly identifying a banana. Here we have some sort of book.
And finally we have a collection of tools. Now, we expect that this should have worked: remember
that Google AI was trained for just this sort of use-case. But, what I want to know, is what
happens when we add in these adversarial patches designed specifically to trick the Google AI.
Here we see that the banana is still a banana, the book still a book, and this collection of
tools is still just a collection of tools. Either Google were lying to us
when they said that they had been able to trick their own AI… or more likely the version AI that
I’m using has been improved upon the 2017 edition which was discussed in the paper.
Like with a game of whack-a-mole, what I think must be happening
is that whenever research comes out that says the AI can be fooled in some way, the engineers are,
behind the scenes, quietly patching the issue. While this may work fine in the short term,
I’d rather not trust my life to an AI which may-or-may-not stop at the next stop sign before
Google manages to catch and then patch the issue. In order to get a truly safe autonomous vehicle,
not only does it need to reliably identify objects, but it also needs to understand
how these fit within the broader road context. As our vehicles become smarter and smarter,
where do we draw the line between the human passenger and the vehicle becoming more like us?
This has been James Dingley from the Atomic Frontier. Keep Looking Up.