Oh world it's Siraj and lo robots
We're gonna learn a bit about how military
Robots work and then teach a simulated robot how to walk using a reinforcement learning technique called proximal policy
optimization
Military warfare is as old as mankind and the better the technology the more likely the chance of winning is so it makes sense that
Militaries want the most advanced weaponry they can get their hands on by whatever means necessary
ideally militaries can minimize deaths of their own soldiers while
Maximizing their effectiveness that means removing the human from the loop as much as possible
Which is where the field of military robotics comes in the first wheel advances their team from none other than
electrical wizard, Nikola Tesla in 1898
Tesla demoed a controlled motor boat to a government representative it laid the foundation for the first
unmanned vehicles and weapons as World War 1 began Germany used
radio-controlled motor boats to ram into enemy ships containing more than 300 pounds of explosives during both world wars
Inspired by Tesla's initial invention using radio waves as a communication signal was a revolutionary idea
The Germans use it to deploy the first remotely piloted aerial drone called the fritz they dropped a device at a high altitude
From a bomber then Bombardier would steer the fritz via a radio link using a joystick
And it wasn't just the Germans getting in on the radio-controlled fun Americans dropped more than
450 radio remote-controlled glide bombs during World War two
Remote-controlled vehicles on land air and sea have improved in the decades since World War two as more than 40 countries have developed
Technological breakthroughs in that regard there are three important trends happening right now that are causing rapid advancement in
military technology
Namely the plummeting costs and soaring performance of computer hardware
the rise of cloud network robots and of course advances in machine learning
Put together these trends mean. It's getting easier to create more powerful robots soldiers
No longer need to aim at targets if they can just use a convolutional network to detect enemy's
commnets our neural networks that can learn the features of any type of object its trained on be that a certain type of
vehicle or a type of soldier
using a pre trained network soldiers can Don a set of goggles that show a
Computed bounding box around our targets or even feed the computed coordinate to a weapon and have its position
adjusted accordingly and fire automatically effectively using auto aim
Reinforcement learning has been an effective tool to get robots to learn how to navigate virtual environments some soldiers go through
extensive simulated training before being deployed to the real world and an RL agent could prove useful as an
unpredictable and adaptive enemy during combat
simulation for a range of soldiers from fighter pilots to ground soldiers these agents could use techniques like
q-learning or policy gradients or
Actor critic to learn how best to avoid enemy fire and even create effective strategies to confuse the enemy before attacking
AI can also be used to change the medium with which soldiers control their weapons
They can issue orders via voice command instead of having to use a keyboard in intense situations
Thanks to developments in natural language processing
Speech recognition is getting really good and recurrent networks helped make this possible
If we have some labeled audio data set we can train a recurrent Network by slicing the audio up into small chunks
Feeding those chunks into the RNN sequentially and generating a prediction for the next chunk
Computing an error back propagated the error gradients and repeating that process
Lots of times since we're current networks are made for sequential data. Audio is a perfect use case
After training they can recognize human speech with incredible accuracy and this essential for mission-critical applications
Governments have mountains of data from surveillance systems and satellites of all kinds
finding disturbances using unsupervised learning techniques like anomaly detection via
Auto-encoders war doesn't just have to happen in real life
It can happen online as well today the process of finding and countering bugs
Hacks and all sorts of cyber infection vectors is still effectively artisinal
professional security experts search millions of lines of code to find and fix vulnerabilities
That could be taken advantage of by users with ulterior motives all of this could be automated using deep learning to find the features of
vulnerable code and learning to patch bugs by itself so rule-based intelligence can only do so much but by
Integrating learning algorithms machines can become far more capable when a human is in the loop
They are able to make the final call the highest level decision
That's the one that the human makes but as AI gets better its able to make more decisions for itself so how many?
Decisions should we delegate to AI how much is too much?
Take the idea of a swarm for example any aircraft manned or unmanned can be brought down by a single missile
But a swarm can take multiple hits and keep going
It's a collection of drones that can coordinate together as a single entity
One could take out enemy systems while the other creates a distraction while the other picks up important cargo
evolutionary techniques like particle swarm optimization
Can make this happen the US Department of Defense
Recently released a document called a human systems roadmap review that reveals a plan to use AI to create
autonomous weapons using social media analytics to make decisions on lethal force with minimal human
Involvement it shows that while having a human in the loop is necessary for the near-term
Therefore turn vision is self-aware systems they define this as systems that have perception
reasoning and intelligence allowing for entities to have existence intent
relationships and
understanding in the battle space relative to a mission
That means drones that coordinate amongst themselves
make decisions for themselves
Prioritize by themselves and of course decide who to kill and when to kill them by themselves
That's kind of scary and reminiscent of the infamous Skynet system from the movie
Terminator in the movie the military system becomes self-aware and decides to kill all humans and no one can shut it down
we know of one system that can't be shut down and that's Bitcoin because the blockchain miners must verify
Transactions using the proof-of-work algorithm it would require more computing power than the 500 fastest
Supercomputers in the world combined to shut it down and no one has that much computing power an AI agent
That lives on this kind of blockchain couldn't be shut down and if its objective was to
Say destroy humanity and it has access to weapons of mass destruction
That's how Skynet happens. IRL, so how do we stop this?
we've got to spread awareness and
Educate ourselves on how AI works the more people that understand the power of AI the less likely it is that?
Governments will abuse this power and the more likely it will be used for beneficial purposes
more research into creating explainable AI systems that give us a detailed list of
why an AI made a certain decision will help unravel the black box of deep learning and help fuel a debate for policy makers as
they regulate this technology so to give an example of how a robot can learn to walk we can use a
reinforcement learning technique called proximal policy optimization
This is two randomly initialized neural networks and a teacher that rewards forward progress
The policy gradient technique takes steps in the direction that improves the policy a similar technique is called trust region policy
Optimization or TRP Oh the idea is to take steps in the direction that improves the policy while simultaneously
Not straying too far from the old policy
that's the difference making too big a change from the previous policy in high dimensional environments can lead to a
dramatic decrease in performance a little forward lean can help running speed but too much causes a crash a
Naive solution is to take minuscule policy steps
But the question then becomes how small a step TR Pio takes a principled approach to controlling the rate of?
Policy change the algorithm places our constraint on the average KL divergence
between the new and old policy after each update so proximal policy
Optimization is an implementation of TR Pio that adds the KL divergence
Term to training a loss function with this loss function in place we can train the policy with gradient descent
like a typical neural network in our PPO algorithm
We capture sequences of states actions and rewards from our environment and added to our data batch line 10 adds value
estimates to each visited state from the rollouts
with predicted state
values in hand we
Calculate the advantages and add these to the data set in line 11 the advantage of a state action is how much better
or worse an action performs than the
expectation of present policy from the same state we update the policy in line 13
finally we update our value function to reflect our latest data in
114 we use the present data batch and the previous data batch to smooth changes in the value function
For our policy update function we store the old policy and compute the KL divergence as we make policy
gradient updates and after only
25,000 training episodes our humanoid will start learning how to walk
It's pretty hilarious to watch the progress in the meantime all right so three ending points here
militaries can use AI to create autonomous weapon systems and that means less and less of a need for humans in the loop a
Skynet's like scenario can occur the public isn't made aware of AI dangers and governments go unchecked and
Proximal policy optimization uses two neural nets and the teacher to forward progress to train an AI to complete an objective
This week's coding challenge is to use the PPO technique on a game of your choice
Details are in the readme poster github links in the comments section and winners will be announced next week
Please subscribe for more programming videos and for now I've got to go prevent Skynet, so thanks for watching