Hello everyone.
I have my shoes couple from Microsoft research and today I'm going to talk about high Fidelity civilizations and specifically making a case for a while they are the next big thing in there.
You know you might have heard of recent advances in.
Machine learning specifically things like reinforcement learning so I'm showing you a snapshot from atari games you know we have had tremendous advances in solving games for instance something like solving Alpha.
Or more most recently solving you know PAC man and that's coming from Microsoft.
So if you think about it these are video games and we have a our technology specifically machine learning technology where you can attempt to solve some of these incredibly hard tasks are superhuman levels.
Right there just ring thing is while this is a great advance the field of A&ML the big question is how does it translate to real world if I know how to play a game of go or how to solve PAC man can these technologies help me solve.
Real hard woltosz for instance trying to fly a drone in an uncertain environment or trying to drive a car with pettistree and's etc.
So one thing to notice that while these are great advances they are closed worlds.
So when a agent needs to make the next move they only need to worry about the next move in the game of go for the next move in the game of pacman they don't have to worry about the exogenous variables things like when things like petition jumping in front of the car or certainly the.
You know the the lights changing the signs etc.
Consequently since this is a closed world and since we have access to these games we have the luxury of generating millions and millions of frame you know limited unlimited amount of data.
And if you are familiar with some of these deep neural networks you realize that they need a lot of data so consequences chicken and egg problem I mean we want to solve this hard am built ask but you know sure we have advances in terms of algorithms and they're all these platforms.
But without the existence of such label data none of this is going to matter so really the bottleneck is going to be how do we get this real data So what I'm going to propose is that let's consider these techniques but now let's think about building a video game.
Which would make reality so specifically so the whole idea is that we are going to think about building virtual worlds which memec reality the entire pipeline in as much detail as possible?
So that you can actually hope that you can use these machine learning methods in real world so the whole idea is very simple it's a video game on steroids as real as possible except that game is being played by an AI agent.
And while playing these games these a agents becom you know they've been experience a lot of different episodes and based on their experience they can learn it up so let me.
You know show you a demonstration from an open source project that we have from Microsoft research is called ayresome how many of you have heard about ayresome.
OK so you know so let me just go through give you a very brief introduction on what eresimus so.
So when we think about you know developing real world technologies you know we have scenarios like that developing autonomous system in real world is incredibly hard trying to test like this takes immense amount of time and energy.
And you know manpower.
So you know especially in Seattle where we get like 3 months ago sunshine you know trying to do this thing is extremely expensive so consequently we embark on a journey of Hey let's try to do this in simulation for us.
Let's try to mimic reality as closely as possible and let's solve those problems simulation and have that simulation is very close to real world we can hope that some of these technologies for transfer right so here is a snapshot.
From our open source simulator which is basically what I'm showing you is a self driving car being simulated in virtual world that's was developed on top of Unreal Engine they are other vehicles what you are seeing.
Is a camera being simulator which is in front of the hood and you can see the you know what the camera is seeing since it's a simulation we have the exact ground truth so consequently you know you know you have the exact segmentation you know where the car is you know how far?
Each tree is you know how far a pedestrian is it doesn't need to be a car it can be a drone so in this case a camera is being similar on top of grown this is what rowena seeing this is the ground truth it knows what it.
This pixel to respond to it knows exact the depth map so now you can imagine that this grown if allowed by you know if an autonomous stack drives this drone and does these.
Experiments millions and millions of times it's going to gather valuable experience and that value will experience can be done fed through machine learning system So what I'm going to do is I'm going to show you 2 or 3 different different kinds of scenarios that we have tackle that will give you a little bit more insight on how this could be used.
So again what is the simulation as I said it's a video game on steroids do anything about a video game you think about the actor in this case it can be a car it can be drawn it can be any robot it can be an iOS device it can be just a bunch of cameras for instance.
Then you have real world physics right so when the drone moves in the environment with the car moves in the environment it needs to mimic how it would how it would move in the real world so there is a physics engine associated with it.
It has models of when it has more love gravity the magnetism you know all those details it has simulations of sensors each and every single sensor that you would put on a robot things like a camera.
Ax rimator gyroscopes minor to major barrameda all of those things are being simulated and they're being simulated to an accuracy such that they would you know they mimic the behavior in the real world so we are simulating all the gyroscopic treffz an all the noise and all the.
All the different noises that come through with these sensors.
And it's a it can be built on top of rendering engine in this case you were saying Unreal Engine but you know it's possible to do it overunity another ring injured as well.
But you let the simulation run and you can collect the data once you have the data which is the goal that you squeezed out of the simulation you can hope to train machine learning and intelligence methods on it.
Again when we think about simulations we think about simulations of sensors so in this case as I said we're trying to mimic each and every sensor So what I'm going to show you here a few statistics so we have simulated Barrow meter.
And a real Barrow meter and what we can do is we can collect races and make sure the statistics of those traces match so that we can be sure of that if we have these sensors pipeline urinae system they they make reality.
We can do an end to end integration testing where we take a real grown and we put it in a simulation with exact configuration all the physical properties fly it in the real world which is the magenta trajectory.
And then you know playback those signals and fly it in the real world and you make sure that the.
Simulation in reality match you know very closely so we have done some of these kind of integration testing.
So let me take a step back and walk you through 3 scenarios where you know we can use the simulation how many of you know about reinforcement learning the whole methodology been reinforced learning OK.
At least fyrv you know but in a nutshell it's a methodology where you have a machine learning agent.
That's taking all different actions that can take in the world so for instance if I have a self driving car it can decide to go straight it can go to set to go left it can go decide to go right can take various actions based on different scenarios.
And if it's going to make a mistake you say that all right you made a mistake I'm going to give you a very high negative reward you shouldn't do that again but where does the right thing you give a positive reward and if you keep doing these kinds of episodes over and over.
For sale you know many million times eventually there is a mechanism there is a machine learning algorithm at the back end it's trying to get that feedback and adapt to it right now you can imagine trying to do reinforcement learning in real world is very expensive with the cars and drones.
Because if they make a mistake you're going to probably do a lot of damage right but however this is the same algorithm that's been used to train Alpha go atari games like chess and things like those so the real big question is how do you then translate to real world.
So that's where the simulation comes in So what I'm going to show you so we have an agent this is a car it doesn't know anything about driving all it has an access to images and based on the image the RGB put map it needs to make a decision.
The car is going to get a high reward if if it wilocity is nontrivial it's basically high and it's going to give a high negative reward if it comes to do something if it crashes if it goes out of the sidewalk right.
So let's see how that works right so it's exploring it doesn't know how to do things right it made a mistake in highlighted reward go back this is simulation I have the luxury of making mistakes millions and millions of times without.
Ever incurring real will cost however you cannot cannot expect this kind of algorithm to work in real world and eventually after you do you know many many iterations it starts to get the hang of all right this is what I actually need to do in order to get the high reward.
Right so eventually you know it basically tries to solve these kinds of episodes so you can see how you know we are utilizing some of the recent advances in machine learning but actually using high Fidelity simulations in order to.
In order to solve real world tasks.
The second example that I'm going to talk about is more in the lines of supervised machine learning which is the more traditional form of you know learning that you guys might be familiar off so the whole idea is that if I have.
A bunch of labeled images I can train a neural network right so in this case.
What I'm showing you is a very bare bones pipeline for a self driving car right you have some kind of a sensor in this case the stereo vision camera and based on the stereo vision camera?
So you can generate some depth map this depth map goes into a mapping system which is slam if you're familiar with and then based on that you can plan each of these boxes you can think of it as a machine learning system at all and air system but however let's just focus on one.
Which is the stereo box what it does is it takes a stereo pair and generates a depth map it just tells me how far things are that's all and we are going to use machine learning to do this right so?
1 easy way to do is is I again use a simulation but what I have done is I have instantiate a stereo pair on the on the car so you can see the left frame and the right frame and sisters since this is a simulation.
I have the ground truth depth map I exactly know how far each pixel is right So what I'm going to do is I'm just going to let this car drive right and this since this is some relation I can perfectly drive this car.
Without bumping into anything but as I'm driving I'm going to record my left frame and the right frame and the ground truth depth map and I do that for hours that are going to give me a million possibly more frames.
At once I have this data I basically train a neural network input to the neural network are these left and right frames and the output is the depth map right and you train this thing right and you have a neural network where you can then feed any left and right frames and outcomes the depth map.
Very simple idea.
The good thing is that you do it and then you can try it on real world images to see if this translates so the great thing is that you know.
Since this is high Fidelity simulation a lot of these kinds of machine learning task translate really well to the real world.
And so.
And the thing is that when you think about power simulation it's not just limited to just machine learning you can do some fantastic things with simulations to hear what I'm showing you is something very bare bones pipeline of how a self.
I'm driving car might be designed you know you have you have to define the architecture of your car for instance the next generation of cars are going to have probably 250 or more sensors are you going to build every single car and test it and then figure out if that's right configuration probably not.
That's the simulation comes in I can take a car chassis and I can instantiate all different kinds of sensors all over the place and let it run in simulation and see how well it works once I'm happy with the belt that's when I go and build a.
You know a simulation.
Let me go back same thing you know you could lies deep learning to build things like you know that actors signed it actors street.
You know street light detectors all kinds of different things all kind of different tasks that are very important and you know Thomas systems then you can integrate and you can build the entire end to end system and you can actually simulate the into an system to see if it works on all right so they are many ways in which simulations can be used.
Very actively throughout.
Here is another example where I have basically taken the same reinforcement learning algorithm but now it's instantiate on a drone and the goal of this drone is to learn to follow these power lines you know why why do you want to do that.
Because you know again you know power lines section is a is a pretty important problem that you want to solve.
Here we can leverage simulations plus machine learning again it's doing reinforcement learning when it does the right thing where it follows the power line nicely it gets high reward whenever it drifts away or banks into some of the things you've got a very negative reward.
Keep running the single civilization and cloud from millions and millions of creation and suddenly you have a model that works in the real world.
The great thing with these similar solution scenarios are that you can use third party content right you don't have to create all those 3 D virtual worlds orione what I'm showing you is 100 square miles of New Zealand that was created by epic games.
And what we are doing is we have instant sheated are drones are robots with all the sensors and we have flying through that we can use this third party environment to actually learn you can do you know interesting amount of things and great thing about you know some of these technologies that.
You know you can you can have detailed.
So world at at level that wasn't really possible for.
So when we think about simulation centric for robotics let me just show you the statistics why does this matter and why do I believe this is the future.
So I showed you the first slide atari games and Alpha go you know the number of data points that you are pushing through any of these machine learning deep machine learning algorithm is immense for instance watari games you are looking at pushing through.
100000000 frames and similar out of frames for Alpha go as well pacman at 30 frames per second 200000000 frames is 77 days worth of compute I mean what I'm going to do about it I mean are you going to.
Do that much especially with the real world are you going to just drive your car for 77 days and to get all that data possibly not you need simulation and you would simulation in parallel massive parallel simulations?
That would enable you to solve these tasks.
Right.
So I will conclude with one.
You know one real-world study that we are working on So what you're seeing here is.
Is a is a collaboration with the University of Southern California where we are trying to identify approaching activity happening in a African Savannah so this is an African Savannah this is real world footage?
Which is captured with the camera on a drone and what you are seeing is these white pixels so these are animals and humans and poachers and the whole goal is to build a detection system because you are flying these drones at night and you want to detect any kind of human moment because that might be?
You know correlating with the with the poaching activity you have to do this wire machine learning methods but where are you going to get this kind of data are you going to park yourself in Africa and so on or 4 months and then try to capture this data and not.
So what we're going to do is we're going to create simulation so here the entire African Savannah is being created a simulation with animated animals and poachers and we're simulating our cameras and now you can do is you can now generate.
Artificial data that then you can combine with real world footage that you have you have captured and then build these detection recognition models at the scale that was not possible for.
If you have a technical report coming out on this which is where you know we basically are showing that augmenting simulation with real world data boosts your recognition performance is very significantly?
Right so when we think about hyfryd simulation it's a descriptive way to represent reality as I said it's video game on steroids basically except the video game is being played by.
Software and by playing this game on and on for millions of times you can expect the system to learn the fantastic things that weren't possible before and the whole the whole vision is that.
You know once you're happy with your model you can do all kinds of validation testing and then you just flip a bit and then I'll code based on goes on the real world and the real world robot then takes care of things.
As I said ehrsam is an open source project they are several thousand users of it it's an active community of researchers and developers you are Welcome to go all of the stuff that I talked about is publicly available and then you can read about.
About it here is the URL and with that I'll conclude I'm happy to take any questions and happy to discuss more thank you.