Recently, Nvidia announced a synthetic data engine for training artificial intelligence.
Synthetic, meaning that researchers can use it to generate fake images
of the real world for training self-driving AIs.
The concept of leveraging modern open-world video games and their engines to create huge
amounts of fake data for your AI might seem like a disaster in the making.
But recent events seem to imply the otherwise. Using computers to
generate computer data so that we can train other computers is a trend that is working.
In this video, we are going to talk about training
autonomous driving AIs with data from a synthetic world.
## Deep Learning
Deep learning models are changing the world around us. But in order to work correctly,
a model has to be trained with data.
A basic pipeline for applying machine learning to a problem goes like this:
One, you collect raw data related to the specific problem;
Two, you label the data, according the problem parameters;
For instance, semantic labeling of pixels. This is where you assign each pixel a value
so that the AI can recognize what exactly is in the image at a pixel level view.
Three, you train the machine learning model with the labeled data sets.
You validate that model's results using subsets of the data specifically set aside for test runs.
This is to avoid a phenomena called "over-fitting" - a failure mode when
a model can't generalize or fit well to a brand new data set.
Then finally, you deploy the model in the real world. At the heart of this entire process
is data. Frequently, this data comes from real life. And you need a whole lot of it.
## The Data Problem
But many people are uncomfortable with the idea of their data being collected,
shared, and used - especially without permission.
More so when it comes to AI applications in certain sensitive areas.
For instance, companies are training machine learning models to identify cases of skin cancer.
But that means feeding real photos of patients' skin markings into the model.
Another example has to do with the finance world. Financial services companies want
better working models. But financial data - loan applications, customer service logs,
transaction data - are some of the most sensitive there is.
Thus, researchers are increasingly generating and using synthetic data.
The practice started as simple augmentation,
making slight edits or recombinations to real data to improve performance.
But it quickly evolved into a way to get the data they might possibly need
without spending an immense amount of money or violating someone's privacy.
Probably the most influential example is that of AlphaZero.
Its predecessor AlphaGo was largely trained on actual Go games played by professionals.
But AlphaZero took it another step by generating its own data using self-play. The model simulated
games between itself for data to reinforce and improve its own future performance.
In another example, researchers used Generative Adversarial Networks - the same type used to make
your pictures look like a Van Gogh - to generate realistic photos of cancerous skin lesions.
So all in all, it might seem a little bizarre at first glance but training machine learning
models with synthetic data has become a generally accepted practice. Thus naturally,
researchers have looked to apply it to one of the most challenging tasks in the industry:
Self-driving cars.
## Self-Driving AI
Self driving cars have world-changing potential. Most car accidents are caused by human error.
Replacing every sub-average human driver - like myself - with a highly automated AI can
save countless lives with additional environmental and mobility benefits.
People have been dreaming about robotic driving since the 1920s.
But one of the first real stabs at autonomous driving using neural networks
was ALVINN, which stands for "Autonomous Land Vehicle In a Neural Network".
First introduced all the way back in 1989 at Carnegie Mellon University,
the little car drove a retrofitted Army ambulance around the campus.
ALVINN's network took in two types of data: A 30x32-sized "video" of the road,
and 8x32-sized range finder. To properly train the system, the ALVINN team needed road images
under various conditions. Due to the challenges of finding enough consistent real-life road images,
the team generated them artificially.
Side note. ALVINN's network had just 3 layers. By comparison, GPT-3 is said to have 96.
Amazing how far things have progressed.
## In the Real World
Since the days of ALVINN, the technology has vastly improved.
Automated driving systems are today capable of amazing feats. For instance,
a KUKA robot can autonomously carry a payload to a certain position with millimeter accuracy.
But these autonomous robot navigators do their navigating
in gated neighborhoods with little else going on.
Drivers drive out in the real world. One quarter mile at a time.
A self-driving AI needs to have the computer vision to see and recognize everything an
alert human driver is capable of. But this requires having access to a lot of labeled data.
Previously, that means a real person has to look at scenes of actual sites
and trace it out for the AI to consume.
There are a few public datasets that actually did this. For instance,
Cityscapes - which provides up to 5,000 annotated images of real streets in Germany.
But 5,000 in the context of machine learning is not that much.
And getting significantly more would be prohibitively expensive.
Thus researchers have turned to video games and game engines as a creative alternative.
## Training with GTA V
The video game industry spends hundreds of millions of dollars to create games with
more photorealistic scenes and situations that closely hew to real life physics. Researchers can
leverage their work to rapidly and affordably create labeled relevant imagery for training.
Grand Theft Auto V is a game released in 2013 and is the second best selling video game of all time.
The game is known for its highly realistic environment and driving scenarios.
Shortly after its release,
researchers started hacking the game to generate synthetic data for self-driving AIs.
They used two open source plugins to capture screenshots of a particular scene in the game.
Just as importantly, the plugins allow them to capture the auxiliary information about the scene.
For instance, the pixel bounds of an object's position in the
scene. Like a car or a stop sign or something.
With this, researchers have been able to generate tens of thousands of images of realistic road
scenes under a variety of conditions - rain, fog, overcast, clear. This variety helps the
driving AI respond correctly if it were to encounter such situations in real life.
The initial paper notes that their GTA trained models actually perform
better than ones trained with real world images, because there is so much more data.
## Beyond GTA
It is cool that researchers are using a video game engine to train their self-driving AI.
But the reality is that GTA V was meant to be a game first,
leading to some crucial differences and inconveniences.
First, a video game has to run on a relatively weak computer,
which limits its capabilities. GTA V first ran on the Playstation 3 and the Xbox 360.
By now, this hardware is over a decade and a half old. We can do better than that.
Second, you need to hack the game using scripts to collect data from it.
These scripts aren’t supported by the original developers and can occasionally cause problems.
Third, GTA V's Los Santos is a twin of Los Angeles,
which is an urban environment. Suburban and rural scenes are not as well represented in the game.
And last, a video game is meant to be fun more than it is meant to be physically accurate.
And making something fun sometimes means having to occasionally break the rules of physics.
So researchers have sought virtual environments from which they can have photorealistic images.
But also have complete control over the objects, extensibility, and greater ease of use.
That is the idea behind Nvidia's Omniverse Replicator.
## Omniverse Replicator
Nvidia released this tool in November 2021. It is part of their Omniverse platform initiative,
which the company describes as an open platform for photorealistic simulation.
There are two implementations - Drive Sim, which is for training outdoors driving agents;
and Isaac Sim, which is for robots. The goal is to allow developers to create
synthetic data that otherwise would be too hard to collect in the real world.
It is built on top of a software platform first developed by Pixar:
Universal Scene Description or USD. This is a common language used for rendering 3D scenes,
kind of like how HTML is a common language for rendering web pages.
In some ways Omniverse Replicator is like a video game. It has a physics engine with accurate
lighting effects and more. There is a world in which you can see and interact with things.
Yet at the same time, it is not a video game. It
is a real-time simulator meant to be physically accurate to the real world.
## Training a Robot
Nvidia published a case study of a situation faced by Boston Dynamics,
which faced onboarding challenges when deploying new industrial robots. These
robots' AI have to be tuned to deal with these unfamiliar indoor environments.
One way to do this would be to send a team of photographers into the deployment area to
take pictures of the whole environment. But that is expensive, potentially dangerous,
and requires a lot of pre-planning. What if you were to generate the photographs with code?
The team imported into Omniverse Isaac Sim a bunch of 3D CAD models of various indoors environments
like an office building or a factory. From then they can generate segmented training
images complete with boxes, depth and more.
A key point is the ability to change the conditions of the indoor environment.
An AI has to be react to a wide variety of different visual parameters - scale,
perspective, color, lighting, and so on. All of this was possible with this custom tool.
USD is pretty powerful in this aspect. It lets you customize the various attributes of the assets
in the scene. You can transform the data, randomize it, and so on.
Such complicated tools can no longer be built by individual or even a small team
of developers. So Nvidia releasing something like this for their use is a big help.
So long as those guys keep dropping thousands of dollars on big honkin' A100 GPUs, that is.
## Simulators
Using photorealistic engines to generate training images is cool,
but self-driving AIs have to do more than just recognize the scene around them.
For instance, they have to track and predict where things are so to adjust the driving accordingly.
This means a simulated world and there’s been a few interesting ones out there for such use.
First is the Open Racing Car Simulator or TORCS. This started off as a Linux game in
the late 1990s but evolved into a popular virtual simulated world for driving agents.
TORCS is admirable, but it lacks pedestrians, intersections, and other aspects of urban driving.
And like GTA V, it was originally a video game,
which brings up all the other issues we talked about earlier.
So a simulator that got quite a lot of attention during its 2017 release
was CAR Learning to Act or CARLA. This is a free and open simulator for urban driving,
built on top of Unreal Engine 4 - a video game engine developed by Epic Games.
CARLA is a philosophical descendant of the SYNTHIA dataset, which provided thousands of labeled
synthetic scenes for training self-driving AIs. But SYNTHIA was created with a closed
source engine. CARLA's license on the other hand allows it to be shared with everyone.
CARLA allows people to put their driving AIs into scenarios far too dangerous to recreate
and practice in real life. Like a child running in front of a car, for instance.
Researchers can plug driving AIs - and all their sensors too,
LiDAR, GPS, RGB cameras and so on - right into CARLA so that the AI can
respond to various scenarios like as if it were in the real world.
Outdoor simulated environments like CARLA also have real applications to training
AIs for autonomous aerial vehicles. Unmanned drones can use simulators
to train on search-and-rescue scenarios, and military ones too.
## Conclusion
The entire concept of synthetic data just fascinates me. It is another
example of today's game and visual engines advancing far beyond their original purposes.
To a point where their image output can actually transfer over to the real world.
And perhaps even do more than that. It makes me think.
In an IEEE interview, Nvidia VP of simulation technology Rev Lebaredian says,
"We believe that if you can simulate the real world closely enough, then you gain superpowers."
I wonder where the pathway takes us down the line in the future.
What happens when a world simulator gets powerful to simulate entire alternate futures
and make optimizations to them? What might that be able to do for us?
So cool. Alright, enough science-fiction.