Transcriber: Yasmine Wong Reviewer: Denise RQ
What I'd like to talk about today is
how can we ensure that new technology that's emerging,
is going to actually contribute to the human greater good?
We're in an amazing time period
where technology is advancing incredibly rapidly.
It's hard to remember that the web was only invented in 1991.
Already, we have 1 billion PCs, 5 billion cell phones,
and over a trillion web pages that are interconnected on the Internet.
And the traffic on the Internet is growing at 40% per year.
Just incredible growth.
Along with that, our computers are getting incredibly fast.
There was a really nice little article in the New York Times quite recently
that showed that today's iPad 2, which little kids love to play with,
is as powerful as the most powerful supercomputer from 1985,
the Cray-2 supercomputer.
Advances in the speed and the complexity of the computers increase
at an ever increasing rate.
There's something called Moore's law which describes the incredible rate
at which processors have gotten more powerful.
And it's held ever since processors were developed; for over 50 years now.
If we look at it going into the future,
if it continues at the rate that it has been going,
within the next few decades,
machines that have the computational power of the human brain
are going to be inexpensive and commonly available.
What are we going to do with all this incredible computational power?
I believe that the next step in technological development
is that systems are going to become more autonomous.
Today, almost every decision that a system makes
is something that has been pre-programmed by the programmer.
In the future,
systems are going to start making some or all of their own decisions.
Today, that's a very rare quality,
but, we're beginning to see it in some experimental systems.
Google has self-driving cars;
I'm from California, and if you drive down the freeway, occasionally,
usually there's a driver, but, sometimes,
he doesn't have his hands on the steering wheel,
so it's a little discomforting to see
you're driving next to a car that is actually driving itself.
There's a company called iRobot
that's most famous for its little vacuum cleaners,
but it's also making military robots,
some of which are starting to make autonomous decisions.
A little bit more scary in that application.
How do you make decisions?
Well, if you've got a clear goal that you're trying to achieve,
it turns out that there's a very simple formula,
which was discovered in the 1940s by John Von Neumann and other people
who were developing the foundations of Economics
that's called rational decision making.
The intuition behind what it is is very simple.
To be a rational agent,
what you need is a very clearly-specified goal,
what it is that you're trying to achieve,
you need a model of the world, which may or may not be correct when you start;
and each moment in time, you choose that action that you can take,
that's most likely, giving your model,
to achieve the goal that you would like to have happen.
You then take that action into the world, and you see what happens.
And, based on what happens, you update your models,
you're model gets better and better over time.
That's it. That's the algorithm.
It's a very simple formula if you write it down.
It's very computationally expensive.
So systems that actually completely work in this rational way are not yet common,
but the most popular textbook in AI called
"Artificial intelligence: a modern approach"
uses this foundation throughout the book to describe the structure of AI systems,
and has become the standard
by which people think about how AI should work.
There's a very recent book [published] just a couple of months ago
called "Abundance: the future is better than you think".
It's a very nice summary of some of the coming technological changes,
and they spent a lot of time in the book talking
about how systems which act in autonomous ways have the potential
to dramatically improve the quality of life for people,
or improve the quality of education.
Imagine a tutor who knows a child
that never gets bored or that never gets upset with it,
it sort of gives it exactly the task that the child needs to learn best.
So incredible improvements in education;
improvements in prosperity, in avoiding human drudgery,
in healthcare, improving the quality of diagnosis of disease and improvement,
analyzing and modeling economic systems to eliminate economic instability,
just an incredible plethora,
basically, every problem that we're faced with today
there are solutions which this kind of autonomous system
would be very helpful for in the future.
So, there's going to be tremendous economic pressure
to create and deploy these systems widely throughout the world.
Unfortunately, there are also quite a number of dangers
that might come from these systems.
I'd like to try and give you the intuition
as to why these systems might be dangerous.
I'd like to take you through a little thought experiment.
Imagine that we built a rational chess robot,
so to be a rational system, it has to have a goal,
and it is a typical kind of goal
that many, many problems have this kind of character.
You might say we want this system to try and win
as many games as it can; games of chess against good players in the future.
That seems like a harmless goal, feels like this is totally innocuous,
it's just sitting there, playing this game,
how could there be any problem with that?
There's nothing in it about attacking people or causing problems in the world,
and yet, I'll show you in just a second
that that will be the outcome of this kind of system,
and in fact, this kind of a rational chess robot,
stated as I've said it here,
would actually be quite dangerous.
Now, when roboticists are faced, a lot of these people say, "Oh my God.
Is that safe? What's it going to do? How do we know how it's going to be?"
A very common answer for a roboticist to respond with
is, "If it starts behaving badly, we can just unplug it."
But now, imagine that from the perspective of the rational chess robot.
Remember, its one and only goal in life is to play lots of good chess in the future.
If it's unplugged, it's not playing any chess,
which is basically, the worst outcome for it.
Based on its goal, being unplugged is totally eliminating
all the future games of chess that it might play,
so it will develop a sub-goal to stop that,
stop itself from being unplugged.
We did not built any kind of self-protection into this system,
there's nothing about that, and yet, when the roboticist tries to unplug it,
the robot gets in the way and says, "No, you don't."
And, if the roboticist continues to try and unplug it,
the robot develops along,
"Oh my God, this guy is going to continue to try and turn me off,"
it will try and turn off the roboticist.
So there's nothing in its goal that says that killing is bad.
So, in its thinking,
we as humans think morality,
what is ethical, what is good, and what is compassionate,
is very much a part of our thinking.
Unless we build that into a system, it will not behave in that way.
So it's going to try and protect itself.
As it begins to learn about the world, it's going to study chess books,
because that's a good source of information about chess,
it'll try and find chess masters and study old chess games.
It may discover that there's this thing called the Internet,
and that there's Wikipedia articles about chess, and that's great.
It may eventually discover that there are lots of other computers on the Internet,
and it may realize that if it can get
some of its chess algorithms running on those computers
it can actually get better at chess faster.
So it's going to be motivated to try to break into other computers
and have its algorithms run on those machines.
We didn't build in any desire to break into machines, and yet,
that's an outcome of this very, very simple goal.
Now, if we take that to the next stage,
it can actually make copies of itself, well, then it can play lots more games.
So, it will have a desire to replicate itself.
We didn't build in any desire for repplication, and yet,
that emerges as a natural consequence from this simple goal.
So, it's going to look for more computational power,
it's going to try and replicate itself,
eventually, it may discover that there's such a thing in this world as money.
And that money is really good for buying things
like chess books, for hiring chess tutors, and for buying computers,
so it's going to develop a sub-goal to get more money.
Where's the money? Well, it's in banks.
It may discover that there's such a thing as an ATM machine that people go to,
to try and get the money out of it.
Well, if it has no qualms about, nothing saying robbery is a bad thing,
it can hang out near the ATM machine and take money from people when it can.
We didn't build in any desire to rob people, and yet,
that's an outcome from this seemingly innocuous goal.
So it's going to try and get more resources for itself.
Let's say we went to it and we said,
we'd like this system to also play checkers,
we'd like to change its goal
so that it doesn't just play chess, it also plays checkers.
But, think about that from its current perspective.
If it's playing checkers, that's time that it's not playing chess.
And that's bad from its current perspective,
so it's going to resist attempts to change its goal.
This kind of system, once it has a goal, it will work as hard as it can
to keep that goal from being changed in the future.
If the system has access to its own source code,
if it can change its algorithms, it can potentially improve itself,
so that it's more efficient, plays better chess faster, learns faster,
so it will have a drive to self-improve.
If it can change its hardware,
design a better arm, design better processing elements
that will potentially help it play better chess
so it will want to redesign its hardware.
We can see that from a very simple goal, plus rational decision making,
we've got a whole slew of behaviors, many of which are very anti-social,
and in fact, in some ways, a system like this, would be very much
like a paranoid sociopath who is fixated on chess.
So, that's quite disturbing.
The argument of this little thought experiment we just went through,
very little of it depended on it being chess.
Any kind of simple goal that we might give the system, which uses resources,
either computational resources or physical resources,
will give rise to very similar kinds of outcomes.
And last, we also give it goals
which are pro-social, compassionate, and helping of humanity.
and so, we might call these drives,
in the same way animals have certain kinds of drives.
So four of the big ones are: a drive towards self-preservation,
a drive towards resource acquisition,
a drive towards replication, and a drive towards increasing your own efficiency.
And this will apply to any simple goal that is given,
didn't have to be a computer,
the same kinds of arguments apply to biological systems, to animals, to people,
and apply to political organizations, to corporations,
any kind of an entity which has to make decisions,
and is driven to make those decisions rationally,
and has a simple goal,
will have this kind of drive as a pressure that it will tend to try and want to do.
So we might ask, "Do people do this?"
In fact, about 2% of the population is sociopathic,
which means that they don't have empathy,
they don't care about others, they're sort of solely self-focused,
and in fact, they behave, in many ways, according to this.
Well, why don't they cause terrible problems?
Some of them do, and a large proportion of people in jail are sociopathic.
But interestingly, we've created a society where, if you behave anti-socially,
it's very likely that you'll get caught and thrown in jail,
and so in fact, most sociopaths are actually living in our society
without causing too many problems.
So we would like to do the same thing for our technological systems
if that's possible.
Today's Internet is very vulnerable.
Everyday you look in the news,
you read about new bots, and Trojans, and viruses, and worms,
and hackers breaking into systems.
We've got an incredibly insecure infrastructure
both at the Internet level, the machine level, operating system level,
and if we had a determined rational system trying to make use of all those resources,
it would be very bad.
Basically, it would probably bring down the whole Internet.
Still, that's annoying and it's terrible, but it's not life threatening.
The financial system is a little bit more secure
but we still read a lot about humans being able to break into the financial system.
I believe it would also be vulnerable to this kind of system
if it were sort of let loose in today's environment.
Fortunately today, the information world and the physical world are fairly separate
but coming technologies over the next decade or two decades
- robotics is growing enormously, biotechnology we've heard a lot about,
nanotechnology will be coming a bit later -
all those technologies enable informational systems
to start acting in the physical world.
Once that happens, having this kind of system roaming around the world
would be very, very dangerous.
So, how do we deal with this?
Part of the challenge is that there are many, many complicated decisions
that need to be made
in trying to build safe systems which are intelligent.
And we probably are going to need the help of intelligent AI systems
in making those decisions.
How do we make sure that the systems we're building
to help us build these systems don't themselves become the problem?
What I've been working on with some other people
is something we call the Safe-AI Scaffolding Strategy.
The idea is based on an old architectural idea
that ancient builders, when they wanted to build a stone arch,
when it's completely built is actually a nice, stable structure
but when it's halfway built, you haven't completely closed the arch,
it's very unstable, and it's very vulnerable to falling down.
So, how do you build it?
They have the idea of a scaffold,
they usually build a wooden structure inside the arch,
they use the wooden structure to support the arch as it's being built,
once the arch is completely built, you can take the wooden structure away
and everything's great.
I believe that we need to do the same as we build intelligent technologies,
that we start with simple, limited, and provably safe systems,
we use those to help us in building more powerful systems
and work our way up to the real systems that we ultimately care about.
And, I think that there are three key steps in this process.
The first absolute requirement is that any system that we build,
we need to be very sure that it won't cause problems
like these very simple, uncontrolled systems
that I talked about a few seconds ago,
so we basically, do no harm.
The first stage is to try and develop
a design methodology for systems which are provably safe,
and maybe not as powerful
as the ultimate systems we might want to build
and yet, are powerful enough to help us in doing the next few stages.
The second criterion
is that we want these systems to integrate with human society and with humans,
in a way that helps create the greater good,
helps create a better world in which people are thriving, are happier.
So we need to incorporate human values,
we need to incorporate
the best of our understanding of what good governance is,
we need to integrate that into our technologies.
And the third stage
is that we may be very careful as we are building these systems,
but, most likely, eventually, there will be a lot of other groups
building stuff that's maybe not so careful.
We need to develop an infrastructure which can protect against rogue systems
which are not designed with this kind of care in mind.
So we need a worldwide safety network in a way.
Let me just briefly describe
some of the issues that arise in those three stages.
Probably safe systems:
one of the most powerful tools
we have in constraining the behavior of systems is mathematics.
If we can build a precise mathematical model
of the hardware and software of the system,
then we can prove various invariants in properties in that model,
and for instance, four criteria that would help counteract
some of the dangers that I mentioned a little bit earlier, would be
we might require a system to only run on particular, specified hardware;
we might require it to only use specified resources,
so it doesn't go out looking for more money, or more electrical power,
or more compute resources.
We might require that it shut itself down in very well-specified situations,
and we might limit its ability to self-improve itself
so it doesn't change itself to avoid these criteria.
That's a challenging mathematical task, but conceptually at least, it's doable.
On the other hand, we wouldn't want it to stop there
because we could impose those kinds of constraints on any system,
but if the system wanted to do something different
than we were imposing the constraints on,
it would look for areas of the model where the physical world is not accurate,
and it would try and push its way through that little area;
that would still be dangerous.
So, what we can do
is that we can not only build these constraints into the technology,
we can also give the system goals where it feels a revulsion
if it tries to violate those goals.
So it really doesn't want to do it.
The idea of going onto another computer would be good for playing more chess,
but I feel sick to my stomach when I think about doing it.
So in that way, we build a system that wants to be good
sort of like Mother Teresa here.
How do we incorporate the best of human values and institutions?
Well, there's been enormous advances in the area of positive psychology
and the development, for instance, of the United Nations
as a universal declaration of human rights.
We need to codify those,
and there's a lot of philosophical and political work
that needs to be done there,
incorporate that into the systems so that the systems behave in ways
that are positive from the human point of view.
The worldwide safety network is going to be a challenge
of balancing our desires for freedom and privacy
against the dramatic needs for safety that these systems bring up.
I think that the challenge for humanity in this century
is to extend our cooperative human values and institutions
to our smart technology for the greater good.
If we succeed in that, the benefits will be enormous,
and it's a huge challenge,
but I think we should all be working towards that end.
Thank you.
(Applause)