in a very basic sense if you've got a
general intelligence which therefore has preferences over world states
and take actions in the world to change the world
we have to make sure that its preferences are aligned with ours
that it wants what we want.
because otherwise it's gonna try and do things that we don't want it to do.
that's the basic idea...
(sci-fi comes to the fore we're talking things like terminator, matrix,
that machines take over the world
I for one salute our new robot overlords)
it makes it difficult to think about the problem right?
when things happen in fiction it's generally what would make a better story
rather than what would actually happen and I think a
a realistic "AI takes over the world"-type story
might not be any fun to read. So on the one side
this stuff is difficult to think about because of fiction, because we've all been
exposed to something similar to these ideas before
the other side that makes this difficult to think about is anthropomorphism
because we are talking about general intelligence
so we're going to compare it to the examples in general intelligence that we
have
which is human minds and human minds and artificial general intelligences
need not be anything alike
in the same way that a plane is not similar to a bird
a supersonic fighter jet is a threat to you in a way that
no bird is. It's not a useful comparison to make in fact
but when you say oh it's a thing it has wings and it flies and people
who don't know anything about planes immediately to go to birds
(presumably machine could be much more selfish than we can ever imagine)
absolutely, the space of minds in general is vast
I like this because we've already talked about spaces so I can do this. If you
take the space at all possible minds
it's huge and then it somewhere within that, you have the space of all minds
that biological evolution can produce
and that's also huge somewhere within that you have the space of
actual minds that exist which is much smaller but still huge
within that, you've got human minds right and they're a tiny, they're a
minuscule dot on a minuscule dot on a minuscule dot of the
actual possibilities for intelligence that exist and
a general intelligence that we create is from a completely different part of the
space
and it's extremely tempting to anthropomorphise
more so even than in another context because
it's a thing that's demonstrably intelligent that makes plans that takes
actions in the real world
but it need not think
anything like us and it's a mistake to think a bit
as basically a person because it isn't one. So there's actually really good example
that we can use. It's sort of a thought experiment
This is not a machine that could practically speaking be built
this is an example of at artificial general intelligence
which is specified an overview
and it gives you something to think about when you're thinking about
artificial general intelligences that makes it distinct from
a sort of anthropomorphized human type intelligence
So the story is there's a
a stamp collector who is also an AI
programmer and he decides he would like to collect a lot more stamps
so he's gonna write an AI to do this for him. So he builds a machine
he has some startling insight into general intelligence and he builds this
machine
which is connected to the Internet, right?
so the the rules for this system are
pretty straightforward. first thing, it's connected to the Internet and it will
send and receive data
for one year. So, he's given himself a one-year time
window within which to collect stamps the second thing is
it has an internal model of reality , of the universe
this is the thing that's a bit magic we don't really know how to build an
accurate
model of reality. The point is this allows it to
make accurate predictions about what will happen if it does different things
the third thing is for every possible sequence of packets it could send
it uses its model to predict
how many stamps it ends up with at the end of that and then
the fourth thing is it outputs as its
actual data to the Internet
whichever output it has predicted will produce the most stamps
you can see here
that this has all the properties of a general intelligence
it has an internal model of reality. It has
a utility function or an evaluation function, which is the number of stamps,
and the optimization is
extremely simple and like so much in computer science there
the simple things to specify are the hard things to compute
it looks at every possible output data in other words every
every point in that space and it picks out
the highest one. So this is a kind of magic intelligence
that takes the entire space at once finds the highest point and says
that one. Which means it's an extremely powerful intelligence
right it is you could say it's extremely intelligent
the question is how does this machine behave? Well, we can look at certain
possible sequences of outputs
and see how they fare in it's evaluation criteria.
first off, the vast majority of output sequences are complete junk
right it's spewing random data on to the network. nothing happens
of any consequence.
No stamps get collected. Those are all rated zero.
but suppose one of the possible sequences
a sends a request to a server let's say
eBay, that results in a bid on some stamps,
right? When that happens, there's a thing of 20 stamps
so that output is rated 20. This is the kind of thing that
the stamp collector machine's creator was expecting to happen
So then that's good, 20 stamps
suppose it could do that lots of times. it could send out bids for example to
30 different stamp collectors on eBay and buy 30 sets of different stamps
and that's even better, right? That would be rated even higher, but the thing is that
particularly highly rated options in this search space
are probably things that the
stamp collecting device's creator did not think of
and did not anticipate. So for example when he made it, he will have presumably given
it his credit card details or something so they could engage in these bids
but ultimately it's searching every possible sequence of outputs. It needn't use
his credit card
it needn't use money at all. There's a huge variety of things that it could do here.
So it might send out an enormous number of emails to all the stamp collectors in the
world
and convince them through persuasive argument that
he is opening a museum and wants to exhibit their stamps
It may build a fake website for that whatever is necessary
it can predict the world, right. It has an internal model of reality that internal
model of reality
includes the people right. In the same way that its modeling
people to understand that if it bids on this
offer, then a human being will mail the stamps to them. They understand this
email might get more people to send stamps for example that something
but then what exactly is a stamp? How is this defined?
What counts as a stamp?
if it's already written the sequence of outputs that collects
all of the stamps in the world you think you're done, right? You built this machine
suddenly is collected all the stamps in the world.
No, there could be more stamps within a year. We've got time to print more stamps
so maybe it hijacks the world's stamp printing factories and puts them into overdrive
producing stamps
as many as you possibly can in that time or perhaps
it writes a virus and it hijacks all the computers in the world to get all of
the printers in the world to do nothing but print stamps.
that's even better right? The highest-rated
outcomes for this machine are not good for people
There comes a point when the stamp collecting devices thinking
okay what are stamps made of?
they're made of paper. Paper is made of carbon, hydrogen, oxygen.
I'm gonna need all of that I can get to make stamps
and its gonna notice that people are made of carbon, hydrogen, and oxygen
right. There comes a point where the stamp collecting device becomes
extremely dangerous
and that point is as soon as you switch it on.
So this is just a thought experiment example where we take a sort of maximally powerful intelligence
and see how it behaves so that we can think about
how powerful intelligence has behaved and most importantly
how they are not like people