So, before, we were talking about A.I. risk and A.I. safety, and
just trying to lay out in a very generalized sort of way how general artificial intelligence can be dangerous
and some of the type of problems it could cause
and just introducing the idea of A.I. safety or A.I. alignment theory
as an area of research in computer science.
And we also talked about super intelligence and
the kind of problems that, the unique problems that can pose
and I thought what would be good is to bring it down
to a more concrete example of
current A.I. safety research that's going on now
and kind of give a feel for where we are,
where humanity is on figuring these problems out.
Supposing that we do develop a general intelligence
you know an algorithm that
actually implements general intelligence.
How do we safely work on that thing and improve it?
Because, the situation with this stamp collector is
from its first instant it's a super
intelligence so we created with a
certain goal and as I said as soon as we
switch it on
it's extremely dangerous, which people
pointed out and, it's true, you know it was a
thought experiment, it's true that that's
probably not what will happen right?
You'll have some significantly weaker
intelligence first that may work on
improving itself or we may improve it.
So, the situation where you just create
the thing and then it goes off and does
its own thing either perfectly or
terribly from the beginning is
unlikely it's more likely that the thing
will be under development
so then the question is how do you make
a system which you can teach? How do you
create a system which
is a general intelligence that wants
things in the real world and is trying
to act in the real world but is also
amenable to being corrected if you
create it with the wrong function with
one utility function and you realize
that it's doing something that actually
you don't want to do how do you make it
so that it will allow you to fix it
how do you make an AI which understand
that it's unfinished they understand
that the utility functions working with
may not be the actual utility function
it should be working with right the
utility function is what the way I cares
about so the the stamp collecting device
if utility function was just how many
stamps in a year
this is kind of like its measure is it
yeah it's what it is the thing that it's
trying to optimize in the world the
utility function takes in World states
as an argument and spits out a number is
broken the idea if the world was like
this is that good or bad
Andy the AI is trying to steer towards
world states that value value highly
black utility function
you don't have to explicitly build the
AI in that way but it will always if
it's behaving coherently it will always
behave as though it's in accordance with
some utility function also before I
talked about about converging
instrumental goals that if you have some
final goal like you know it makes them
the very instrumental goals which are
the goals that you that you do on the
way to your final goal right so like
acquire the capacity to do printing it's
like perhaps an instrumental goal
towards making steps but the thing is
there are certain goals which tend to
pop out even across a wide variety of
different possible terminal goals so for
humans an example of
convergys instrumental goal would be
money if you want to make a lot of
stamps or you want to cure cancer or you
want to establish a moon colony whatever
it is having money is good idea right so
even if you don't know what somebody
wants you can reasonably predict that
they're gonna value getting money
because money is so broadly useful and
before we talked about this
we talked about improving your own
intelligence as a convergence
instrumental doll that's another one of
those things where it doesn't really
matter what you're trying to achieve
you're probably better at achieving if
you're smarter so that's something you
can expect a is to go for even if even
without making any assumptions about
that final goal so another convergent
instrumental goal is preventing yourself
from being destroyed it doesn't matter
what you want to do you probably can't
do it if you're destroyed so it doesn't
matter what the AI want you can let it
wants to be destroyed in from my trivial
case but if you do i want something in
the real world and believe that it's in
a position to get that thing you wanted
to be alive not because it wants to be
alive
fundamentally it's not a survival
instinct or an urge to live or anything
like that it's smooth and knowing that
it's unit available to completed its
cutie would it be almost scares unable
to achieve its goals if it's destroyed
and wants to achieve that goal so that's
an instrumental value is preventing
turned off and i'm getting here we say
want to it's not like a machine want
it's just a turn of phrase
yeah i mean as much as anything it's a
it's closer it actually you know i'm not
even sure i would agree like if you talk
about most machines to talk about that
they want to whatever and it's not that
meaningful because they're not agents in
our general intelligence is where the
general intelligence when it wants
something it once in a similar way to
the way that people want things so it's
such a tight analogy that it wouldn't
even I think it's totally reasonable to
say that energy i want something
there's another slightly more subtle
version which is closely related to not
wanting to be turned off or destroyed
which is not wanting to be changed so if
you imagine let's say I mean you have
kids right yeah
suppose I were to offer you a pill or
something you could take this pill will
like completely rewire your brain so
that you would just absolutely love to
like kill Ricketts right where's right
now what you want is like very
complicated and quite difficult to
achieve and it's hard work for you and
you probably never going to be done
you're never gonna be truly happy right
in life nobody is you can't achieve
everything you want i said this case it
just changes what you want what you
wanted to created and if you do that you
will be just perfectly happy and
satisfied with life right
ok you want to take this go know that
you happy though
yeah I don't want to do it because but
that's quite a complicated specific case
because it directly opposes what I
currently want it's about your
fundamental values and go right and so
not only will you not take that pill you
will probably fight pretty hard to avoid
having at the limited to you
yes because it doesn't matter how that
future version of you would feel you
know that right now you love your kids
and your not going to take any action
right now which leads to them coming to
heart
so it's the same thing if you have an AI
this for example value stamps values
collecting stamps and you go oh wait
hang on a second
I didn't quite do that right let me just
go in and change this so that you don't
like stand quite so much it's going to
say but the only important thing is
stamped if you change me i'm not going
to collect as many stamps which is
something i don't want there's a general
tendency for AGI to try and prevent you
from modifying it once it's running
I i can understand that now in in the
complex reporter right
because thats that's it it in almost any
situation being given a new utility
function is going to write very low on
your current utility function
ok so that's a problem
how do you want if you want to build
something that you can teach that means
you want to be able to change its
utility function and you don't want to
fight you
on is 100 yeah fuck
so this has been formalized as this
property that we want early AGI to have
called courage ability that is to say it
open to be corrected