today i thought we would talk a bit
about logical induction uh paper out of
the machine intelligence research
institute very technical paper very
mathematical and can be a bit hard to
get your head around and we're not going
to get too far into it for computer file
i just want to explain like
why it's cool and why it's interesting
and people who are into it can read the
paper themselves and
find out about it
one of the things about logical
induction as a paper is that
it's not immediately obvious why it's an
ai safety paper
um
but like
my
perception of the way that miri the way
that the machine intelligence research
institute is thinking about this is
they're saying
they're thinking like
we're going to be producing at some
point artificial general intelligence
and as we talked about in previous
videos
there are
hundreds of ways really weird subtle
difficult to predict ways that these
this kind of thing could go
uh very badly wrong
and so
we want to be confident
about the behavior of our systems before
we turn them on
and that means ideally we want to be
making systems that we can um
actually
in the best case that we could actually
prove theorems about right things that
are well specified enough that we could
actually write down and formally prove
that it will have certain
characteristics and if you look at like
current machine learning stuff all the
like deep neural networks and
uh that kind of thing they're just
very opaque do you mean by that because
we don't necessarily know exactly why
it's doing what it's doing
yeah and also just that the
the system itself is very
uh is very complex and very contingent
on a lot of specifics about the training
data and the architecture and like
there's just yeah effectively we don't
understand it well enough but it's it's
like not formally specified enough i
guess and so
they're trying to come up with the sort
of mathematical foundations that we
would need to
be able to prove important things about
powerful ai systems before we were
talking about hypothetical future ai
systems we've got time to print more
stamps so maybe it hijacks the world's
stamp printing factories and when we
were talking about the stamp collector
it was really useful to have this
framework of an agent
and say this is what an agent is it's a
formally specified thing
and we can reason about the behavior of
agents in general and then all we need
to do is make these fairly
straightforward assumptions about our ai
system that it will behave like an agent
so we have idealized forms of reasoning
like we have probability theory which
tells you the rules that you need to
follow
to have good beliefs about the state of
the world right and we have like um
like rational choice theory about
uh
what what rules you need to follow and
it may not be
it may not be like actually possible to
follow those rules but we can at least
formally specify like if the thing has
these properties it will do this well
and all we're trying to do is they're
thinking if we're building very powerful
ai systems
the one thing we can expect from them is
they're going to be able to do thinking
well and so if we can come up with some
formalized method of uh of exactly what
we mean by doing thinking well then if
we do
if we reason about that that should give
us
some insight into how these systems will
behave that's the idea so let's talk
about probability theory then this is
where we should have a demo pretty sure
about gamma has nice
yeah
oh this is a fun one
powers of two i don't know how to play
back cameron anyway here's a die
so
basic probability theory right i roll
the die
now it's under the cup
and i can ask you what's the probability
that that is a five one and six right
one and six because you don't know it
could be anything there was a time when
people reasoned about the probabilities
of things happening in a very sort of
fuzzy way you'd be playing cards and
you'd be like oh
you know that card has shown up here so
i guess it's less likely that he has
this hand
and people would
um
intuit yeah and like
there was some sense of like
clearly we are doing so what we're doing
here is not random it's meaningful to
say that this this outcome or that
outcome is more or less likely by a bit
or by a lot or whatever but people
didn't have
a really good explanation of how that
all worked until we had probability
theory right we didn't it wasn't a
system right right
and like practically speaking you often
can't do the full probability theory
stuff in your head for a game of cards
especially when you're trying to read
people's faces and like stuff that's
very hard to quantify but now we have
this understanding that like even when
we can't
actually do it what's going on
underneath is probability theory so
that's one thing right straightforward
probability
now let's do a different thing and now
i'm gonna now i'm gonna need a pen
now suppose i give you a sentence that's
like the tenth digit after the decimal
point of the square root of 17 is a five
what's the probability that that's true
well it's a much more difficult problem
one in ten
right one in ten seems like a totally
reasonable thing to say right it's going
to be one of the digits we don't know
which one
but
if i you've got the paper here you know
you can do like
division you you could do this if you
had long enough like if i gave you say
an hour with the paper and you could sit
here and figure it out
and
let's say you do it and you you come up
and it seems to be like three i don't
know what it actually is i haven't done
this calculation let's do the
calculation bad expression oh i've done
it wrong way round one two three four
five six seven eight nine ten it's a six
ah it's a six okay i picked that out of
nowhere so suppose you didn't have a
calculator you were doing it on paper i
gave you you know half an hour or an
hour to do the like long division of
whatever it is you would have to do to
figure this out what's the probability
this statement is true now
what do you say i'm gonna say zero right
but
you've just done this in a big hurry on
paper
right you might have screwed up
so what's the probability that you made
a mistake you forgot to like carry the
one or something so it could be one so
it's not zero yeah there's like some
smaller probability
um
and then if i left you in in the room
again still with a piece of paper for
100 years a thousand years infinite time
eventually
assuming that was correct eventually you
say zero right
um
you come up with some like formal proof
that this is this is false
um
whereas you imagine if i leave you for
infinite time with the cup
and you can't look at you can't look
under the cup
it's one-sixth and it's going to stay
one-sixth forever right when you're
playing cards you you've got these
probabilities for which cards you think
depending on the game
and as you observe new things
you're updating your probabilities for
these different things
uh as you observe new evidence and then
maybe eventually you'll actually see the
thing and then you can go to 100 or zero
whereas in this case
all that's changing is your thinking but
you're doing a similar sort of thing you
are still updating your probabilities
for things
but probability theory kind of doesn't
have anything to say about this scenario
like
probability is how you update when you
see new evidence and here you're not
seeing any new evidence
so in principle whatever the probability
of this is it's one or at zero
just as a direct logical consequence of
things that you already know
right so your uncertainty
comes from the fact that you are not
logically omniscient in order for you to
figure things out it takes you time
and this turns out to be really
important
um because so most of the time
what you have is actually uh kind of a
mixture of these types of uncertainty
right so let's imagine a third scenario
right suppose you're like an ai system
and that's your eyes because you have a
camera
i do the same thing again
and now
i ask you what's the probability that
it's a five
you would say one sick because you don't
know but on the other hand
you have
recorded video footage of it going under
the thing the video is your memory of
what happened right so you're not
observing new information you're still
observing the same thing you observed
the first time
but you can look at that and say
hmm you know it looked like with the
amount of energy it had and the way that
it was rotating and which numbers were
where it looked like it probably wasn't
going to land on a 5. if i asked you on
a millisecond deadline you know
what what was it what's the probability
that it was a five you're going to give
the same number that you gave at the
beginning here right one and six
but
you can
uh look at the data you already have the
information the observations that you've
already made and you can do
logical reasoning about them you can
think okay based on the speed it was
going the angle it was turned out and
which which pieces uh which faces were
where and so on
it seems like you know maybe you can run
some simulations internally something
like that
say it seems like actually
less likely than
one in six right if before i thought it
was like 0.16 recurring now i think it's
like 0.15 because
all of i ran a million simulations and
it seems like that the longer you think
about it
the
more precise you can be maybe you keep
thinking about it again and you get it
down to like you think it's actually
0.13
right something like that
but you don't you can't
you don't actually know
right because there's still things you
don't know like
you don't know exactly the physical
properties of the paper or what exactly
the inside of the cup is like or the
waiting of the diet like you still have
uncertainty left because you haven't
seen which way it landed but
by doing some thinking you're able to
take some of your
uh you're able to like reduce your
logical uncertainty the point is that
probability theory
uh in order to to do probability theory
properly to modify your beliefs
according to the evidence you observe in
a way that satisfies the the laws of
probability you have to be logically
omniscient you have to
be able to immediately see
all of the logical consequences of
everything you observe and propagate
them throughout your beliefs
and this is like not really practical in
the real world like
in the real world observations do not
form an orderly queue and come at you
one at a time and give you enough time
after each one to fully integrate the
consequences of each observation
um because human beings are bounded
right we
have limited
processing ability and limited speed
with this kind of logical uncertainty
it feels very intuitive
that we can do probabilities based on
our logical uncertainty that we can we
can think about it in this way it makes
perfect sense to say that one in ten is
the probability until you've done your
thinking one in ten seems like a
perfectly reasonable number but like
why is one in 10 a good answer and like
50 is not a good answer because you
might look at it and say well this is a
logical statement
um
it's either true or false that's two
possibilities you know 50
why why is one in 10 more sensible and
in fact you had to do a bit of thinking
to get there right you had to say oh
yeah it's going to be some digit there
are 10 digits i don't have any reason to
prefer one digit over the other so 10
so you are like
always
doing reasoning to come up with these
probabilities but the thing is that
according to standard probability theory
this is like kind of a nonsense question
because if you imagine like from the
perspective of probability theory this
kind of
statement is equivalent to saying what's
the probability that one equals one
right or what's the probability that
like true equals false they're all just
it's just a logical statement
um
it doesn't have a probability it just is
true
um but if you
can't think infinitely fast then you
need to
have
answers you need to have estimates it
estimates you need to have estimates of
your answers
before you've thought for infinite time
about them this is like an important
thing for actually getting by in the
world because as i say like observations
don't just line up and wait for you to
reason all of their consequences through
so when it comes to empirical
uncertainty we have probability theory
which has these axioms
um and these are sort of the rules that
you need to follow in order to
do probability well
and they're sort of things like that
probabilities
can't be negative they have to be
between zero and one um
the
if you have a a set of mutually
exclusive possibilities and that that
constitutes everything that can happen
then they all need to add up to one
um
and then if you're if you're doing
probabilities uh about
like logical sentences then there's all
these extra rules that that make perfect
sense like you have statement a and
statement b you also have statement a
and b then statement a and b has to be
less likely than a or b or the same it
could be the same but like
if a has
20 chance of happening
then a and b
couldn't possibly have more than 20
chance of happening right even if b is
guaranteed to happen it can't so that's
like a rule if you find in you're doing
your probabilities you have a and b that
has a higher probability than either a
or b then you've made a mistake uh that
kind of thing you have these rules it
would be nice if we could find
some way of doing of dealing with
logical uncertainty some some similar
set of rules that is useful
and so um
and so related to that we have these
things like um
like dutch book arguments
which are
about uh
i don't know why they're called dutch
i feel like the english language just
like
is so harsh to the dutch
for no reason
unjustly maligned but anyway um
where like if you if basically there's
there's theorems which uh
which you can prove that if your beliefs
don't obey
these rules
then there will exist some set of bets
by which you're guaranteed to lose money
some set of bets that you will happily
take by looking at your probabilities
um that you cut you can't possibly win
whatever happens you lose money
and that seems like it's stupid like you
don't want that in your beliefs so
that's like an argument that says
um
that your beliefs should obey these
rules because if they do you're at least
never gonna end up in a situation where
you're guaranteed to lose by betting and
that's like one of the
things we want beliefs for
we want some kind of equivalent thing
for logical uncertainty
um and that's what this paper is trying
to do it's trying to come up with
a rule so yeah you could you could say
if if you are doing your probability
theory stuff in such a way that there
are no dutch books that can be made
against you
then that's good and you're doing well
and so when we're talking about advanced
ai systems we're gonna it's like a good
assumption to make that they're at least
gonna try
to not have any dutch books against them
in the way they do their probabilities
so they will probably be obeying
probability theory
and it would be nice if we had some
equivalent thing for logical uncertainty
that said if you are satisfying this
criterion in the way that you do this
then
uh you're not going to be like doing
obviously stupid things and that's what
this paper is trying to do when you're
talking about probability there's you
can kind of think about what properties
you would want your
system of deciding probabilities to have
people have written down various things
they would want from this system right
that would be a system of like assigning
probabilities to statements logical
statements
and and then being able to like update
those over time
so uh one thing you would want is you
would want it to converge
which just means if you're thinking
about
a logical statement you might think of
something that makes it seem more likely
and then think of something else like oh
no actually it's less it should be less
likely and whatever you can imagine
some systems for some statements would
just think forever and constantly be
changing what they think and um never
make their mind up that's no good right
we want a system that if you give it
infinite time to think we'll eventually
actually decide what it thinks
so that's one thing you want to converge
secondly you obviously want it to
converge
like to good values so if something
turns out to be provable within the
system that it's using then you would
want it to eventually converge to one if
something turns out to be disprovable
you want it to eventually converge to
zero and then the rest of the time you
want it to be like well calibrated which
means
if you take all of the things that it
ever thought were eighty percent likely
to be true
of those
about 80 should actually end up being
true right it should when it gives you
an estimate of the probability that
should be
right in some sense another one is that
it should never
um and this one's like a bit
controversial but it seems reasonable to
me that it should never assign
probability one
to something which
uh is not proved
or that can't be proved and it should
never assign probability zero to
something that can't be disproved
um they call that non-dogmatism at the
end of all of this again at infinity
after thinking forever the probabilities
that it gives to things should
follow the rules of probability right
the ones we talked about before those
rules right it should at the end of it
you should end up with like a valid
probability distribution over all of
your stuff their criterion that they've
come up with which is the equivalent of
there are no dutch books
is based on this algorithm that they've
made which
uh
satisfies this criterion and therefore
has a whole bunch of these nice
properties that you would want
a logical inductor to have
so
the way the algorithm works is
it's one of the zaniest algorithms i've
ever
seen um
it is
it's weird and wonderful and if you're
if you're a person who is interested in
interesting algorithms i would really
recommend checking out the paper it's
very technical because we don't have
time in this video and i may go into it
deeper on my channel at some point if i
ever come to actually understand it
which to be honest i don't fully right
now
um
but it's based around the idea of a
prediction market
which is
a super cool thing that's worth
explaining in financial markets as they
currently exist you can have um futures
right and a future futures contract it's
a contract which says
i promise to
sell you
this amount of stuff at this price
of a particular thing right so you say
take a date a year from now and you say
i'm gonna sell you this many gallons of
jet fuel
for this price
um at least that's the way it used to be
down in practice the actual jet fuel
doesn't move you just go by the price of
jet fuel
and you pay the equivalent as if you had
and then they can go and buy the jet
fuel themselves
but this is like a really useful
financial instrument to have because
let's say you run an airline
your main cost is jet fuel prices are
very volatile if prices go up you could
just go under completely
so you say okay i'm going to buy
a whole bunch of these jet fuel futures
and then if the price goes up i'm going
to have to pay loads more for jet fuel
but i'll make a load of money on these
contracts
and it if you do it right it exactly
balances out
the cost of that is if the price of jet
fuel falls through the floor
then um
you don't get to save money because
you're saving money on jet fuel but then
you're losing all this money on the
contracts so it just sort of like
balances out your risk it lets you lock
in
the
the price that you're going to pay a
year in advance and that's just like
super useful for all sorts of businesses
really really good in agriculture as
well because you don't know what your
yield is going to be in
so um
so the point is that the price that you
can buy these contracts for
has to be a really good prediction of
what the price of jet fuel is going to
be
at the time when the contract ends
and you can kind of treat the price of
that as a combination of everybody's
best estimate because if you can predict
the price of jet fuel a year in advance
better than anybody else you can make
like almost arbitrary amounts of money
doing that
um and in doing so
you bring the price
to be a more accurate representation
right if you think the price is too low
and it's going to be more then you're
going to buy a bunch and when you buy it
that raises the price of it because you
know supply and demand
so
these prices end up representing
humanity's best estimate of the price of
jet fuel year from now
but the thing that's cool is
you could build arbit like this is just
a piece of paper with stuff written on
it right and these days it's not even a
piece of paper in principle you can
write anything on there right
so what if you wrote a contract that
said i promise to pay
a hundred pounds if
such and such horse wins the grand
national
and zero otherwise
and then that's on the market and people
can buy and sell it right so if this
horse is like guaranteed to win the
grand national then
this contract is effectively a hundred
pound note right it's it's a sure it's
money in the bank as if
why is that a phrase are banks banks
aren't that reliable but anyway if the
horse is like guaranteed to lose then
it's worth zero and if
the horse has a 50 chance of winning
that thing is going to trade for 50
pounds
and so by making these um
contracts and allowing people to trade
them
you can
get these good like unbiased and as
accurate as you can hope for predictions
are future events
and that's what prediction markets are
for you can make money on them by being
good at predicting stuff
and anybody else can just look at the
prices things are trading at and just
directly convert them into probabilities
uh and that's
in a in a spherical chickens in a vacuum
kind of way obviously in practice
there's various
uh things that can go wrong but like
in principle this is a really beautiful
way of effectively doing distributed
computing
where you have everybody is doing their
own computation and then you're
aggregating them using the price
mechanism as communication between your
different nodes and you yeah like
super cool
so
that's what logical induction does
it like simulates a prediction market
where all of the contracts are a logical
statement i think in this it's not a
hundred dollars it's one dollar so if
it's trading at one dollar then it's for
sure it's
certain um
saying you know this is currently
trading at uh 60 cents and i think it's
more more than 60 likely to be true so
i'll buy some and that seems like a good
investment
and so all of the traders in the
algorithm are
programs they're computer programs
um
and it turns out that if you run this
uh and this is computable it's not like
tractable in a practical sense because
you're running vast numbers of like
arbitrary programs but um it is in
principle computable
it
ends up having the market it as a whole
ends up having uh ends up satisfying all
of these really nice properties is that
because it comes into balance yeah as
the as the traders trade with each other
the ones who are like good at predicting
end up making more money
and the ones who are bad at predicting
like go bankrupt effectively you might
imagine some
trader in this system which is a program
that just goes around looking for
situations in which you have
a
and b and also a and b in the system and
the prices don't work and it's like
arbitrage
right
so um there are people who
do this currently on the stock market
there's different things you can do
where you can say oh you know anytime
the
um
any time the price is different between
two markets you can make money by buying
in one and selling in the other and
stuff like that you can do arbitrage um
and it's the same kind of thing you can
have so some of these programs will get
rich by just saying oh i notice
that
this statement for a and b
has a probability that's actually higher
than the probability of b so i'm gonna
sell that you know or i'm gonna i'm
gonna buy some b to razer you know i'm
gonna like respond to that
in a way that makes me money and because
there's all of these traders
they will uh
they will eventually
cause everything to line up right and so
the thing will converge
this is the equivalent of saying there's
no dutch book
for your probabilities
the logical induction criterion the
market
satisfies it if there's no efficiently
computable trader
that can exploit the market which means
that it can
like find a reliable strategy for making
loads of money without risk um as long
as you have that
then it satisfies the logical induction
criterion which then means you get all
of these other properties for free some
of the properties that that it has are
kind of crazy
like
it's okay with self-reference
it like doesn't care about paradoxes
there's all kinds of really cool
things that
um that don't trip it up bringing us
back to what we started talking about
how does the paper relate to the ai
safety thing right right so we're trying
to
do reasoning and possibly prove theorems
about very powerful ai systems and that
means that we want to be able to think
of them just in terms of
being good at thinking
and we've got
a lot of good theory that pins down like
what does it mean to be good at
empirical uncertainty we have all of
probability theory and like statistics
and we can we can say
these are the things you need to do to
be good at probability and we have like
rational choice theory
and we can say this is what it means to
be good at making decisions
um and so then when we're reasoning
about ai systems we can think well it's
probably going to be good at reasoning
about uncertainty and making decisions
um but we have to also assume that a
very powerful hypothetical future ai
system would be good at reasoning under
logical uncertainty because it's going
to be a physical like bounded system it
is going to need time to think about
things and it probably is going to need
to like make decisions based on things
that it hasn't
logically thought through every possible
consequence of yet so it's probably
going to be good at this too and we need
some like
formal framework with which we can think
about what it means to be good at
reasoning about logical uncertainty um
and that's like what this paper is
trying to do
they're erasable
yeah okay
um i do also have oh i had sharpies yeah
i do have sharpies oh yeah these they
squeak on the paper people don't like
okay fair enough in fact all pens do but
these are slightly better right friction