So that actually brings us to the end of the PomDP topic.
>> How do you know at the end of the PomDP topic?
>> We have high probability of believing that we're at the end of the pond E.P.
topic because the observation that we're making is the what have we
learned slide.
>> [LAUGH] >> So what have we learned?
>> Well, we learned about PomDPs.
>> Partially Observable Markov Decision Processes?
>> Well we had been exposed to these things before but we really kind of dove
into sort of some detail about what they are and how to solve them.
And in particular,
we've talked about belief states as kind of a representation, which represents
the states that we might be in and how much we believe that were in them.
We talked about PomDPs as a strict generalization
of MDPs which is sort of important.
And that if we can solve a PomDP that happens to be an MDP,
we will solve the underlying MDP.
One of the unfortunate consequences of all that is that dealing with PomDPs
is hard they're hard to solve they're to sort of fundamentally difficult.
I can't remember the exact result now but
there was something in there about undecidability.
>> Yeah. >> And that just that was difficult.
That was hard for me deal with.
>> [LAUGH] It was hard for you to accept hard to accept.
[LAUGH] >> Right,
despite the fact that it's hard, we did come up with some algorithms for
solving MDPs like a value iteration.
I remember something about piecewise linear and convex.
>> Right, and representing value functions as piecewise linear and
convex functions.
>> Or pulc and one of the things that was nice about that is that you show
that you could build up these sets of kind of linear functions.
And because you only cared about the max in a given point,
you could throw away a bunch of them that might be unnecessary and
sort of keep the hard problem possibly manageable.
>> Good. >> That was really nice.
Then we went from the solving of PomDPs to reinforcement learning with PomDPs,
and if solving PomDPs is hard then RL for PomDPs definitely has to be hard,
and that turns out to be true but, we tried to do it anyway.
Let's see, we did memoryless.
>> First we talked about a model-based method.
>> Right, model-based method.
>> Where you try to actually learn the PomDP from
experience wandering around in it.
And do you remember how we how we did that learning?
>> With great difficulty?
>> Well sure, we talked about explicit expectation maximization as a way of
trying to organize that computation.
>> Sure, that's a favorite of my expectations.
>> Your maximally favorite?
>> [LAUGH] Yeah, my maximally favorite one.
We've got to go all the back and try to remind ourselves how Worked.
>> Yes. >> That was fun.
[LAUGH] And then we did memoryless?
>> Yeah, and that we needed in the memoryless case we might need to
be random.
>> Again, just like grad school.
>> [LAUGH] I don't think that's true.
You're saying that if you're in grad school but
you don't remember where you are in grad school, then it helps to act randomly.
>> Well it certainly explains the behaviors on my grad.
Okay, then we had, we probably did more stuff there but I can't remember it.
>> [LAUGH] >> And so I do remember though that we
started talking about Bayesian RL.
>> Yes.
And the cool thing about Bayesian RL being that it blurs the line between
planning and learning so.
And Bayesian stuff does that in general, this sort of notion that,
well all learning really is is estimating probabilities of things,
it's not actually learning, so just changing your posterior.
And so, decision making in the Bayesian RL setting ends up becoming just a kind
of version of planning in a continuous ugly kind of PomDP.
>> Speaking of which, if we have Bayesian RL, do we have frequentist RL?
>> Well sure, I mean Bayesian and
frequentist are kind of rivals in a sense.
So anything that's not Bayesian maybe is frequentist.
>> Okay.
>> So all the other reinforcement learning that we did like hue
learning and whatnot.
As it certainly has a frequentist flavor in that what is frequentist statistics
is about, it's about you know counting up how many times something happens and
divided by the number of times you asked.
And that is definitely, queue learning has that flavor model based RL where
you're trying to estimate the parameters of the model by
from experience they tend to have that flavor.
>> Okay, so good.
So then we have been learning about frequentist RL.
And now that we have Bayesian RL,
it reveals the frequentist things that we'd been doing in the past.
So I'm okay with that.
>> Yeah and the Bayesian RL is very nice because it doesn't make an exploration
exploitation distinction anymore.
You're always acting given your knowledge of what the truth is and
you're acting in a way that maximizes your award given that knowledge.
>> Right and so exploration is exploitation.
[SOUND] >> Planning is learning.
Exploration is exploitation.
War is peace.
>> [LAUGH] And we have always been at war with supervised learning.