>> You're not going to want to miss this episode of The AI Show,
where we learn about how to make
your own video game robots that learn on their own.
It's pretty cool, make sure you tune in.
[MUSIC]
>> Hello. Welcome to this episode of The AI Show,
where I am pleased to welcome again,
Dr. Katja Hofmann, Principal
Researcher at MSR. How you doing, my friend?
>> I'm very good. Thank you.
>> Fantastic. Well, it's been almost years since we've chatted.
Why don't you remind everybody what you
work on so they can get a sense for what you do?
>> Fantastic. Last time we
chatted we talked about reinforcement learning.
It's a type of AI that we're very excited about in my team.
Myself and my team work on reinforcement learning algorithms,
especially in the context of gaming.
So understanding what the impact of reinforcement learning
could be in gaming and what new player
experiences it would enable.
>> For those that are familiar with reinforcement learning,
can you give us like an elevator pitch
for what it is and why it's so cool?
>> Absolutely. Reinforcement learning is
a very general type of
machine learning that formulates a problem from
the perspective of an agent that
interacts with the world by taking
observations about the current state of the world
and by taking actions that have some kind of effect in the world.
You can imagine, of course,
in a game that's pretty appealing because it allows you to create
game characters that experienced
the game from the point of view of getting some observations,
some sensing about what's going on in
the game at this point in time, their actions.
Learning takes place through
a reward signal that is created, for example,
by the game designer or game developer to
put some pressure on
the agent by interacting with the environment.
In this way, the agent observes what kinds of
rewards are associated with its action,
so it can learn about long-term,
if I take this action at this point in time,
does that lead to a good outcome or a bad outcome.
Reinforcement learning is really a formal method for working out
long-term consequences in taking
sequences of actions in a complex environment.
>> In this case, we're doing it in the context of games.
If I remember right,
when we chatted last,
you were working on this project called Project Malmo.
Why don't you describe that briefly?
>> Absolutely. So Project Malmo was
the first project that my team and I went public with.
This was really mainly targeted at researchers,
allowing them to use the game
Minecraft as a platform for doing reinforcement learning research.
If you remember, I was hugely
excited about Minecraft as a platform for this,
and I still am,
because Minecraft is such an open-ended game.
We're really seeing this as
a kind of game that allows this whole trajectory,
from relatively simple agents and relatively simple environments,
all the way up, eventually,
to agents that solve complex problems within the game Minecraft.
That also tells us something about how we can create
such agents for other real applications that
require agents to learn about their environment and
take complex sequences of actions in order to achieve their goals.
>> I'm a huge fan of Minecraft,
particularly because it's so open-ended
and you can have any type of goals that you want,
like not die, for example,
otherwise to build amazing things.
So what you're saying is that
reinforcement learning is a technique that allows
an agent to learn based upon what is going on around it?
Does that work or does it really anything?
>> That is really a good way to think about it.
If we have a game like Minecraft,
then a researcher or
a game creator or someone who
was to train an agent within the game,
Minecraft, would still have to provide a reward signal.
So this is really a way of expressing that goal.
You mentioned not dying.
You could imagine an agent that you
let loose in the game of Minecraft and
you just rewarded for surviving for as long as possible.
We have an ongoing great collaboration with
various university and other partners
including the CMU, for example,
where the goal is to create
reinforcement learning agents that learn
to mine a diamond in Minecraft using reinforcement learning,
so interaction with that environment,
and a very small amount of
demonstration data from a human player showing how get a diamond.
You can imagine, this is a really hard task by itself.
It's a really nice way to test out
the current state of the origin reinforcement learning
and drive improvements there.
But you could also imagine millions of
other tasks that we could be training our agents to do.
>> This is really cool, but it's been several years.
What's new?
>> One project that my team and I just announced in
collaboration with several other partners
is called Project Paidia.
This is a research collaboration with Ninja Theory,
which is a game studio located
very close to us here in Cambridge in the UK.
It just so happened that as we looked at the research
challenges that we were addressed and identified
that one of the key challenges that we want to tackle
is understanding how to create
AI agents that learn to genuinely collaborate with human players.
It turned out that Ninja Theory just had
the perfect game to allow us to test that out.
Their game, Bleeding Edge,
is a team-based game that really relies
on tight cooperation within a team.
It's four versus four.
One of the challenges that we set to us
in this research collaboration was to, one,
understand how to get
reinforced material into a modern video game,
as Bleeding Edge was something that the team
was working on right at that very moment.
Then also, understand how we can create agents that,
for example, can perform
well in a team when coupled with human players.
Agents that really have to understand,
what is this human player trying to do?
How do I respond to this as well as possible?
How do I coordinate my actions so that my team is successful?
>> It feels like the difference between
Bleeding Edge and Minecraft is very subtle.
Would you might diving into how it's actually different?
>> Absolutely. Minecraft is
usually open-ended, as we just mentioned.
There, we could in principle create a variety
of tasks that require collaboration between,
let's say, a human player and AI agent.
The nice thing about Bleeding Edge was that it allowed us to have
a more focused proof point or
research case study where there already is this game,
under development, where we know that human players are able to
effectively coordinate and are able to,
for example, make strategic decisions
depending on their teammates abilities.
That provided that more focused platform, really,
around understanding how to
get agents to do this cooperation with human players.
>> That's interesting because if I'm understanding it right,
in Minecraft, in the Project Malmo,
you had to give it a reward function.
But here, it can actually learn
what the rewards are based upon other players,
is that fair to say?
>> Yeah, ultimately in the game the reward is quite obvious.
If your team wins, you succeed.
There, the higher level goal is already given,
which wouldn't necessarily be the case in Minecraft,
there we would have to decide this on our own.
We can still tweak the reward function that's given to the agent.
But ultimately, the goal is clear.
If you're able to successfully
collaborate with your teammates, then you win the game.
>> That's interesting. I'm thinking about the Minecraft situation.
There really isn't someone else that you can collaborate with,
and so it doesn't have anything.
I mean, I guess it learns from its surroundings,
but now, it can actually learn from a human as
opposed to a reward function in some artificial environment?
>> Absolutely. One of the types of
techniques that we are exploring in the context of
our collaboration with Ninja Theory is to understand
how we can use demonstration data,
so human player data,
to start imitating off and
essentially, interpolating between trying to imitate
human players as much as possible
with also using reinforcement learning
to really learn about the consequences of
actions and try to perform as best as possible.
>> Well, this is really cool. Do you have anything to show us?
>> Absolutely, I'm glad you asked.
In the context of project Paidia,
we just released this AI innovation page.
Let me share this page.
What you're seeing here is
the AI Innovation page for Project Paidia that we just announced.
What I want to show you here is
this interactive experience where you can experience,
right in your web browser,
what it's like to interact with
a trained reinforcement learning agent.
For this game, we trained
a reinforcement learning agent to collaborate
with a human player in this escape room game.
In this game, I am,
or the human player, is this white robot right here,
while the blue robot here is
controlled by a reinforcement learning agent.
As you can see, the reinforcement learning agent
is trying to be headfirst.
In this game, there are two things you need to do.
One agent needs to be on this red switch in order to activate
these platforms for the other agent
to toggle and once these are toggled, I'd be able to escape.
Now, we've already seen that
the trained reinforcement learning agent
is trying to be head first,
given that I'm not moving,
but it can also move close to it and to see if it can pick up
that I want to change rows and stand on the switch.
We can see that it has correctly understood what I'm trying to do,
and now, again, it's collaborating with
me by flipping those switches and opening the door for me.
Thank you so much, and now, I can escape.
>> That's really cool.
Basically, it understood that
you two needed to collaborate somehow,
and it was doing what it thought was the right thing.
But then when it saw you get close like, "Oh, okay,
we're switching our responsibilities here,"
then it went ahead and did that act.
>> Absolutely. That's what it's been trained to do,
and it's able to understand the human pairs behavior that way.
>> That's really cool.
What's your call to action?
Where can people go to start writing or playing with
this system or creating more reinforcement learning algorithms?
>> There are two resources that we can
show right here in the video.
First, make sure they link to this AI innovation page.
You can not only try the interactive experience,
but you can learn a lot more about reinforcement learning
and what it takes to make it possible in video games,
as well as learn something about
the latest research that we're doing in the space.
We'll also show a link to an Azure resource where
you can get started training
your first reinforcement learning agents today.
>> I'm curious, can you show us this particular one as well?
What's the robot trying to do here,
and how do you escape?
Is it the same kind of game?
>> It's the same game. The only thing that changes between
all the instances is the position
of the switches and those platforms.
Here we're showing that the agent
has learned the general behavior,
the general meaning of what the switches and
the platforms are and what their functionality is,
rather than having been hard-coded to follow a particular path.
>> Well, this has been amazing.
Any final words, Dr. Hofmann?
>> Well, we're just hugely excited to
share this interactive experience as well as our learning.
Do try those things.
Try to train your first reinforcement learning agent,
and think about how to use reinforcement learning,
maybe in your own video game.
>> Well, this has been amazing.
Thank you so much for spending some time with us.
Thank you so much for watching.
You've been learning all about reinforcement
learning from Dr. Hofmann.
Make sure you try out those amazing links.
Thank you for watching and we'll see you next time. Take care.
[MUSIC]