[MUSIC PLAYING]
DALE MARKOWITZ: Hey, I'm Dale.
This is "Making with ML."
And today I'm going to show you how this adorable little fox is
the future of video game interactions.
[MUSIC PLAYING]
So imagine you're playing this sick new game where
you decide to visit your local shopkeeper,
but the shopkeeper doesn't have what you need.
What if instead of selecting from a list,
you could ask for what you want in plain English,
while the game uses machine learning to come up
with a fitting response?
Or what about making objects, like cats and trees,
more interactive, action the game developers didn't even
consider?
Can ML expand the ways players interact with the game world?
Today, I sat down with Anna Kipnis,
a senior creative technologist with Stadia Games
and Entertainment, and we talked about how
machine learning can make game worlds more interactive.
ANNA KIPNIS: In my lifetime, we've
seen games go from very low fidelity, like maybe two
colors, like "Pong" or something like that,
to extremely realistic 3D.
So you know, the visual development of games
has really exploded, but I think that the interactivity is still
really lagging behind.
Like, we're not that far from where we started, really.
And so what I'm really interested in
is kind of closing that gap.
DALE MARKOWITZ: Why is it so difficult
today to build a game that supports
all of these rich, varied, and spontaneous interactions?
ANNA KIPNIS: Right now, the process
is that the developers have to anticipate
what all the questions that the player might
ask of their world.
Like, are they going to ask about this specific weapon
or something like that?
What should happen as a response to that, right?
So, at the moment, the developers
have to account for both the cool answers
that they can have in the world, but also every single question
that the player might ever have.
What semantic ML allows you to do
is that it allows you as a developer to focus mainly
on the interesting answers.
DALE MARKOWITZ: The technology Anna is talking about here,
semantic ML, is a way of making sense of words or sentences
with machine learning.
Given a set of words, you can use a model
to see which pairs are the most similar.
For example, an ML model can tell you
that a jetpack is more similar to a flying broomstick
than it is to a garbage can or an apple.
You can also use semantic ML to help pick out
likely responses to queries.
So if I say, "I would like some coffee,"
Semantic ML can tell me, "here's a cup" is a better
response than "I hate ur butt."
Anna uses semantic ML to describe her game worlds
in plain English and then have a model infer how
the world should behave.
Let's take a look.
ANNA KIPNIS: I've labeled objects
using English that may be interesting for the fox
to interact with.
And in pink, you see the sorts of actions
that the fox can do on these objects.
It can predict, OK, of the list that you just gave me,
if this is what the player said or did,
it will be able to rank them from best
to worst in terms of what is the most likely thing
that somebody in conversation might have done.
So, first, let's say hi to everyone.
It waves.
And then let's see if we can have some coffee here.
So it goes to pick up the cup.
DALE MARKOWITZ: So cute.
ANNA KIPNIS: Yeah, then it brings it to us.
And the coolest thing is I didn't program the Fox how
to answer these questions or even, you know,
what the heck coffee is.
I just labeled that object that kind of looks like a cup,
and I put it like a small mug.
That's just English words.
And semantic ML kind of did everything else.
You can type sentences, they don't even
have to be particularly spot on, right?
Like, you can say, "sofa," and it will know it's couch.
I actually have the system rigged
up so that you can just do them straight up in Google Sheets.
And then you can try these really bizarre things,
like get really stoked.
I get excited.
Cool, make some [INAUDIBLE].
Oh, whoops.
It conjured a painting because I typed
"Monet" instead of "money."
And you know, this thing actually
has cultural associations.
And so it understands that Monet is a painter,
and so it conjured a painting.
So, yeah.
And so here now that it's spelled correctly,
it creates a coin.
DALE MARKOWITZ: What you just showed me,
it's exactly what you're saying where like, you
as-- even as the person that built the demo,
it's still surprising and spontaneous to you.
ANNA KIPNIS: Exactly.
This is like what's so magical to me
is that there is essentially a way
to condense some of these patterns
that people have in conversation,
condense all of that human interaction into a usable form
that you can have in a game.
To me, that's like, it's essentially
indistinguishable from magic.
DALE MARKOWITZ: If you want to learn more about the tools
that Anna used to build her demo and how you can use them,
make sure you check out the blog in the link below.
And let us know what you build in the comments.
See you next time.