LEON NICHOLLS: Welcome to the next episode
of this season's "Assistant on Air," where
we're chatting with designers and developers who recently
created some fun games for the Google Assistant
smart displays.
[MUSIC PLAYING]
I'm Leon, and I'm your host for today.
I'm a developer relations engineer
for the Google Assistant.
Today I'm talking with Richard, technical director
at MediaMonks who worked on two of our games, "Cookie
Detective" and "Junior Chef."
Hi, Richard.
RICHARD HEUSER: Hi, Leon.
And thanks for this opportunity.
When you're working on projects like this,
you're constantly looking ahead, focusing on the challenges.
So it's great to be able to take some time
and go over the highlights with you.
LEON NICHOLLS: Yeah, it was certainly a fun project
to work together.
So Richard, tell us a little bit more about yourself.
RICHARD HEUSER: Sure.
I started programming back when 8-bit color was
the state of the art and pixels had
to be coaxed to the screen one by one.
And a lot has changed since then.
But I liked it that way.
So new tools and new tech to apply
in exciting and creative ways, that's my thing.
In 2018, I joined the MediaMonks as a technical director.
And before that, I did everything
from game development to working with massive data sets.
But being able to work with Google's Voice tech
is definitely one of the more fun experiences.
LEON NICHOLLS: So that's great to hear.
So Richard, tell us a little bit more about the games you
worked on.
RICHARD HEUSER: Yeah.
"Junior Chef" is a game that puts you
in the role of a chef in the restaurant kitchen.
You have to tell cooks what to do in order
to make sure that the recipes get executed properly
and in time.
So it's a fun way for kids to learn a bit about time
management and planning ahead and, of course,
about cooking as well.
And it could even inspire their parent,
who is maybe cooking dinner alongside them, because it
features a variety of recipes from all over the world.
Now, "Cookie Detective" challenges
you to find a cookie which has hidden itself in the kitchen.
And you have to do that by asking
clever questions to narrow down the hiding spots.
LEON NICHOLLS: Great.
So it's good to hear how you thought
about the opportunities of using voice input in the game
because a lot of developers don't have experience
with voice design.
Now, you were also one of the early developers who had access
to some of our new tools, including the Actions Builder
GUI, and then later the Actions SDK.
So can you talk to us about how you
used those tools in the development of the games?
RICHARD HEUSER: Yeah, sure.
We started out by setting up everything in the new Action
Builder UI to get familiar with the structure and terminology.
But once we received the SDK, we immediately
moved everything to the YAML files,
and we started automating and scripting all the processes.
This was also the way we were working with the old platform.
And we really prefer having everything
in code in the repository so that we can easily
share it across the team and so there's versions.
So yeah, the SDK was really a breath of fresh air,
because it allowed us to very easily automate
all these processes and use the configuration
files to do everything.
So yeah, it was a great time-saver
and a very welcome addition.
LEON NICHOLLS: That's great to hear.
So we knew the SDK was going to be a hit with developers
because they can keep using their existing development
tools and their existing processes.
Now, let's dig into the conversational design of it.
Now, one of the main mechanisms for trying
to understand what the user is trying to do is to use intents.
Can you talk about how you design the intents
and also how that helped to improve the speech biasing?
RICHARD HEUSER: Yeah.
Yeah, that was an interesting challenge.
And I'd like to think that we helped improve
the platform a bit along the way there,
because due to the all the different ingredients in steps
of the recipes in "Junior Chef" and all the different hiding
spots and attributes of the hiding
spots in "Cookie Detective," there
are an enormous amount of different inputs
that the user can give.
So it really wasn't an option to create
an intent for every variance.
So we decided to rely heavily on parameters.
But as it turned out, that initially
didn't play very well with the speech biasing.
So we tried a few different approaches to tackle that.
We added thousands of training phrases
to each intent to get as many different variations in there
as possible to help the speech biasing alone.
We even used the explicit speech biasing in the webhook,
although we later removed that when the automatic biasing was
improved.
But we did leave all the training phrases in.
LEON NICHOLLS: Yeah.
I must admit these games definitely
helped us find some gaps in our speech biasing.
So thank you for that.
Now, there was some other interesting challenges as well.
Given the capabilities of the hardware of the smart displays,
what kind of design decisions did you
have to make to ensure that the user experience is still good?
RICHARD HEUSER: Well, you have two opposing forces--
the memory capacity and the loading time.
You want to preload your assets as much as possible so you
don't have any stutters, but you always
have to manage your memory.
So we decided for the audio, because that
is by far the largest amount of data,
to go for a lazy loading approach.
So all the recorded audio is loaded right
before a scene starts and is released from memory again
at the end of the scene.
And for the visual essence in the animations,
we just fetched all of that at the start of the game,
because that wasn't so much.
And yeah, the available memory actually
allowed having that in memory all the time.
So yeah, to be honest, we could have
gone a lot further in optimization there,
but it wasn't really necessary.
LEON NICHOLLS: It's great to hear
that you could leverage some common way of optimization
techniques in the games.
Now, you were also an early tester of one
of our new features called Home Storage, which
is a way to store data across users and devices
in the same home.
So, how did you use that in the games?
RICHARD HEUSER: Well, eventually for everything,
we started out thinking that we should only
store the high scores there, because you
want to share those across the household typically.
But then we figured why not store everything in there
because users would also want to pick up a game
that they left off in one room of the house
in another room on another device.
So just storing all the user information,
and all the game history, and the unlocks in the Home Storage
allowed them to basically switch devices and continue
where they left off.
And who wouldn't want that?
LEON NICHOLLS: Great.
It's nice to hear how you considered
how users would use the smart display devices in the home.
By the way, there's a fun fact.
The animals that we use for the users' names in the game
is actually borrowed from the Google Docs team.
They use that for the anonymous users.
So thank you very much, Richard, for giving us
the technical insight into the development of the games.
RICHARD HEUSER: Yeah, my pleasure.
I hope we can inspire some of the viewers
to go and build their own games and other creative applications
for smart displays, because I think
it's a medium with an enormous untapped potential.
And it's unique in how it combines
voice interaction with web capabilities
and the accessibility of an appliance.
LEON NICHOLLS: And that's a wrap for today's episode.
If you want to try out the games we
are highlighting in this season, check out the links below.
Thanks for watching.
[MUSIC PLAYING]