YUFENG GUO: Welcome.
Today on "AI Adventures," we're joined in the studio
by Justin Zhao, a Google research engineer.
Hi, Justin.
JUSTIN ZHAO: Hi.
YUFENG GUO: Thanks for joining me in the studio today.
JUSTIN ZHAO: Yeah, it's great to be here.
YUFENG GUO: We're going to be talking today
about natural language interfaces
and how computers and humans can talk to each other in ways
that are natural and not awkward.
JUSTIN ZHAO: Yep, sounds good.
YUFENG GUO: Awesome.
So I want to start by talking a little bit about your team's
area of research and the general natural language processing
field.
And then we'll delve into your area of research
and see where our conversation takes us.
JUSTIN ZHAO: Yeah, that sounds great.
So broadly speaking, the area of my research
is natural language processing, or NLP.
YUFENG GUO: OK.
JUSTIN ZHAO: And what that is, NLP
is all about trying to understand
how humans communicate with each other
and how to get a computer to replicate that behavior
so that we can interact with computers
in a more natural manner.
YUFENG GUO: Wow.
You guys really picked a small field to target there.
JUSTIN ZHAO: [LAUGHS]
YUFENG GUO: Yeah, NLP sounds super broad.
JUSTIN ZHAO: Yeah.
YUFENG GUO: It's like everything.
JUSTIN ZHAO: Yeah, it's pretty broad.
So in fact, I have some slides that we can pull up just to try
to focus it a little bit.
YUFENG GUO: Yeah, that'd be great.
Yeah.
JUSTIN ZHAO: Yeah, so first, I think
it's important to talk about the conversational user interface.
And for something like the Google Assistant,
there's two big domains of NLP problems that come into play.
On one side, you have the problem
of understanding, which is, what do the users say?
What was the user's intent?
And on the other side, you have the problem
of generation, which was, what should we say to the user?
And how do we respond in a way that's
intelligent and conversational?
YUFENG GUO: Right, that makes sense.
JUSTIN ZHAO: So I work on the generation side.
And the ultimate goal of natural language generation
is to teach computers to turn some kind of structured data
into natural language, which we can use to respond
to the user in a conversation.
YUFENG GUO: Wow.
And this is definitely something that I
feel like conventionally, NLP has really been broadly thought
about as a field where it's all about processing the words
and understanding what text means.
But you are working on the generation
side, which, in a lot of ways, often get overlooked.
And so it's really great that you're
able to tell us more about this side of things.
JUSTIN ZHAO: Yeah, that's what I'm here for.
[LAUGHTER]
YUFENG GUO: So how do you then teach a computer
to generate natural language, rather than just understand it?
JUSTIN ZHAO: Right.
So for now, let's set aside the structured data part
of natural language generation.
And we can focus on the natural part of the natural language
generation.
So what makes a conversation like the one
we're having feel human?
YUFENG GUO: Speaking of the one we're
having, it's a little meta that we're having
a conversation about what makes something conversational.
JUSTIN ZHAO: So that's a common remark on our team.
YUFENG GUO: Yeah, we have to not be too
robotic in our conversation.
JUSTIN ZHAO: Yeah.
[LAUGHS]
So I think this breaks down into two kinds of requirements.
First of all, the content of what we have to say
has to make sense in the context of the conversation.
So is what I'm saying an appropriate response
to what you're saying?
Or is it out of the blue?
YUFENG GUO: Hey, what are you having for dinner?
JUSTIN ZHAO: [LAUGHS] So--
YUFENG GUO: That's kind of out of the blue.
JUSTIN ZHAO: That's kind of out of the blue, yeah, definitely.
So yeah, exactly.
And then I have to think about if what I'm going to say
is actually going to answer your question.
So if you were going to ask me where we want to go for dinner,
it would be weird to suggest a coffee shop or a clothing
store.
YUFENG GUO: Right.
JUSTIN ZHAO: Yeah.
YUFENG GUO: Yeah, unless you really
wanted to get some coffee stains on your clothes for dinner.
JUSTIN ZHAO: Yeah, I guess so.
The second requirement is that you actually have
to use the language correctly.
So this is like, how's my grammar?
Do my verbs agree?
Or if I'm using a pronoun, is it ambiguous?
YUFENG GUO: That makes sense.
So it's basically what do you say,
and then how do you say it?
JUSTIN ZHAO: Exactly, yeah.
YUFENG GUO: OK.
And you also mentioned earlier, there is this structured data
that we put aside.
Where does that come into play?
JUSTIN ZHAO: That's a great question.
So structured data primarily helps
us figure out the first requirement, which
is what we want to say.
For example, let's say a user asked us about the weather
next week in Santa Clara.
In Google Search results, we see a box
filled with all this information about the weather
for the next week.
And somewhere within this data, hopefully,
answers the user's question.
And we just have to figure out how
to turn all this data into a response to the user.
That's the problem that we're focusing on
in natural language generation.
YUFENG GUO: And that's because we're talking about a situation
where we're going to say our answer
and not just show them a box to look at.
JUSTIN ZHAO: That's correct.
YUFENG GUO: OK, so It's like an audio interface.
Gotcha.
JUSTIN ZHAO: Right.
YUFENG GUO: And in that case, I guess
I can imagine a naive solution for this sort of problem.
We already have the data.
Right?
JUSTIN ZHAO: Yeah.
YUFENG GUO: But I don't know if it would be sufficient.
JUSTIN ZHAO: Well, you know, that depends.
By all means, go for it.
YUFENG GUO: All right, so let's say we
make some kind of a template.
Right?
And we can say, on blank day, it will
be blank temperature, and then some blank weather condition.
Like on Tuesday.
It will be 72 degrees and partly cloudy.
And then you could build a full forecast
by just iterating through all the days of the week like that.
JUSTIN ZHAO: So I will say that that is a very straightforward
approach.
And some assistance do use that implementation.
However, in practice, it's a lot less conversational
than you might think.
So how about you try asking me what's
the weather like this week?
And then I'll use your algorithm to generate a response.
YUFENG GUO: All right, sounds good.
We'll call this the Justin Assistant.
JUSTIN ZHAO: That's perfect.
YUFENG GUO: All right.
OK, Justin.
What's the weather like next week?
JUSTIN ZHAO: Hi, Yufeng.
Sunday, it'll be 66 degrees and partly cloudy.
Monday, it will be 63 degrees and cloudy.
Tuesday, it'll be 66 degrees and partly cloudy.
Wednesday, it'll be 68 degrees and cloudy.
Thursday--
YUFENG GUO: Oh, boy.
OK, that's getting too long and just too robotic.
Yeah, let let's call it.
Let's call it at that.
JUSTIN ZHAO: Yeah, even saying it, for me,
felt a little strange.
YUFENG GUO: Yeah.
So clearly, generating natural language from structured data
is non-trivial.
How would you actually go about using a computer system
to answer the user's question then?
JUSTIN ZHAO: Well, first, I would
want to think about how I would answer it as a human.
So as a human, I would hope that I'd
be a little more contextually aware.
And I would realize that there's actually
a lot of repetitive information in the data.
So I'd probably try to summarize it,
something like, it'll be cloudy until Thursday
with showers the rest of the week.
Temperatures range from the high 40s to the mid-60s.
YUFENG GUO: Hey, you might want to consider
a career as a weather forecaster if, you know,
this whole research thing doesn't work out.
JUSTIN ZHAO: Yeah, maybe.
[LAUGHS]
YUFENG GUO: All right, so we've done
a little bit of an overview of natural language generation,
about what makes conversation natural.
And we even gave kind of a admittedly silly example
of leveraging the structure data to select content
for a natural language response.
JUSTIN ZHAO: Yeah, and we've also included some links
with more info in the video.
YUFENG GUO: That's right.
That we have.
All right, so then getting back to the topic at hand.
How does machine learning then get involved?
JUSTIN ZHAO: Well, that's the ultimate question that our team
is trying to answer.
Without machine learning, everything
that we've talked about so far, from parsing
the data to figuring out what to say to actually figuring out
how to say it, you have to do this
with writing lots of rules.
And rules are great.
They're very stable.
They're very predictable.
But they're usually very specific.
And they require a lot of engineering.
And because of that, it's not really scalable to new inputs
and outputs.
For example, if we wanted to talk about finance instead
of weather, or if we wanted to support an entire new language
altogether, it would require writing
a whole new set of rules.
YUFENG GUO: Yeah, and it sounds like that
would be way harder to maintain as well,
keeping all those rules lined up as things change.
And it would also be hard to replicate that
creativity and spontaneity that comes with human conversation.
JUSTIN ZHAO: Right.
So that's exactly one of the motivations of our research.
Our hope is that by giving the model examples
of data and the language it needs to generate,
we can let the model form its own rules about what to do.
And not only does this save us from having
to write these rules ourselves by hand,
but it also gives the computer more free rein
to be creative in its own way.
YUFENG GUO: So showing many examples to answer questions,
you might say, so that you can write fewer rules.
I mean, that's the crux of machine learning as a whole.
That's wonderful.
JUSTIN ZHAO: Yeah, exactly.
YUFENG GUO: And so what kind of machine learning architectures
then are you guys exploring to try to tackle this problem?
JUSTIN ZHAO: Well, so far, we've seen really promising results
with recurrent neural networks.
But that's just one kind of neuro-architecture
that we're exploring.
YUFENG GUO: OK.
Recurrent neural networks.
So on our previous episode, we looked at deep neural networks
on the show.
And that had neurons connected in layers,
resulting in something in a lattice structure.
Right?
And for our viewers, can you explain
what it means to have a recurrent neural network?
JUSTIN ZHAO: Yeah, so you can think
of a recurrent neural network as a deep neural network,
but just wrapped in a For loop.
And the network is recurrent because the outputs
of the network feed back into itself.
And instead of this one shot input-output,
the model can make decisions over several time steps.
YUFENG GUO: OK.
Awesome.
That's a really great way to conceptualize it.
I really love that.
And we've also included some links
about recurrent neural networks down below.
And if you have more questions about this network structure,
feel free to leave them below in the comments,
and we'll try to get to them.
For now, we'll talk about why recurrent neural networks will
be useful for doing natural language generation.
JUSTIN ZHAO: Right, so it's point to keep
in mind that language, just in general,
is extremely sequential.
YUFENG GUO: Sure yeah.
JUSTIN ZHAO: For example, the cat sat on the mat
is a very different sentence from cat sat the mat on.
YUFENG GUO: Yeah, order matters.
Definitely.
JUSTIN ZHAO: So RNNs are especially
good at remembering what it saw earlier,
because it enforces a sequential policy over the data.
The inputs are decided in a very ordered manner instead
of in these large conglomerates.
YUFENG GUO: OK.
So I guess it's both amazing and not entirely surprising
that recurrent neural nets would be useful for natural language
problems, it sounds like, where, as humans, we
rely a lot on what we previously said to figure out
what we will say next.
JUSTIN ZHAO: Mm-hmm.
exactly.
YUFENG GUO: So let's talk a bit more then on how
you're using these recurrent neural
nets to generate this language.
JUSTIN ZHAO: So one fun variation
when it comes to recurrent neural nets
is that since the output is generated one step at a time,
you can choose the granularity of your output.
So some models can choose courser outputs,
like entire word phrases, or just words in general.
And then this goes all the way down to models
that output bytes, single bytes at a time.
YUFENG GUO: One byte at time, OK.
JUSTIN ZHAO: And for us, we've been using outputs
at the character level.
YUFENG GUO: OK.
So you're, like, spelling out the words.
JUSTIN ZHAO: Right.
YUFENG GUO: OK.
JUSTIN ZHAO: And this kind of model is a character-based RNN.
And you can find out more information in the links below.
YUFENG GUO: So when we first talked
about having you on the show, you
showed me this interesting graph here.
JUSTIN ZHAO: Right.
YUFENG GUO: I would love to understand it a little better.
What is it showing us exactly?
JUSTIN ZHAO: So this is a small visualization
of our recent research.
Each row here represents different pieces
of our structured data.
YUFENG GUO: Gotcha.
JUSTIN ZHAO: The shading of the squares
indicates how much the model actually
cares about that piece of structured data.
And lastly, each column represents a single step
in our model.
So as we travel across the columns,
you can see how the model has learned
to pay attention to the structured data
at different time steps.
YUFENG GUO: OK.
So we're traveling left to right, character by character,
for each column.
And so the lit portions, the lighter parts,
are the parts that the model is paying attention to.
JUSTIN ZHAO: Right, exactly.
YUFENG GUO: OK, and then on this model over here, for example,
it means the model is paying attention
to this bit to decide what character to output.
It's not saying that that's the character it'll say.
That's just the data it's looking at?
JUSTIN ZHAO: Right, exactly.
So it's going to look at that particular piece of data
to try to figure out what character to output.
Exactly.
YUFENG GUO: All right.
JUSTIN ZHAO: And then one really cool
result is this diagonal line in the middle.
YUFENG GUO: Yeah, how about that?
It's kind of formulaic.
It almost looks like you guys added that in afterwards
to make for something interesting.
JUSTIN ZHAO: It's like hardcoded.
YUFENG GUO: Yeah.
JUSTIN ZHAO: So those particular pieces of data
are basically the characters for a specific location.
And what that diagonal line is showing
us is that when the model has reached
the part of the sentence where it wants to spell out
the specific location, it's learned
to read that from the data, character by character.
YUFENG GUO: Wow.
That is awesome.
And no one taught the model to do that.
They were just able to learn how to do that
just by looking at examples.
JUSTIN ZHAO: Exactly.
That's the magic of it.
YUFENG GUO: Incredible.
That's super outstanding, yeah.
JUSTIN ZHAO: So the diagonal line is pretty cool.
But if you dive into our data, there's
actually a lot of other intriguing ways
that the model learns by itself how
to reference the data to decide what character to output.
So that said, there's still a ton to explore.
But I am super excited to see what
we come up with in the future and how far we
can push our research.
YUFENG GUO: This looks super cool, Justin.
And I'm really excited to hear about what your team comes up
with next.
Maybe you'll write a research paper
using one of these networks in the future.
JUSTIN ZHAO: Yeah, that sounds pretty fun.
YUFENG GUO: Justin, I want to thank
you so much for coming into the studio
today and teaching our viewers about natural language
generation.
Looking forward to catching up again in a minute.
I'm going to wrap up here.
JUSTIN ZHAO: Yeah, OK.
Sounds good.
It was my pleasure.
YUFENG GUO: All right.
Sweet.
Well, I hope you enjoyed this episode of "AI adventures."
I certainly did.
In our conversation, we talked about using machine learning
for natural language generation and its role
in conversational user interfaces.
I had a blast chatting with Justin.
And if you like this format, please, let
us know in the comments below.
And for more information and details about everything
that we talked about, we've included tons
of links in the description.
And be sure to subscribe to the channel
to catch future episodes and maybe some more interviews
as they come out.
[MUSIC PLAYING]