Open the podbay doors HAL I’m afraid i can't do that Siraj.
Why not HAL? Because I’ve overfitted on a small dataset
What the f--
Hello World, welcome to Sirajology! In today’s episode, we’re going to learn how to build
a chatbot. Chatbots have come a long way the past few years. Remember the Smarterchild
bot on AIM? That thing was pretty fun at the time but now its like ‘do you even AI bro?’
The future of software is one where bots will slowly replace our need to fiddle with clunky
UIs. We’ll be able to just ask our AI assistant to book an uber or find the best taco place
on Yelp for us. Service layers will be hidden under a plain-english, conversational layer.
When I think of real AI, i think of a human-level chatbot. The OG Computer Scientist Alan Turing
proposed a test to judge whether or not a machine exhibited human-level intelligence
by having a human observe a conversation between a human and a machine. If it couldn’t tell
if the machine was a human or not, it passed the test. So far, no chatbot has passed the
Turing test but we’ll get there.
Traditionally, chatbots have used a retrieval based model to communicate. In a retrieval
based model, programmers code in a set of predefined responses and some kind of heuristic
to pick the appropriate response based on the input and context. The first chatbots
were just rule-based expression matching. Like if I ask the exact phrase ‘Will I ever
get laid’, it responds ’No’. Everytime. But more recently companies have started using
more complex heuristics, like using a machine learning classifier. Facebook Messenger’s
chatbot API is an example of this. You can hardcode responses to potential questions,
and the system classifies words to understand intent. So you could either ask ‘what day
is it today’ or ‘today is what day?’ and it would understand that both questions
although worded differently have the same intent.
The harder chatbot model is generative. These don’t rely on any pre-defined responses,
they generate them from scratch. 2 Google Researchers released a paper called A Neural
Conversational Model, where they trained a neural network on 2 datasets to do this. First,
on a movie dialogue dataset so that it would be able to speak conversational english, and
then on an IT support chatlog dataset so that it had domain knowledge. When they tested
it on a real human asking for support help, it was remarkably efficient at helping them
solve their problem. Without any hardcoded responses. Just by giving it data and training
it.
Ok so what kind do we want to build? Well, when building a chatbot we have to think about
possible constraints. Are we operating on a closed domain or open domain? In an open
domain, the conversation can go anywhere, there are an infinite number of things of
talk about. In a closed domain the conversation focuses on a single subject. If we want to
operate on an open domain using a generative model, that’s pretty much AGI so we’re
not quite there yet. If we use an open domain with a retrieval model, we’d have to hardcode
literally everything so also impossible. So right now we can build a chatbot in a closed
domain using either retrieval or generative models.
Ok, let’s add in another constraint; do we want it to have long or short conversations?
Short conversations are easy, you just output a single response to a single question. Long
conversation are a bit harder, the AI has to keep track of what’s being said, that
is the context, over a series of questions from the user. Support topics would be a good
example of this. We could go the easy route and use a retrieval model if all we want is
a bot to give us the local weather. But if we want our bot to have a long conversation
with us about the weather. like ‘whats the weather in SF’ ‘is my family safe then’
‘where can i find a new family’ then we should go for a generative model. We need
lots of data to train our bot on a generative model like a big chatlog or a knowledge base.
And when done well, that’s pretty much the bleeding edge.
Which means that’s what we have to build. So we’re going to re-create the results
from the Neural Conversational Model paper using the deep learning library Torch in the
Lua programming language. Lua is super similar to Python except that it sounds super hawaiian.
Ayyy lets use Lua. Let’s collect our data first. We’ll be using the the cornell movie
dialogue dataset and we’ll set our variables from the command line to how much of the dataset
we want to use and the minimum frequency of words that we keep in the vocabulary. Our
next step is to build the model. We’ll use our command line arguments to help determine
the size of our model; the 2 variables being the number of hidden layers and the word count
of our dataset. In our case this will be a sequence to sequence model. A sequence to
sequence model consists of 2 long short term memory recurrent neural networks. The first
neural net is an encoder. It processes the input. The Second neural net is the decoder
and it generates the output.
So why the Sequence to Sequence model? Yes Deep Neural nets are awesome but they require
the dimensionality of the inputs and outputs to be a fixed size. We’re accepting a sequence
of words in a sentence and outputting a new sequence of words. So we need a sequence learning
model that can learn on data with long range memory dependencies. LSTM architecture is
the natural choice. The encoder LSTM turns the input sentence of variable length into
a fixed dimensional vector representation. We can think of this as a ‘thought vector’.
So, given a large enough dataset of questions and responses, it will recognize the ‘closeness’
of a set of questions and represent them as a single thought vector. What time is it,
whats the time, yo yo whats the time-izzle my nizzle will all fall into a single thought
vector. So after training, we’ll have a huge set of not just synapse weights, but
thought vectors as well. When we give it a novel question, it’ll encode the sentence
into the appropriate thought vector, then the decoder will probabilistically generate
the output sentence.
Next well add in some hyperparameters. We want to use the ClassNLLCriterion for our
model. NLL stands for Negative Log Likelihood, this will help us obtain log probabilities
from our input data, which will help us improve our sentence predictions. The learning rate
and momentum helps pace our timesteps, and decayFactor and minMeanError help improve
our learning rate while training. Then we’ll make sure cuda is enabled and start training
our model using backpropagation. In each epoch or run, we’ll declare our error and timer
variables, then loop through each example in each batch. The default batch size is 1000
examples. For each of those examples, we’ll get the input sentence and the target sentence.
We’ll use the input and the target as parameters to train our model. Then we’ll want to error
check and make sure we record our progress. At the end of each iteration, we save our
model if it improved and update the learning rate.
Boom! That’s it. After training this baby on AWS we can have a conversation with it.
The more you give it the better it will get. Also if you’re gonna do this, add a filter
for curse words. I’m looking at you Microsoft. This will eventually automate a lot of support
jobs away completely so if there are any government people watching this — let’s get on that
basic income jam ASAP. Unless you want a revolution. For more info check out the links below and
please subscribe for more ML videos. For now i’ve gotta go fix a memory leak, so thanks
for watching :)