[MUSIC PLAYING]
NIKITA NAMJOSHI: Hi, everyone.
I'm Nikita Namjoshi.
ZACK AKIL: And I'm Zack Akil.
NIKITA NAMJOSHI: As applied AI engineers,
we spend a lot of time building machine-learning applications
on Google Cloud.
But with such an extensive set of tools available,
it can be hard to know exactly how to use each of these tools
and what they're good for, even for us.
ZACK AKIL: So we decided to use machine learning
to make our jobs just a little bit more
easy by building a tool to help answer these kind of questions
for us.
For example--
NIKITA NAMJOSHI: --how many languages does the Translation
API support?
ZACK AKIL: Translation API Basic uses Google's neural machine
translation technology to instantly translate texts
into more than 100 languages.
Question-and-answering systems like these
are incredibly helpful for quickly extracting
valuable information from documents
that can sometimes be too difficult to read through
manually.
Let me give you an example.
Imagine you've just joined a new company,
and you've probably got a whole bunch of questions, questions
like how do I access the printers?
Or how do I get parking validated?
Or where's the bike storage?
Today, maybe that data is stored somewhere
within an internal wiki, but wouldn't it
be cool if you had access to a chat bot
that could find the answers in that wiki for you?
NIKITA NAMJOSHI: Today, we'll show you
how to build a question-answering system
on Google Cloud based on your data.
But first, let's start with a little bit of context
on two NLP concepts, embeddings and nearest neighbor search.
And don't worry if these concepts
are totally new to you.
You don't need to be an expert.
Embeddings are a way of representing data
as points in space where the locations are semantically
meaningful.
This data could be pretty much anything.
It could be text, images, videos, audio, users.
And nearest neighbor search is the technique
that you use to figure out how similar data points are.
One of the most famous embeddings
is word2vec, which was created by Google in 2013.
Word2vec takes as input a word, and it returns an N-dimensional
coordinate or vector.
If you plot these word vectors in space,
you'll notice that synonyms are clustered together.
You can see an example over here,
where similar words cluster together in space.
So we have peach near apple.
A little further away is pizza.
But then words about animals, like leopard and dog,
are clustered together a little further away.
Embedding single words is not that interesting.
But if we embedded entire articles,
we could create a function that, given one news article,
recommends semantically similar articles.
Or if we embedded movie plots, we
could recommend similar movies.
So how do we get from vectors to recommendations?
Well, if we compute the distance between two embedded data
points, we'll be able to get a sense of how similar they are.
Distance in this case is a trigonometric measure,
such as Euclidean distance or cosine distance.
And it turns out you can do a similar thing with questions
and answers, or, in our case, given
a question about Google Cloud, return an answer from the docs.
So to build this question-answering system,
we'll start by embedding paragraphs from the docs
and build up a database.
To answer a query, our system will
map the query to the embedding space and then
search among all of those database embeddings
to find the ones that are closest to the query.
Closest in this case means the best answer.
And, of course, to do this search,
we'll use the nearest neighbor algorithm.
So let's take a look at how we can use Google ML products
to help us build this system.
ZACK AKIL: To build our question-answering system,
we'll start with a set of reference documents.
Then we'll extract that text and split it into paragraphs,
and we'll use these as our answers.
We'll do this preprocessing within the Vertex AI Workbench
and store the results in Cloud Storage.
After we have our text chunks, we'll need to embed them.
Instead of training an embedding model from scratch,
we're actually going to use a pre-trained text-embedding
model directly from TensorFlow Hub.
We're actually going to use the universal sentence encoder
or use model, which encodes text into high-dimensional vectors
that can be used for text classification,
semantic similarity, clustering, and other language tasks.
This module actually contains two embedding models,
one for embedding the questions and one
for embedding the answers.
We're actually going to import both of these encoding models
to Vertex AI Model Registry, which
is a repository where you can manage
the lifecycle of all your ML models.
Now we're going to use these models to embed
a lot of text in our reference documents
alongside all of the questions that the users will
ask on the fly.
To run these prediction jobs, we're
going to use the Vertex API Prediction service.
And once all of our reference text is embedded as vectors,
we'll use the Vertex AI Matching Engine.
Now this will store those vectors
and perform nearest neighbors search on them at scale.
Then we'll use Firestore for storing
the text answers themselves, which is what
will get returned to the user.
To tie all of these pieces together,
we'll write a function in Python for handling user requests
and calling our ML system.
This code will be served in a Cloud Function.
Finally, we want to actually talk to our application.
So we're going to use the text-to-speech and
speech-to-text APIs to transcribe what we're saying
and have the system produce spoken responses.
NIKITA NAMJOSHI: All right, Zack,
should we build this thing?
ZACK AKIL: Let's do it.
NIKITA NAMJOSHI: All right, so our first step
is going to be to download the reference documents.
And we'll break them up into paragraph-sized text chunks.
We'll do all of this preprocessing
within a Vertex AI Workbench Manage notebook.
So here in the notebook, you can see some examples of the text
that we'll be embedding.
And we'll save these text chunks out
to a CSV file in Cloud Storage.
Next, we need to download our question-answering
sentence-encoder module from TensorFlow Hub.
You can see here in the hub that this module actually
contains two models, one to embed
the questions and the other to embed the answers.
And we can test this out in our notebook
and call the model right here locally.
So let's try that.
If we pass in the text, what's the meaning of life,
the universe, and everything, we'll see that what we get back
is this 512-dimensional embedding vector.
Now, these numbers don't mean a whole lot
to us, but to the use model, they
are semantically meaningful.
Now, we could use this model to embed text
right here in our notebook.
But because we have so many text snippets to embed,
this would take forever, and it would use a lot of memory.
Plus, even though we're going to embed all of our answers
in advance, we need to be able to embed questions
from users on the fly as they ask them.
This isn't really something we can do from a notebook.
So instead, we'll import our models to Vertex AI Model
Registry, and then we'll run prediction jobs
with Vertex AI Predictions.
So let's kick this off with the answer model.
We can use the Vertex AI Python SDK to upload the model.
So that's what we're doing right here in the notebook.
And to serve our model, we'll specify
the optimized TensorFlow Runtime Container
for running our prediction jobs, which leverages optimization
techniques used internally at Google for lower latency
predictions.
If we go to the Cloud Console, we'll
see that our model has been created here
in the model registry.
Now, we want the answer text to be embedded ahead of time.
So to do this, we'll start a batch prediction job.
We'll pass in the data that we saved earlier to a CSV file.
And there is a lot of data, so this could take a while.
But we'll receive an email when it completes.
So let's specify our hardware profile.
We can add a GPU.
We'll click Create.
And let's start this prediction job.
When the job is finished, we can go to Cloud Storage.
And we'll see that the output is the text alongside the vectors
themselves.
So here's the text.
And right after that, you can see the embeddings.
OK, our answers are ready to go.
Next, we need to upload our question model.
While we embedded the answers ahead of time,
we need to embed the questions on the fly
as our users ask them.
To get these embeddings with low latency,
we'll deploy our question-embedding model
to an endpoint.
And we can hit this endpoint like any other REST endpoint.
So that's what I'm doing in the notebook right here.
Deploying takes a few minutes, but once it's ready,
we'll see our model and the endpoint in the UI
alongside our answer model in the Model Registry.
Now that we have our database embeddings
and our models deployed, Zack is going
to show us how to implement some blazingly-fast and nearest
neighbor search on these embeddings.
ZACK AKIL: Thank you, Nikita, for embedding all of that data.
The way that we find the best answer for our question
is to do a similarity search on those vectors.
But it turns out, even though the similarity metric is just
a distance calculation, like .product,
for large data sets with hundreds of millions or even
billions of vectors, performing this nearest neighbor search is
a computationally challenging task and requires sophisticated
approximation algorithms if you want to do it quickly
and at scale.
However, thanks to cutting-edge Google research,
Vertex AI Matching Engine is a fully-managed solution
that does exactly that.
To create a Matching Engine instance,
we can run a script from our Vertex AI Workbench
notebook that will create a Cloud Storage
bucket to store our vectors.
We also have to configure the dimensions of the embeddings
that we're going to be searching through, which, in our case,
is going to be 512.
Next, we need to prepare our answer embeddings
to be uploaded into Matching Engine, which
expects a text file of dictionaries with keys, ID,
and embedding.
Here is an example of what the data looks just before it gets
ingested into Matching Engine.
Next, we'll upload that data into a Cloud Storage bucket
and kick off a script to tell Matching Engine to start
ingesting the embeddings.
You can see, within the configuration,
the main thing we have to declare
is the location of our storage, which is there.
Behind the scenes, Matching Engine
is building indexes so that our embeddings can be
queried quickly and at scale.
Here is our Matching Engine instances
within the Cloud Console.
And we can even see the indexes when they have actually
finished ingesting the content, and they're all
visible in the Console as well.
Finally, to query a Matching Engine,
we could pass it a single vector,
and it will return the IDs of all the nearest neighbor
vectors and their distances.
We need to be able to connect those IDs back
to the actual answer texts, so we've also created a Firestore
database to store the actual text of our answers
because, in the end, a user wants the actual answer text
and not just some random ID.
Here's what the answers look like within Firestore.
And you can see that, within Firestore, we have
the text string and the ID.
And if you're not familiar, Firestore
is an easy-to-use serverless database on Google Cloud.
In the end, these IDs are what's going
to connect the text back to the IDs
from the Vertex AI Matching Engine.
To tie all of these pieces together,
I've written a Python function that
first takes as input the user question text then
feeds that text into our model that's going to produce
an embedding on the Vertex AI.
That embedding is then fed into the Matching Engine,
which is going to return back all of the nearest neighbors.
And we're going to take the IDs from those nearest neighbors
and get the original text from Firestore,
and then, at the end, return back the original text back
to the user.
To get this whole system production ready,
I've just thrown all this code into a Cloud Function
so that we can head it as a REST endpoint.
We can call this REST endpoint from pretty much anywhere
we want, like a Jupyter Notebook, a web app, or even
a chat bot.
To round off this tool, we've added the speech-to-text API
and text-to-speech API so that we can literally
talk to our question-answering system.
Let's try it out.
NIKITA NAMJOSHI: What does Vertex AI Feature Store do?
SPEAKER: Vertex AI Feature Store provides
a centralized repository for organizing, storing,
and serving ML features.
ZACK AKIL: Cool.
So today, we showed you how you can take the best of Google ML
and create your own scalable questioning-answering system
to unlock valuable information from your unstructured data.
NIKITA NAMJOSHI: We hope you enjoyed our demo.
ZACK AKIL: Cheers.
Goodbye.
NIKITA NAMJOSHI: Bye.
[MUSIC PLAYING]