SPEAKER: In this video, we'd like
to introduce how to use Google Cloud AI tools to quickly
bring the power of large language models
to enterprise systems.
Many people are now starting to think
about how to bring Gen AI and LLMs to production services
and facing with several challenges.
You may be wondering how to integrate LLMs or AI chat
bots with existing IT systems, databases, and business data.
We have thousands of products.
How can I let LLM memorize them all precisely?
Or how to handle the hallucination issues in AI chat
bots to build a reliable service?
Here is a quick solution-- grounding with embeddings
and vector search.
What is grounding?
What are embedding and vector search?
In this video, we will learn these crucial concepts
to build reliable Gen AI services for enterprise use.
But before we dive deeper, here is an example.
This demo is available as a public live demo.
Please open the demo with this QR code
or the URL on the video description.
Then select Stack Overflow, and enter
any coding question as a query.
So it runs a text search on 8 million questions
posted on Stack Overflow.
For example, if you ask, "How to shuffle rows in SQL?"
then it instantly shows Stack Overflow questions
that have similar meaning to your question.
The following points make this demo unique.
First, it is LLM-enabled semantic search.
The 8 million Stack Overflow questions and query text
are both interpreted by Vertex AI generative AI models.
The model understands the meaning and intent or semantics
of the text and code snippets in the question body
at librarian-level precision.
The demo leverages this ability for finding
highly-relevant questions and goes
far beyond simple keyword search in terms of user experience.
Second, grounded to business facts.
In this demo, we didn't try having
the LLM to memorize the 8 million items
with complex and lengthy prompt engineering.
Instead, we attached the Stack Overflow data set to the model
as an external memory using vector search
and used no-prompt engineering.
This means the outputs are all directly
grounded to the business facts, not the artificial output
from the.
So the demo is ready to be served today
as a production service with mission-critical business
responsibility.
It does not suffer from the limitation of LLM memory
or unexpected behaviors of LLMs, such as the hallucinations.
Third, scalable and fast.
The demo gives you the search results
in tens of milliseconds, while retaining
the deep semantic understanding capability.
Also, the demo is capable of scaling out
to handle thousands of search queries every second.
This is enabled with the combination of LLM embeddings
and Google AI's vector search technology.
There are two key enablers of this solution--
first, the embeddings generated with Vertex embeddings API.
Second, fast and scalable vector search by Vertex
AI matching Engine.
Let's start by taking a look at these technologies.
So what is embeddings?
With the rise of LLMs, why is it becoming important for IT
engineers and IT DMs to understand how they work?
In traditional IT systems, most data
is organized as structured or tabular data using
simple keywords, labels, and categories in databases
and search engines.
In contrast, AI-powered services arrange data into a simple data
structure known as embeddings.
Let's see how AI organizes data using embeddings.
Once trained with specific content
like text, images, or any content,
AI creates a space called embedding
space, which is essentially a map of the content's meaning.
AI can identify the location of each content on the map.
That's what embedding is.
Let's take an example where a text discusses movies, music,
and actors with a distribution of 10%, 2%, and 30%
respectively.
In this case, the AI can create an embedding
with three values--
0.1, 0.02, and 0.3 in three-dimensional space.
And AI can put content with similar meanings
closely together in the space.
Please note that this is simplified example
for the purpose of explanation.
In reality, embedding space may have hundreds or thousands
of dimensions that can represent almost infinite numbers
of nanocategories of the content.
According to the Machine Learning Crash Course,
embedding refers to the process of mapping
high-dimensional raw data, such as images and texts, which
have thousands or millions of dimensions
to a lower-dimensional space using deep learning models.
This is done to extract the meaning
or semantics of the data and create
a map of the contents meaning.
This is how Google organizes data
across various services like Google Search, YouTube,
Play, and many others to provide search results
and recommendations with relevant content.
Embeddings can also be used to represent
different types of things in businesses,
such as products, users, user activities, conversations,
music and videos, signals from IoT sensors, and so on.
AI and embeddings are now playing a crucial role
in creating a new way of human computer interaction.
AI organizes data into embeddings,
which represent what the user is looking for,
the meaning of contents, or many other things
you have in your business.
This creates a new level of user experience
that is becoming the new standard.
Now we have learned what is embeddings.
Next, let's see how we can use it with Vertex AI.
On May 10, 2023, Google Cloud announced Vertex AI Embeddings
API for text that is available on Vertex AI Model Garden.
This API is designed to extract embeddings from texts.
It can take text input up to 3,072 input tokens and output
768 dimensional text embeddings.
With the embedding API, you can apply the innervation
of embeddings combined with the LLM capability
to various text processing tasks such as LLM-enable semantic
search.
Text embeddings can be used to represent
both the meaning of texts in the embedding space.
Texts that have similar meaning to the user's query intent
will be found fast with vector search technology.
LLM-enabled text classification--
LLM text embeddings can be used for text classification
with a deep understanding of different contexts
without any training or fine-tuning.
LLM-enabled recommendation-- the text embedding
can be used for recommendation systems
as a strong feature for training recommendation
models such as two-tower model, resulting
in next-gen user experience with semantic product
recommendation.
LLM-enabled clustering, anomaly detection, sentiment analysis,
and more can be also handled with the LLM-level deep
semantics understanding.
Vertex AI embeddings for text has an embedding space
with 768 dimensions.
As explained earlier, the space represents a huge map
of a wide variety of texts in the world organized
by their meanings.
With each input text, the model can find a location
or embedding in the map.
The API can take 3,072 input tokens
so it can digest the overall meaning of a long text and even
programming code and represent it as single embedding.
It is like having a librarian knowledgeable
about a wide variety of industries,
reading through millions of texts
carefully and sorting them with millions of categories
that can classify even slight differences of subtle nuances.
By visualizing the embedding space,
you can actually observe how the model
sorts the texts at the librarian-level precision.
Nomic AI provides a platform called
Atlas for storing, visualizing, and interacting with embedding
spaces, and they work with Google
for visualizing the embedding space of the 8 million Stack
Overflow questions.
You can try exploring around the space,
zooming in and out to each data point
on your browser on this page, courtesy of Nomic AI.
Here are the examples of the librarian-level semantic
understanding by Embeddings API with Stack Overflow questions.
For example, the model thinks the question,
"Does moving the request line to a header frame
require an app change?"
is similar to the question, "Does an application developed
on HTTP 1.x require modifications
to run on HTTP 2?"
That is because the model knows both questions talk
about what's the change required to support the HTTP 2
header frame.
Note that this demo didn't require any training
or fine-tuning with computer programming specific data sets.
This is the innovative part of the zero-shot learning
capability of the LLM.
It can be applied to a wide variety of industries,
including finance, health care, retail, manufacturing,
construction, media, and more for deep semantic search
on the industry-focused business documents
without spending time and cost for collecting
industry-specific data sets and training models.
The second key enabler of the Stack Overflow
demo shown earlier is the vector search technology.
This is another innovation we are having in the data
science field.
The problem is how to find similar embeddings
in the embedding space.
Since embeddings are vectors, this
can be done by calculating the distance
or similarity between vectors.
But this isn't easy when you have millions or billions
of embeddings.
For example, if you have 8 million embeddings
with 768 dimensions, you would need
to repeat the calculation in the order of 8,000,000 x 768.
This would take a very long time to finish.
Actually, when we tried this on BigQuery
with 1 million embeddings five years ago, it took 20 seconds.
So the researchers have been studying
a technique called Approximate Nearest Neighbor
or ANN for faster search.
ANN uses vector quantization for separating the space
into multiple spaces with a tree structure.
This is similar to the index in relational databases
for improving the query performance,
enabling very fast and scalable search
with billions of embeddings.
With the rise of LLMs, the ANN is getting popular
quite rapidly, known as the vector search technology.
In 2020, Google research published a new ANN algorithm
called ScaNN.
It is considered one of the best ANN algorithms
in the industry-- also, the most important foundation
for search and recommendation in major Google services
such as Google Search, YouTube, and many others.
Google Cloud developers can take the full advantage
of Google's vector search technology
with Vertex AI Matching Engine.
With this fully-managed service, developers
can just add the embeddings to its index
and issue a search query with a key embedding
for the blazingly-fast vector search.
In the case of the Stack Overflow demo,
Matching Engine can find relevant questions
from 8 million embeddings in tens of milliseconds.
With Matching Engine, you don't need
to spend much time and money building your own vector search
service from scratch or using open source
tools if your goal is high scalability, availability,
and maintainability for production systems.
By combining the Embeddings API and Matching Engine,
you can use the embeddings to ground LLM outputs
to real business data with low latency.
For example, if a user asks a question,
Embeddings API can convert it to an embedding and issue
and query on Matching Engine to find similar embeddings
in its index.
Those embeddings represent the actual business data
in the databases.
The Stack Overflow demo architecture has two parts--
first, building a Matching Engine index with Vertex AI
Workbench and the Stack Overflow data set in BigQuery
on the right; and second, processing vector search
requests with Cloud Run on the left and Matching Engine.
In this video, we have seen how the combination of embeddings
for text API and Matching Engine allows
enterprises to use Gen AI and LLMs
in a grounded and reliable way.
The fine-grained semantic understanding capability
of the API can bring the intelligence
to information search and recommendation
in a wide variety of businesses, setting
a new standard of user experience
in enterprise IT systems.
To get started, please check out these resources.
The links are available on the video description.
If you're looking for semantic search solution
with less effort for the integration,
please also check General App Builder Enterprise Search.
The product provides an out-of-the-box document search
solution that integrates keyword search and semantic search
as a fully managed service.
Thank you so much for watching this.