So imagine you're a journalist
and you want to write an article
on a specific topic.
Now you have a pretty good general idea about this topic,
but you'd like to do some more research.
So you go to your local library.
Now, this library has thousands of books
on multiple different topics.
But how do you know, as a journalist,
which books are relevant for your topic?
Well, you go to the librarian.
Now, the librarian is the expert on what books contain,
which information in the library.
So, our journalist queries the librarian to retrieve
books on certain topics.
And the librarian produces those books
and provides them back to the journalist.
Now, the librarian isn't the expert on writing the article,
and the journalist isn't the expert
on finding the most up-to-date and relevant information.
But with the combination of the two, we can get the job done.
Luv, this sounds like a lot like the process of RAG,
or Retrieval Augmented Generation,
where large language models call on vector databases
to provide key sources of data and information
to answer a question.
I'm not seeing the connection.
Can you help me understand a little bit better?
Sure.
So we have a user.
In your scenario, it's that journalist.
And they have a question.
So what types of questions would you want to ask?
Maybe we can make this more of a business context.
Yeah, so let's say this is a business analyst.
And let's say they want to ask,
"What was revenue in Q1 from customers in the northeast region?"
Right, so that's your prompt.
Okay, so a couple of questions on that user.
Does it have to be a person or could it be something else too.
Yeah. So this doesn't necessarily have to be a user.
It could be a bot
or it could be another application.
Even the question that we're talking about,
"What was our revenue in Q1 from the northeast?"
You know, the first part of that question,
it's pretty easy for, you know, a general LLM to understand, right?
What was our revenue?
But it's that second part in Q1 from customers in the northeast.
That's not something that LLMs are trained on, right.
It's very specific to our business and it changes over time.
So we have to treat those separately.
So how do we manage that part of the request?
Exactly.
You'll need multiple different sources of data potentially
to answer a specific question, right?
Whether that's maybe a PDF, or
another business application,
or maybe some images,
whatever that question is, we need the appropriate data
in order to provide the answer back.
What technology allows us to aggregate that data,
and use it for our LLM.
Yeah, so we can take this data
and we can put it into what we call a vector database.
a vector database is a mathematical representation
of structured and unstructured data
similar to what we might see in an array.
Gotcha, and these arrays are better suited or easier to understand
for machine learning or generative AI models
versus just that underlying unstructured data.
Exactly.
We query our vector database, right?
And we get back an embedding that includes
the relevant data for which we're prompting.
And then we include it back into the original prompt, right?
Yeah, exactly.
That feeds back into the prompt.
And then once we're at this point,
we move over to the other side of the equation,
which is a large language model.
Gotcha, so that that prompt,
that includes the vector embeddings now
are fed into the large language model,
which then produces the output
with the answer to our original question
with sourced, up-to-date and accurate data.
Exactly. And that's a crucial aspect of it.
As new data comes in to this vector database,
where things that are updated back to your relevant question
around performance in Q1,
as new data comes in, those embeddings are updated.
So when that question is asked the second time,
we have more relevant data in order to provide back
to the LLM who then generates the output in the answer.
OK, very cool.
So Shawn, this sounds a lot like my original analogy there
with the librarian and our journalist, right?
So the journalists trusts that the information in the library
is accurate and correct.
Now, one of the challenges that I see is when I'm talking to enterprise customers
is they're concerned about deploying this kind of technology
into customer facing, business critical applications.
So if they're building applications, taking customer orders,
processing refunds,
they're worried that these kinds of technologies
can produce hallucinations or inaccurate results, right?
Or perpetuate some kind of bias.
What are some things that can be done
to help mitigate some of these concerns?
That brings up a great point Luv.
Data that comes in on this side,
but also on this side,
is incredibly important to the output that we get
when we go to make that prompt and get that answer back.
So it really is true: "Garbage in and garbage out", right?
So we need to make sure we have good data that comes into the vector database.
We need to make sure that data is clean,
governed and managed properly.
Gotcha, so what I'm hearing is
that things like governance
and data management
are of course crucial to the vector database, right?
So making sure that the actual information that's flowing through into the model,
such as the business results in the sample prompt we talked about
is governance and clean,
but also crucially,
on the large language model side,
we need to make sure that we're not using a
large language model that takes a black box approach, right?
So, a model where you don't actually know
what is the underlying data that went into training it, right?
You don't know if there's any intellectual property in there.
You don't know if there's inaccuracies in there
or you don't know if there are, pieces of data
that will end up perpetuating bias in your output results.
Right?
So as a business,
and as a business that's trying to
manage and uphold their brand reputation,
it's absolutely critical to make sure that we're taking an approach
that uses LLMs that are transparent in how they were trained,
and we can be 100% certain
that there aren't any inaccuracies
or data that's not supposed to be in there, right?
Yeah, exactly.
It's incredibly important, especially as a brand, that we get the right answers.
We've seen the results of impact, and especially back
to our original question around "what was our revenue in Q1", right?
We don't want that to be impacted by the results of a question
that comes from, you know, that prompts one of our LLMs.
Exactly, exactly.
So very powerful technology.
But it makes me think back to the the library.
Our journalist and librarian, they both trust the data and the books that are in the library.
We have to have that same kind of confidence
when we're building out these types of generative AI use cases for business as well.
Exactly, Luv.
So governance, AI, but also data and data management
are incredibly important to this process.
We need all three in order to get the best result.