(soft music)
- Today we take a look at how you can bring
OpenAI's ChatGPT model in Azure
to your own enterprise-grade app experiences,
so that you can interact
with your organization's private internal data,
while respecting the information protection controls
that you have in place, and along the way,
we'll deconstruct how it all works
with a breakdown of ChatGPT prompts.
And joining me again on the show to go hands-on with ChatGPT
is Microsoft Distinguished Engineer, Pablo Castro.
Welcome!
- Thanks Jeremy, it's great to be back!
- And it's great to have you back on the show.
Now before we go hands-on with all the tech,
it's worth mentioning
that since our last show on the topic,
the Azure OpenAI service, it's now generally available.
So, this is the service that gives you programmatic access
to OpenAI large language models to use with your own apps.
The GPT model from OpenAI in the Azure service
now adds support for chat interactions.
In fact, if you missed our last show with Pablo,
it's worth checking out at aka.ms/OpenAIMechanics
to learn more about the fundamentals of building prompts
that guide the output of the OpenAI models
as you build app experiences.
Now, ChatGPT is of course
one of the fastest adopted technologies in recent years.
And at Microsoft, we're in fact integrating OpenAI models
with related experiences across the Bing search service,
GitHub Copilot for AI generated code,
and as recently highlighted,
the Microsoft 365 portfolio of apps with Copilot,
just to name a few.
So Pablo, what potential do you see then
in terms of applying ChatGPT
to enterprise-grade applications on the Azure service?
- Well, it's pretty exciting.
We can now build applications
that combine the ChatGPT model with your own data.
This can transform not only the way we interact with apps
but also our ability to effectively use vast amounts of data
to answer questions, generate content,
or any number of new and emerging tasks.
ChatGPT is a new model
optimized for conversational-style interaction,
though it can be used for other tasks as well.
It uses a particular convention and syntax
to denote turns between the user
and ChatGPT as "assistant" in a conversation.
And while it's trained on public data,
we can construct prompts that include both instructions
and additional data to generate responses.
So, imagine taking ChatGPT and applying it to your own data
but with precise control over the knowledge base
for in-context and relevant responses.
We can do that using an approach
often called "Retrieval Augmented Generation".
In this case, we combine the Azure OpenAI service
with Azure Cognitive Search
to index and retrieve data of all kinds,
knowledge that is private
and external to the ChatGPT large language model.
The retrieval step in Azure Cognitive Search
finds the most relevant pieces of information,
even if it's millions of documents or data points
and presents the top ranked results to the language model,
and this lets you have detailed
informed interactions with your data.
And because the knowledge
lives outside of the ChatGPT model,
you're in control of it,
and it's not used to train the model.
And equally important from an enterprise perspective,
any chat session state
lives entirely within your application.
And whether you keep it or not and where
is fully up to you.
- Just to clarify, by "private knowledge",
we mean only data that exists within your organization
or your application's boundary.
- That's right, or equally you could be a SaaS vendor
that wants to enlighten your application
to provide an in-context conversation
or content generation experience for your customers
by using the data you manage for them.
So, you're just using
the large language model's understanding
and reasoning capabilities
and building your own app experience around it.
- That makes a lot of sense.
So now we've had all the context,
so can you show us an example
of how this would work in an app?
- Sure, we'll walk through a typical app experience
that you can build around ChatGPT.
I have here a sample Human Resources web application.
By the way, we've published the code for this whole app,
including the UX, the backend, and the sample data
in a public GitHub repo at aka.ms/entGPTsearch
for you to try it out
or use as a starting point for your own apps
once you've watched the show.
So, this app lets employees generally chat about topics
related to their employment benefits and employee handbook.
In this case, I want to ask about healthcare coverage,
which can be unique by plan, location, individual, etc.
So, I'll type "Does my plan cover annual eye exams?"
In the generated response,
you will see that not only does ChatGPT
use the knowledge necessary to derive a response
but as a best practice also cites its sources.
That's because a key area we are actively exploring
is how do we make responses trustworthy?
These models are not perfect,
so we see this as a collaboration
between the user and the app
where the app reads through millions of data points
and picks a few to formulate an answer.
In this case it showed the source it used for the facts,
enabling the user to validate the response generated
from the "Benefit Options PDF" source if needed.
As I mentioned earlier,
we achieve this by coordinating the Azure OpenAI model,
Cognitive Search, and how we pre-process the data.
Now, I'll type a follow-up question.
"How about hearing?"
While the question in isolation wouldn't make sense,
in the context of the chat history,
it can figure out what information it needs to answer.
For each response,
I can see the preview of the file from the citation.
And it also shows all the supported content
used to formulate the response.
And so, in a way, what we're doing here
is we're conditioning ChatGPT
to produce source citations in its response,
which then helps you to validate them.
So, can you explain how then
this was able to figure out the response?
- Sure, we wrote this app, so it would expose the details
of what happens in each turn of the conversation.
Let's go back to our follow-up question,
"How about hearing?"
which, as I mentioned on its own,
doesn't present enough context to be answered in isolation.
When I click on this light bulb,
I can see the process it went through to provide a response.
Here we can see that GPT first takes the chat history
and the last question to produce a good search query.
You can also see the rest of the prompt,
including some of the mark up
used to tell ChatGPT where turns are.
By the way, there's also a new API
that has more structure around this,
so you don't have to construct the prompts
with markup manually.
Finally, you can see the "sources" part,
which is where we inject
the fragments of documents we recalled
from the search index.
And here's the final prompt we send to ChatGPT
to generate a complete response.
By the way, here we used the "Retrieve-Then-Read" approach
for generating responses.
This is an easy-to-understand approach
that can be effective in simple cases.
That said, we've explored other approaches
we include in the app that you can experiment with.
For example, "Read-Retrieve-Read"
would present the model with a question
and a list of tools it could select from,
such as "search the knowledge base"
or "lookup employee data".
So here, it would decide to search the knowledge base
to find out more about the healthcare plan,
it would see that there are different plans,
and so to determine which plan applies,
it then would look up the employee information
using another tool to finally arrive at an answer.
Another approach is what we call "Read-Decompose-Ask"
which would follow this
"chain-of-thought" style of prompting.
This would break down the question
into individual steps of the thought process
and answer intermediate sub-questions
to accumulate partial responses
until it arrived at a complete answer
that it would be ready to send back to the user.
- What I really like about this app sample
is it really helps you to deconstruct the fundamentals
for building these types of app experiences.
To that point though,
with the logic included into this template,
presumably you can tweak the prompts more, right?
- You can.
Beyond the user experience,
the template lets you easily experiment
and configure exactly how responses are generated
and channel what a user would expect to see.
As a developer, you can include additional instructions
using a prompt override.
For example, this can influence the tone
or style of a response.
So, I can change the style of the response.
Just for fun, I'll make it to answer like a pirate.
I'll copy this question
and start a new session using this override
and paste the question again.
And once the new response is generated,
you can see it responds like a pirate might, I guess.
In all seriousness, this is very powerful
because you can easily adapt the style of response.
For example, you might want to format the responses
to be more concise for a mobile device or add structure.
One of my favorite uses of this
is to have it generate a response
that can use formatting
to better organize the information if requested.
I'll paste some text to inject a few more instructions
into the prompt to do that.
Let me start another chat session and this time,
I'll use this suggested question
to compare two healthcare plans.
You'll see it gives me this long answer at first.
And now I can ask it to summarize the response in a table
and it gives me this nice table comparing the two plans.
These are just a few examples.
You could do any number of things,
even switching responses
to a person's native spoken language on the fly,
it's up to you
and the application experience you want to create.
- So, these stylistic and near real-time format changes
are a big part of GPT
and can also help make information a lot more accessible.
Switching gears though a bit,
I know a lot of people are probably wondering
how information protection in this case works.
So, can we talk about how you might make sure
that the information that's surfaced
is indeed what that user is allowed to see?
- So, since the data of the knowledge base
is in an Azure Cognitive Search index,
there are a variety of building blocks
for security and filtering you can use for access control,
like implementing document-level granular access control.
So, let's demonstrate this together.
In this case, let's say our organization
is working on an office move
to optimize collaboration between teams.
Information about the project
is restricted to only those of us involved.
I'll type "Is there a move or office space change coming?"
And the generated response says, "I don't know."
Because there are no documents I can see
that talk about this.
The information available in the sources
does not have an answer.
So Jeremy, why don't you try this out on your laptop?
- Sounds good!
So, here I have actually got the app open on my Surface.
I'll go ahead and type in the same question.
"Is there a move or office space change coming?"
And I can immediately see from the generated response
that there is a move indeed happening
and it's also cited the source: a plan for the office move
that's called "TheShuffleProject.pdf"
- Right.
So, the information is restricted to those involved for now.
I'm not part of it
but you're actively part of the project team,
so you have access to this information.
- Okay, so in this case,
you've actually written the solution
to make sure that only people can see
what they're allowed to see.
So, why don't we switch gears again though
and talk about the process
for adding new information into search.
So, how long does something like that take?
- Well, let's try it out.
This time I'll ask something a little bit more random,
like "Can I get scuba gear covered by my benefits?"
And you can see,
it found documents focused on benefits coverage
but says it doesn't have information
on scuba gear being covered.
I have this script here.
And in a real application,
this would be an automated process
that runs as data changes
but here I need to make it happen on the spot.
I'm going to manually run this script
and add the information on a new benefits plan
into our knowledge base.
So, now with the new content added,
I'll ask the same question again.
And you can see with the new knowledge available to it,
it generates an updated response.
- And just to be clear, we didn't speed anything up
in terms of using any Movie Magic.
The response actually reflects the change almost instantly
after Cognitive Search has access to the new information.
So to understand the logic a little bit better,
can you show us then the code
that's running behind your sample app?
- Sure.
As part of the sample, we included notebook versions
of the interesting parts of the backend.
The nice thing about showing this in a notebook
is that you can see how the state from a chat session
is passed from one prompt to the next.
Here we are in the notebook in Visual Studio Code.
You can see that everything is wired up to Azure services
with the right sources,
and managed identities to authenticate.
Then it create variables for parts of the prompts.
So now, let's run the whole thing to the end
for a first conversation turn.
And here we can see the output.
Now, let's do a second turn
but this time we'll do it step by step.
First let's check the history
and we can see the previous user question
and the response from the model.
Now I'll update the question in line with
"How about hearing?"
And it is first sent to GPT
to map the history and context and generate a search query.
From there it uses cognitive search
to run a query and get candidate documents.
Next, I'll pull up the content
and this was returned from our top search results.
And this is really the magic with ChatGPT.
We can see the prompt evolves with each interaction.
This pattern uses everything to construct a prompt:
the user question, the chat history, the search results,
all to make one big prompt with instructions.
Then in our final step,
it calls the Azure OpenAI completion API
to get a response based on the entire prompt.
You can see that the session history
is just kept in memory in this case.
The model itself doesn't track it
and it's up to your application.
You could choose to store it or, like we do here,
simply let it go when you close your session.
In either case, the session history
is not added to the large language model.
Now, I showed you some prompt experimentation
in the notebook.
Now, if you don't want to experiment using the notebook
or using the code,
you can also use the Azure OpenAI Studio Playground
to experiment with ChatGPT prompts interactively.
The GPT-3.5 Turbo model has been added to the playground
along with a new chat interface
and all the configuration parameters.
- And by the way, you can watch our entire show
at aka.ms/OpenAIMechanics
to learn about how to use Azure's OpenAI Studio.
Now, you also mentioned that this is based on
GPT-3.5 Turbo but with the release of GPT-4,
how does change the approaches that we've shown today?
- Indeed.
We were running the sample on GPT-3.5 today
because that's the model
that most people will have access to.
That said, everything we're discussing here
applies to GPT-4 as well
where some scenarios will perform similarly
and others will work much better,
thanks to its advanced reasoning capabilities
and a much prompt length limit.
- And apparently, it can also even pass the bar exam.
So, what else would you recommend then
for anyone who's watching who's looking to build out
their own enterprise-grade ChatGPT enabled apps?
- So first, try out the sample app I demonstrated earlier.
You can find it on GitHub at aka.ms/entGPTsearch.
The sample has everything you need to get started,
including creating the Azure services
and even the sample data we used.
In just a couple of hours, you can have a version
of what I showed you running with your own data.
Then, to learn more about the Azure OpenAI service,
you can go to aka.ms/azure-openai.
And for Azure Cognitive Search, check out aka.ms/azsearch.
- Pablo, it's always a pleasure having you on the show.
And also, don't forget to subscribe to Microsoft Mechanics,
the latest in tech updates.
Thanks for watching, we'll see you next time!
(soft music)