Hi everyone, I'm Dan Taylor, a product architect working on
Azure AI Studio.
In this video, I'll be showing how you can build your own
copilot in Azure AI Studio with custom Python code by using the
Azure AI SDK and CLI in our hosted VS Code web development
environment.
Our goal is to build an API that grounds your copilot and your
enterprise data, and to do that we'll be creating a chat
function that will take a list of chat messages from the user,
generate A vector representation of the user's question using an
Ada embedding model from Azure Open AI service.
We'll then use this vector to perform a vector search on an
Azure AI Search index to retrieve relevant documents, and
then we'll use those documents to generate a prompt that we
send to a GPT 35 Turbo model hosted in Azure Open AI service.
We'll deploy this API to a managed online endpoint so that
it can be consumed by a web application which is outside of
the scope of this video.
And then all of this will be hosted in an Azure AI Studio
project so that we have a single place to organize all the
components of our application.
In the video, we're gonna be taking the following steps to
start from complete scratch without skipping anything.
To deploy this application, first we will go to Azure AI
Studio to create an Azure AI resource and project.
We'll then open VS Code Web and clone a sample app that we'll
start with.
Then we'll move on to using the Azure AI CLI to create the model
deployments in the Azure AI Search index, and then test the
model deployments and search index to make sure they're
working.
And finally, generate environment variables that our
code can use to connect to the Azure AI resources.
Then we'll switch over to the Azure AI SDK, where we'll use
the SDK to run and evaluate the chat function locally to make
sure it's working well.
We'll then improve the prompt used in the chat function and
evaluate how well those improvements are improving the
quality of our Copilot responses.
We'll then deploy the chat function to an API, and then
we'll just invoke the API and get a streaming Jason response
as an example.
All right, we're going to get started building everything
starting from scratch from ai.azure.com.
Let's dive right in.
Let's go ahead and head on over to ai.azure.com.
From the homepage, we'll click the Build your Own Copilot
button to create a new project.
Here we'll specify a project name
and we'll select to create a new Azure AI resource to host
our project.
When creating the resource, we'll just pick our subscription
and the location that we want to host our resources in, which
will be E us.
Now we'll go ahead and wait for this project to be created.
This project is connected to an Azure AI resource which
organizes the compute data and models that we'll need for our
solution.
A resource can have multiple projects connected to it, which
allows you to share your cloud resources amongst different
members of your team.
Now that we've created our project, we'll go ahead and get
started coding by opening the project in VS Code Web.
First, we'll need to create a compute that VS Code Web runs
on, so we'll go ahead and click Create here.
Now that our compute is created, we'll set up an environment for
VS Code to run in on this project.
Note that you can have different environments and different
projects running on the same compute.
The environment is basically a container that is available for
VS Code to use for working within this project.
Now that our environment is ready, we'll go ahead and launch
VS Code Web, which will open up in a new browser tab.
When VS Code opens, we get a helpful read me telling us about
how to use this VS Code Web environment.
In particular, I want to point out that there is a personal
code folder for us to use for our own personal work for
cloning git repos, and a shared folder that has files that
everyone that is connected to this project can see.
So if we have shared assets or data that we want to share with
other people, we can go ahead and move it to that shared
folder.
And if we created any prompt flows in our AI Studio project,
they would also show up here in the shared folder as well.
First, we just need to allow VS Code to trust the code in this
repo.
We'll press CTRL Shift` to open the terminal and we'll go ahead
and clone the sample repository that we have in the read me
here.
So we'll CD into our code folder and we'll go ahead and clone
that sample repo and then we'll CD into that sample repo.
We can see that there is a helpful README that comes with
the sample repository which contains instructions that we
can use to install packages and create our application.
First, we'll follow the instructions to create a virtual
environment for installing packages, and then we'll install
the requirements dot TXT file which contains the Azure AI SDK,
including the generative package for running evaluation, building
indexes and using Prompt Flow.
And when the packages are finished installing, we can go
ahead and follow the remaining steps in the README file.
We will run the AI init command which will create and configure
everything we need for our code to run.
The first thing that the CLI prompts us to do is it helps us
log in and authenticate with the Azure CLI.
So we'll see that this supports 2 factor authentication
and then we can go ahead.
Once this login is completed, we can go back to VS Code Web and
continue the steps from there.
I can now pick my Azure subscription to use, select the
project that we just created, and now we'll be able to create
the models from the Azure Open AI service that we need.
So first we're going to select a chat model.
Let's go ahead and use the GPT 35 Turbo 16K model.
It's created a default name for us to use for that deployment.
Now we want to select our embeddings deployment, which
will be used to vectorize the data from the users, and again,
we'll just select the default name.
And then for evaluation we'll go ahead and also use the GPT 35
Turbo 16K model, although GPT 4 is also a good choice for better
results.
We're going to create a new Azure AI Search resource here to
host our vector index, and we'll just pick the location E US.
We want that in the same place as our project, and we'll go
ahead and use the same resource group that was generated for our
Azure AI Search resource.
And again, we'll just pick the name that the CLI suggested for
us here for the Azure AI Search resource.
And now the CLI will wire up all of our choices and
configurations so that they can be used by our code.
First it outputs this config dot Jason file which is used to
point the sample repo at the project that we created.
And now we'll also run the AI search command to create our
search index by ingesting the markdown files that are in the
sample repository.
So here in the data folder we have all these product catalog
informations for this company, Contoso Trek, which is product
information for our enterprise that we want to use to ground
our copilot in.
And so now that we've gone ahead and created that vector search
index before we start running custom code, we can use the
actual AI chat feature to use the built in chat with data
capabilities of the Azure AI Studio just to make sure that
everything's working properly.
And so we can ask a question like what tent is the most
waterproof of the Assistant And we can see if the built in chat
with data capabilities are working.
And we can see that that is now retrieving information from
these markdown files that we just ingested.
Awesome.
So now that we can tell our resources are all set up
correctly, now we want to generate environment variables
for use with our code so we can use the AI dev new command to
generate a N file and so this N file contains all of the
configurations that we just set up with the AI init command and
wired that up to our code.
If you're a developer, you may be used to creating these N
files manually, which is a lot of tedious work.
And the CLI has automated all of that for us, which is great.
And now that we've configured our environment for our code to
run, we can move on to running the sample Copilot application.
So first, we're just going to copy this command here, which
invokes the run dot PY file.
This run dot PY file contains a reference implementation that
shows how to use the Azure AI SDK to implement the Copilot.
So you can see that we asked that same question that we just
ran from the CLI and we can see that in response we got the
answer to the question that which tent has the highest
waterproof rating?
And we also can see the retrieved documents that were
returned from our Azure AI Search index.
Let's go ahead and take a peek at this run dot PY file to see
how this works.
So we can see at the top of the run dot PY file we load that end
file that was just output by the Azure AI CLI and if we go to the
bottom of the file we can see here what we do is we just
simply take the question that was passed on the command line
and we run this chat completion function with the question as a
single message from the user.
Now if we go and look at the implementation of this chat
completion function in the sample repo, we can see it's
actually pretty straightforward.
First we take the list of messages from the user.
We get the last message in the conversation and we pass that to
the getdocuments function.
The get documents function then uses the Open AI SDK to 1st
embed the user's question into this vector embedding and then
we use the Azure AI Search SDK to then do a vector search using
that vector query and return the documents back to the user.
Once we have those documents we then generate the prompts.
The prompt is generated using a Jinja template, which we can
take a look at here.
That Jinja template contains the instructions to the Azure
Opening service model and contains the list of documents
and then we return the response back to the user.
It's that simple.
Now you'll notice that this does pretty much the same thing that
the built in chat with data capability did.
However, since it's in custom Python code, we can do whatever
we want here to add calls out to our own APIs, calls out to
retrieve documents from other data stores.
We can customize this however we want.
We can change the prompts.
We can change the parameters.
We can really make this our own, which is going to be especially
important as we want to take this Copilot to production.
And Speaking of production, one critical part of getting your
Copilot to be production ready is to evaluate and improve the
quality of responses that you get with the Copilot.
So we're going to move on to evaluating this Copilot next.
And for that we're going to use this evaluation data set which
contains a bunch of example questions and answers that we
can use to see how well the Copilot is performing.
So to run this, we can go ahead and use the evaluation command
in the run dot PY file.
We can say dash, dash, evaluate.
And while that's running, let's go ahead and look at that
implementation of the evaluation.
This imports the evaluate function from the Azure AI
generative SDK package and it loads that sample data set that
we are just looking at.
And then it runs the evaluation call which takes the chat
function as the target with the data set and generates a set of
GPT assisted metrics to evaluate the quality.
We can see in the output here that for each question we get
the answer as well as the metrics in this nice table
format.
We also can see the evaluation output as Jason files, so we can
look at.
Let's just turn on word wrapping.
Here we can look at for each question, what was the answer
that came back from the copilot and what were the retrieved
documents as well as a score.
So we can use this sort of the debug the answers, but we can
also use this URL here to open the evaluation results in Azure
AI Studio where we can get a nice visual of all of the inputs
and outputs and we can use this to debug and improve.
So here we can see the distribution of scores.
So here we're calculating a standard set of GPT assisted
metrics that help us understand how well is the copilot's
response grounded in the information from the retrieved
documents.
There we've got a score of four, point O 8.
We can see how relevant is the answer to the user's question.
Here we've got an average score of 4.38 and overall how coherent
and natural is the response of the generated text.
And here we get a score of four point O 8.
We can look at the individual rows and questions that are
being asked, so we can see for each question what is the answer
provided, what were the retrieved documents in this nice
visual here and also the provided truth answer.
So as we Scroll down here, we can see that the brand for the
trail Master tent the this score is low with ones we can see that
the answer, the copilot didn't even attempt to answer the
question.
So that's maybe one question that we want to be able to
improve the answer on.
So let's go back and see if we can improve the prompt using our
Copilot here.
We've had a teammate who's really good at prompt
engineering come up with a nice, safe and responsible and helpful
prompt.
And we're just going to update that prompt and rerun evaluation
from the command line.
We're going to say dash evaluate.
And this time we're going to give it evaluation name of
improved prompt so that we can easily keep track of this
evaluation result when we go back to the studio.
All right, now that that evaluation has completed, we can
go back to the studio here.
This time we'll just look at our list of evaluations.
We can see the history of those evaluations.
So we can always go back to previous results.
Here we're going to click the two evaluations and then we're
going to compare.
When we compare, we can see that we can see that the scores with
this new prompt have improved significantly overall.
And then we can for each of these responses see the
differences in the two Co pilots.
And as we Scroll down, we can see that in this case that we
are actually able to get a better, better answer to that
question of what the brand is for the Trail Master tent.
Now let's go ahead and deploy this Copilot to an endpoint so
that it can be consumed by an external application or website.
To do that, we'll go ahead and run the deploy command in the
run dot PY and we'll actually provide a deployment name here,
Copilot SDK Deployment.
So again, this just uses the Azure AI generative SDK to
deploy the code in this folder to an endpoint in our Azure AI
Studio project.
So we can see that it just uses the Deployment class here.
That takes the current folder, the conda dot yaml, as a set of
packages to install, and it points to the chat function that
we just looked at.
As well as setting up all of the environment variables, including
grabbing secrets from our project so that when the code is
deployed in a production environment, it runs just the
same as it does when running locally.
So that endpoint has been created and the source code has
been uploaded.
Now we'll just wait for our endpoint and deployment to be
ready to use.
Now that our deployment has completed, we can run the invoke
command on run dot PY and here we can specify the deployment
name.
This will return the response as a full Jason BLOB.
Here we can see the answer as well as those retrieved
documents.
We can also specify the stream command which will return the
response in small individual pieces which can be used by a
interactive web browser to show the answer as it's coming back
in individual characters and those characters are visible in
the content parameter of each row of the Jason response.
That was a quick tour of how you can build your own custom
copilot using code with Azure AI Studio.
Be sure to head on over to ai.azure.com to get started
today, and you can also check out some of the resources on the
screen to deep dive into more information.
Thank you for watching.