>> You're not going to want to miss
this episode of The AI show where we talk
about combining OpenAI models with the power of Azure.
New from build.
Make sure you tune in.
>> Hello and welcome to this episode of The AI Show,
where we're going to talk about combining
OpenAI models with the power of Azure with Chris.
My friend, how are you?
>> I'm doing great. Yeah, thanks for having me.
>> Fantastic. So tell us who you are and what you do.
>> Yeah. I'm Chris Hoder.
I am the Lead product manager on the Azure OpenAI Service,
and I'm excited here to talk about what we are building.
>> Let's talk about the Azure OpenAI service.
People probably have heard of OpenAI,
but now we're putting it together with Azure.
Can you tell us about what we announced at Build?
>> Yeah. This week at Build, we are really excited to announce
the limited access public preview of the Azure OpenAI Service.
What that means, is customers out there can
start to apply for access,
to be able to use OpenAI's models in Azure today
to use that for a huge number of language use cases.
>> That's really cool. But they could have used that in OpenAI,
what makes it doing it in Azure even better?
>> Yeah. What makes it even better in Azure,
is that we provide a lot of what is
often thought of as just the core goodness of Azure.
We have security with Azure
Active Directory based authentication for managed identities.
You can do private networking with
virtual networks and private links,
and a whole swath of other features that really empower
enterprises to take advantage
of this tech in a really secure way.
>> Interesting. If you want to try all the latest and
greatest OpenAI, still go there.
But if you're trying to put it into an enterprise setting,
you want to do it with Azure. Am I getting this right?
>> Yeah, exactly.
That's a great way to think about it.
>> Fantastic. I was hoping you could
show us a little bit about how it works.
>> Yeah, definitely. Let's hop into a quick demo.
>> All right, let's do it.
>> What you're seeing here is what we
call the Azure OpenAI Studio,
and this is our new UI that lets users
quickly iterate and experiment with their use case.
That becomes really important with
something like OpenAI's models,
because they are super flexible.
I want to show you a little bit about how flexible they are.
What we have here is a simple text interface
where I can copy in text, and call it against the model.
What's unique about these types of launch language models,
is you actually define the task with what you
put into the prompt and send to the model to generate text.
In this example that I'm showing, first,
I have an insurance, an e-mail.
You can imagine you're an insurance company,
and you want to understand what's being
said in e-mails and other communications with your customers.
We have a fictitious e-mail here,
where somebody is e-mailing about
an accident their son had with another driver.
You can imagine you want to
start to ask questions on this data-set,
such as who was involved in the accident?
Can I extract the names?
All I have to do here is ask who are
the names of the people involved in the accident?
You can see that the service analyzes
that text and generated the correct answer with Jacob and Marina,
which if you read through this, son that's driver is Jacob,
the other driver is named Marina.
It's really just that easy.
>> That's amazing. It's amazing for me,
because when I was studying Machine Learning,
my first NLP task was extracting entities.
Just that the model does all of it is really cool.
>> Yeah, it's fantastic.
It can save a ton of time to
really get started and iterate quickly,
and to show you even how powerful it is.
Next, I want to use the same model,
same interface, and do a different task.
Think about extracting entities.
Sometimes you want to extract like this example,
but other times you want that in a structured format.
Here, I have a set of text that
describes a new planet called Goocrux,
and some of the fruits that are available.
>> I've been there and it's wonderful.
>> Wonderful place. If you click "Generate",
you can see that it starts to create
a table summarizing the fruits from Goocrux
in a structured format.
I'll just run that one more time, so people can see.
I gave it a couple of examples.
I click "Generate" and it
understands that structure and start to pulls it out.
You can see loopnovas, neon pink, cotton candy-like,
there's plenty of loopnovas which are
neon pink flavor tastes like cotton candy.
>> That's really cool.
Even things that may seem nonsensical,
it still makes a little bit of sense on it.
>> Yes, exactly. Yeah. We can go even further.
We've shown structured texts.
We've shown asking questions.
It can also do more classical tasks like classification.
A simple example here,
we can classify the following article
into one of these categories.
I'm defining these categories at run-time.
It's dynamic classification that's
not going through that historic training loop,
passing the news article,
and just click "Generate" and it'll come
back with the correct classification.
>> That's really cool.
This is just a model without any frills.
It is just a standard Davinci model.
>> Yeah. It's a great question. Yeah.
Same model in all three cases.
It's called our text-davinci-001 model,
which is one of the standard based GPT3 models.
>> I see. I'm guessing there's other models in there, too,
that you can work against that have varying degrees
of accuracy, size, whatever.
>> Yeah, it's a great question.
So there's actually two different
pivots on this you could think about.
There are models that have various degrees of size,
and we call those the Ada,
Babbage, Curie, and Davinci model series.
Davinci being the most powerful,
biggest model, Ada being much smaller.
What we really see is that the trade-off between inferencing times,
the smaller models will compute faster,
you get higher throughput versus capabilities and performance.
Davinci, you get the best performance,
but it's going to take a little bit longer.
What we see is at the least,
there's a bit of an optimization step where you might look at,
can I solve my task with
Curie and maybe do it a little bit faster, a little bit cheaper?
Or can I go all the way down to Ada?
You start to optimize down as you
progress through that production journey.
>> That's cool.
As we're looking at this,
if you want to show more examples, that's awesome.
Show them. But I also want to ask the question.
There's probably a time when
these models get you most of the way there,
but it feels like you need to do
something a little bit more specific.
After you show us more examples, if you have any,
I'd love to know how do we make these models
more attuned to data that I might have, for example?
>> Yeah, that is a great question.
I have one more example and then we'll hop into that.
The other example looks at,
you mentioned there's multiple different flavors of the models.
We looked at the text-GPT-3 series,
which is the base series you've probably heard about with OpenAI.
But I also wanted to show quickly was our Codex model series,
and the Codex Model series came from a partnership with GitHub,
really looking at training these models
on large corpuses of open source code.
It's how do you do code base tasks.
Natural language to code,
code to natural language for code commenting.
It is one of the underlying model type that's
used to power our GitHub co-pilot feature, for example.
I will change the model,
but I won't change anything else here.
We are using the same API structure,
same type of call, and I will copy in a little example here,
where imagine I want to do some code commenting
which I, personally, always leave till the very end of my projects,
and so there's super helpful tool.
We have a method here that'll randomly split a dataset,
and I'm basically asking for
an elaborate high quality docstring for the above function
>> This is documentation that we would always write.
That's pretty cool.
>> Yeah. Always write this.
>> That's pretty cool.
Because this is code that we've all written,
but the fact that it writes
this good of a docstring is delightful.
>> It's awesome. It can be
a big time saver for tons of different use cases.
But I think it is also
important to call out what you were mentioning earlier.
In some cases, if you have
a complex task or it's really industry specific,
you find that out of the box
it's not going to quite get you there.
That's where we have the fine tuning process,
which is taking these same models,
customizing the actual weights.
Everything here that we're showing,
you're not changing the actual model or any of its weights,
you're just providing in context at that call time.
What you can do right in this UI or VAPIs,
you can go over to this model section
and actually go and create a customized model.
The way you create that customized model is
starting with one of these base model series.
I'll choose out of here, which is one of
the smaller models that we'll train real fast.
Then you update your training data.
You're training your validation data set.
Here, I have a training set and a validation set.
You can also then start to tune some of the hyper parameters.
If you don't quite know what these are,
you can go with some of the defaults and it'll train
a decent model right out of the box.
We'll kick that off and that'll start to train.
Once that's complete, you can then go and
deploy these models for inference and what I'm training here is
a set of data that trains the models to speak like Homer.
We've added a lot of training,
different examples of Homer's writings
and scripts to try to convert how the model talks and provide
a more ancient Greek flavor to our conversations.
>> This is something we did in one of our breakouts and
in the keynote at Microsoft Build
because, literally, I was interested to
see what marketing copy would look like in the style of Homer,
not Simpsons, by the way.
Homer from Homer, Greek Homer.
>> Exactly. I've taken that same data here,
walked it through the process just to
show you how that would look and that we have it
running here and so you can start to ask
very interesting questions to
get some of that Homerian flavor out of it.
We can ask it, "Why is the AI show awesome?"
>> Let's see. These two questions are synonymous.
Why is the AI show awesome?
Why the dark frowns cast over his brows?
Because they weren't watching the AI show, I think Chris.
Is what we're trying to figure out what's going on here.
This is really funny.
>> You could start to play around with this.
You can also see the iterative nature where you do experiment with
those prompts to figure out
the right way to use it or the use case.
Maybe I should have asked, why should I watch this?
>> Someone actually said that to me the other day.
I have watched enough of it.
Thank you for the confidence boost there especially.
Now, this is all very humorous,
but there may be a situation where
organizations have specialized vocabulary,
a specialized way of putting things together
that fine tuning will bear this out because I'm looking at this,
it looks like I'm reading the Iliad from Homer or something.
>> Right.
>> Mike, are we seeing organizations do stuff like this?
>> We are, yes. We're seeing lots of
organizations start to pick up fine tuning,
really apply it to their specific use cases.
Typically, you get a lot of
mileage, understanding what's possible,
how we might use these models by iterating on
these prop designs with the base models,
and then you start to learn,
develop the dataset to go and train
with at least a couple of hundred examples
for things like summarization in very specific use cases,
whether that's financial information, financial statements,
customer support conversations, or even medical documents.
All of those have their own flavor of jargon,
different types of words are being used as well
as you might have different sets of important information.
That really becomes easier
to extract with the fine tuning process.
>> Awesome. This is all really cool.
But one of the things that is always top of mind, at least for me,
when I'm using these kinds of generative models,
is the notion of using them responsibly.
How are we helping folks use this responsibly?
>> That's a great question. It's something that's
really top of mind for us as we start
to broaden access to the service in the public preview.
It's one of the main reasons why
it progressed very slowly with invite only and even now,
we're going to ask customers to apply and submit
their use cases to ensure that the service in
this generative capability isn't being misused or
abused for malicious purposes.
Then within the product,
we're actually starting to develop a set of product tools to help
our customers develop the applications responsibly.
We're starting with what we
call content filtering, content management.
What's going to happen is when
you or a customer makes an API call,
does this generation step,
both that prompt and the generated text will
get evaluated against a series of classifiers to identify abuse
or misuse and so looking for those very high sensitive topics,
like is it hate speech or is there violent content being sent.
We'll actually start to filter that out and block it to
ensure that that kind of content does not go back to the customer.
>> I see. We're doing
content filtering based upon those actions that you drive on
the way in essentially as
well as on the way out, am I getting that right?
>> Yes. It's actually looking for somebody that is on the way in,
trying to misuse it, and that is well stopping
any bad generation on the way out.
>> Well, that's awesome, because like I said,
that's the one concern I have with generative models is,
could it make people feel excluded?
Could it say things that are untoward?
We're doing a lot of work already to
make sure that that does not happen.
>> I totally agree. I think
it's one of many tools we've found that
no one single solution is going to
solve the whole problem of abuse and misuse.
There's the gating, there's these filtering tools,
and then the third big one that we do have, and
we're working on, is solution design.
How do our customers design in a responsible way?
That's not unique to the Azure OpenAI Service,
it's a AI problem and Microsoft is really investing
a lot across the company in providing guidelines,
research, and ideas of how customers can do that.
We'll start to incorporate that into
our documentation in the Azure OpenAI Service as well.
>> Well, amazing.
Where can people go to find out more or maybe sign up, learn more?
>> I think there's a link right
below us on where to go and sign up.
You have to go apply for access.
You can start doing that today.
You just have to provide
a little information on what you intend to do with the service
and so that we can ensure that it matches against
our responsible AI policies.
>> Which is, again, another axis
along the line of how we're doing this responsibly,
I think, which is really cool.
>> I agree.
>> Well, this has been amazing.
Chris, thank you so much
for spending some time with us, my friend.
>> Thanks for having me. I can't wait to be back.
>> Awesome. You've been learning all about combining
OpenAI models with the power of Azure here on the AI Show.
Thank you so much for watching,
and hopefully we'll see you next time.
Take care.