KYLE PFLUG: Hi there,
I'm Kyle working on web apps
and plugging experiences on the Edge Team at Microsoft.
Over the last few months,
Bing Chat and has seen tremendous interest with
over 500 million chats and
100 million daily users
interacting with search in a whole new way,
supercharging their creativity and getting straight to
answers thanks to the power of large language models.
Over the past few weeks,
we've shared more about the next wave
of AI innovation across Microsoft,
including the co-pilot system in Microsoft 365,
as well as new and upcoming improvements
to the Bing Chat experience,
such as multimodal conversation with text and images,
more languages, and chat history.
Today, we'd like to talk more about how
developers can participate in this wave of innovation,
thanks to plugins for Bing Chat
and other co-pilots across Microsoft.
Plugins let you teach a language model
how to interact with your service using
your new or existing API
with end points for the language model to call.
This lets you augment
the capabilities of co-pilots and helps connect
users with your service to answer
their questions or complete their task.
We're already working with
many new and existing plugin developers
to bring their capabilities to chat.
We're embracing the same plugin model as OpenAI,
making plugins fully compatible across ChatGPT,
Bing chat, and the Microsoft co-pilot system.
This means that your investments and
plugins can reach the widest ecosystem of users,
including Microsoft's commercial
and consumer co-pilot offerings,
as well as ChatGPT with a common approach.
All that's needed from you is
an OpenAPI specification documenting
your API and a manifest file that
provides the necessary metadata for the plugin.
We'll use these assets to help users discover
your plugin experiences and engage
with them naturally in their conversations.
Let's dive into the plugin model in a little more detail.
For that, I'd like to turn it over to
my colleague Constantin from the Bing Team.
CONSTANTIN IONUT ROSOIU: Hi there.
I'm Constantin and I am
a developer on the Microsoft tuning team as part of Bing.
I'm thrilled to share some of
the exciting new features we're building for Bing Chat.
As Kyle said, we're adding
plugins to Bing Chat and across
Microsoft co-pilot using
the same plugin model as OpenAI,
allowing developers to seamlessly enable
their plugins to a whole new set of users.
Let's look at an example of what's
needed to enable a plugin in Bing Chat.
We will be looking at OpenTable,
which will soon be available as a plugin for Bing Chat.
First, you will need to create
a plugin manifest which provides
the necessary metadata for making it
discoverable to our users as well as to the system.
Here you will define things like
the name of your plugin, a brief description,
the OT needed when invoking the APIs you expose,
and other things like the icon to using
the UI when you're plugin gets invoked.
The description for human
is a human-readable description of
your plugin and the capabilities
used for discoverability purposes,
for example, when users are
searching for plugins to install.
The description for model is
used when deciding to call your plug-in.
It should contain general instructions about
your plugin capabilities and how the model should use it.
You should use natural language in
a concise yet descriptive manner without
specifically instructing the model to call your plugin.
Because of contexts length limitations,
it's recommended to keep it
short and the exact limits are
subject to change as we experiment and learn more.
The second step is to define
an OpenAPI specification that
documents how your API works.
You will create a standard JSON or YAML document that
contains metadata about the APIs
you want exposed to the model.
You can choose to selectively expose
only parts of your API or specific functionality.
In this example from OpenTable,
we can see that there is only one API exposed,
which is a slush restaurants API.
Here you will add
the standard metadata like the specification version,
title or version number.
Then you will describe the actual APIs
exposed and their parameters.
When handling a user query,
the model will use a description for each of
the APIs to decide if it's
relevant to the user query or not.
The description should be in
natural language and should summarize the functionality
provided by the API while keeping it
concise with the exact limit subject to change.
The details of each parameter should be
similarly descriptive and brief.
Here you can also specify
the parameter types and if they are required or not.
For OpenTable, we can see that the query and
location are the only required parameters when
searching for restaurants while
all the additional filtering parameters like the date,
time, or party size are optional,
and the model will decide when to
set them depending on the conversation.
Once you have a plugin manifest,
an OpenAPI specification,
and your APIs available,
you are now ready to submit your plugins
to Bing Chat and the co-pilot system.
Now, Kyle will talk about how we're going to enable
developers to submit their plugins and what happens next.
KYLE PFLUG: Thanks Constantin.
We'll have
more details to share soon around
how you can submit your plugin
manifest directly into our catalog.
Once your plugin is reviewed and approved,
we're working to make sure users can discover
and add your plugin to Bing Chat in Edge.
To do this, we're adding
new functionality to the Edge sidebar.
I'm excited to preview for you a new sidebar experience,
which you'll see beginning to roll out
in Edge insider builds this month.
We're enhancing the Plus button in the edge sidebar
to be the central place to find add-ons for
your browsing and chat experiences in Edge,
including new third party sidebar apps,
which you'll be hearing more about
in other sessions at Build.
Here, users can find
a feed of recommended experiences from
third-party developers and for
Microsoft now including chat plugins from our catalog.
It's a snap for users to learn more about your plugin,
add it to their account so they can access
it in Bing Chat across their devices.
Once you've added your plugin,
it's ready to go anywhere you use Bing Chat,
including bing.com and the Edge sidebar.
Let's look at a couple of examples.
CONSTANTIN IONUT ROSOIU: My friends are visiting
: this weekend
CONSTANTIN IONUT ROSOIU: and it looks like it's going to be sunny.
I'm going to ask Bing to find
a restaurant for us to enjoy the nice weather.
I am in chat mode on bing.com.
You can see that the list of
: plugins I have installed and
CONSTANTIN IONUT ROSOIU: that are available for Bing Chat
to use in this conversation.
I am going to ask Bing to
find me some restaurants with a view.
These are all great options.
I'm excited to check them out.
You can see how the model automatically
picks up the relevant plugin to use,
in this case, OpenTable,
and suggest a couple of restaurants,
while also adding a short description
for each to help me decide.
Similarly to the web search tool,
Bing shows an interstitial to
inform me about the plugin that was used.
The summary for each of the restaurants
highlights my requirements and you can see
how the model puts an emphasis on
the atmosphere and the views each restaurant has.
Daniel's Broiler in Lake Union seems like a cool place.
I'm sure my friends are going to like it.
I will ask Bing to help book something for Saturday.
You can see that the model
automatically picks up OpenTable again to check
availability without me telling it to do that
explicitly and comes back with a link to book.
It already pre-populated the correct reservation details
such as the group size and time.
Now, I can simply go ahead and book it
directly on the OpenTable website with a single-click.
We had a nice dinner the other day,
and I particularly enjoyed the dessert.
Let me show you what I'm talking about.
Of course, I could have used
the traditional Bing Image search here,
but I wanted to show you one of the features
we have already enabled in Bing Chat,
the Bing Image creator.
This and search add
native capabilities that are enabled by default,
and with plugins we're making it easy for developers to
further extend those skills and
integrate their existing apps.
I'm not much of a cook,
but this looks delicious and I
think I can pull it off and bake my own.
I'm asking Bing to find me a recipe,
an easy one for Peach Melba.
Here are some options and I'm
going to click through the second one.
This seems easy enough to follow.
Now I can simply launch Bing Chat in
the Edge sidebar to pick up
the ingredients from the recipe.
Bing Chat can automatically figure
out the list because it has access to
the web page content as it runs
in the context of the current tab I am on.
This looks like all I need.
I just remember I actually have some of these ingredients
already so I'm going to tell it which ones I don't need.
Then I simply ask Bing Chat to
generate the shopping list for me.
Here we see that since I had
the Instacart plugin enabled,
Bing Chat use it to create the shopping list for me.
Now I can simply submit that list to Instacart and
have all the groceries
delivered to me in time for dinner.
You can see how the traditional Instacart UI
is still very useful as it allows me to adjust
the quantities and make
products substitutions to the list
that was automatically generated.
This puts me, the user,
in control of the final actions.
That's the end of the demo. Now back to your Kyle.
KYLE PFLUG: We'll have more to share
soon about how you
can publish your plugins into our catalog.
In the meantime, developers can
start building plugins for ChatGPT
today and those plugins will work in
Bing Chat and across Microsoft's co-pilot system.
We can't wait to see the experiences that developers
will light up with these revolutionary technologies.
See you soon and happy building.