>> You're not going to want to miss this episode
of the AI Show where we learn,
is your bot actually successful
and how can you tell? Make sure you tune in.
[MUSIC]
>> Hello and welcome to this episode of the AI Show.
Part 2, we're going to talk a little bit more about bots.
We've got Gary, why don't you introduce yourself, bud.
>> Yeah, I am Gary Pretty,
I'm a Senior Engineer on the Bot Framework SDK team.
>> Okay. So bots are cool, I played with them,
they're really fun to create conversational type of things,
but how do you know if they're actually making an impact?
Is it useful? How do you know if it's useful?
>> That's a very good question and it's something that a lot
of developers don't necessarily think about.
They focus on building the great conversational experience,
but then they don't necessarily always understand,
either they've got the capability to track
the conversation or the why is important.
If you think about it, people have been
using things like Google Analytics
to analyze their web traffic for years.
It's something that everybody does.
It's a standard thing because you want to
understand which pages have been visited,
which pages have got the drop-off rates,
how long are people staying on the site,
and it's the same with bots.
>> Okay.
>> Yeah. If you think about it,
you've actually got more opportunity with bots as well.
We can track everything about a conversation.
So how long is somebody taking between messages?
Has somebody started a particular flow
in the conversation and dropped out?
Did someone ask a question that the bot didn't understand?
I see what you can then start to do is actually if I
see things that my bot is being asked and it's not answering,
that can help me with things like
backlog prioritization and what's
the next feature that I need to add to the bot,
or is something going wrong?
So in the same way that we track our website health,
we should also be tracking our conversational health as well.
>> It seems like it would be more
complicated though because intuitively,
I know how to track like someone lands on the website,
I do JavaScript type things.
>> Yeah.
>> Is there a similar facility do you have with bots?
>> Yes. It's super easy.
I think not enough developers
know about it and that's what we're trying to change.
Yeah. So I've got a very simple bot here,
it's a flight booking bot,
so I can say book a flight.
It's going to come back and go backwards and forwards.
So I'm going to go to London,
I'm going to come from Seattle and I'll go tomorrow.
Ideally, we want to be tracking all of that information.
If we switch over to Visual Studio,
I'm going to show you how really simple that is.
So within your startup.cs file,
when you're building a bot,
this is where normally you're registering things
like your state storage,
maybe some dialogues like we've got a booking dialog here,
but actually just by adding these few lines of code,
so registering a new telemetry client,
some telemetry initializers,
literally by just adding this code,
we start to get telemetry tracked automatically
for us every time we talk to the bot.
All of that data is being pushed up as
custom events into Application Insights.
Now, if people don't
necessarily know what Application Insights is,
it's an Azure service where you can
enable it by default and it will track all things like CPU usage,
memory usage for your application.
But we can push customer events
in there and then we can analyze it
using tools within Azure and hopefully we can take a look at it.
>> So a couple questions. So for someone who hasn't built a bot,
this looks very similar to ASP.NET MVC styles,
you're basically adding singletons that are going to
get instantiated or used throughout the life.
>> That's right. So in the example of the bot telemetry client,
we're registering that in our startup.cs file,
and then throughout the bot,
let's say you're in the middle of a conversation,
the dialog is actually going to go and get
the instance of the bot telemetry client,
and it's going to use that to
actually push information up to Application Insights.
>> I don't have to write log event training.
>> No. Exactly. If you want to,
you can actually extend it,
so you can send your own custom event
or you can send your own custom traits.
But actually we've got a fairly substantial schema
of information that we're capturing.
Even if you're using things
like LUIS for language understanding or
Q&A maker for really straightforward to a Q&A type scenarios,
we can add telemetry hooks to those as well with
one line of code and it's done, it just works.
So in this example,
I just had a conversation with the bot.
Within Visual Studio, you can actually open up
the Application Insights console,
and we can actually see all of the events that are being sent
backwards and forwards to Application Insights.
So bot message sent, bot message received,
all the standard events that you're going to see most of the time.
With each of those events,
we're tracking things like the idea of
the activity that's been received,
the type of the activities.
So that could be message or it could be an event,
or it could be a trace activity.
We know what type of channel we're in.
So again, just in the same way that we track
what browsers people use when they're on websites,
we want to track which channels are most popular because
most of the bots built with
bot framework are in multiple channels,
so you might have something that's in Teams and WebChat.
Well, actually, you really want to know if Teams is gangbusters,
but nobody's using WebChat,
I'd say, why is that?
So I can go and look at that.
We get things like the username and providing you enable
it because this is customizable as a flag,
we can actually capture all of
the texts that send backwards and forwards as well.
Now, you need to be careful with that obviously
for sensitive and personal information,
but it can be extremely useful in the right scenario because
you can start to see
full transcripts if you want to at the conversation.
>> But make sure you tell people you're doing that if you are.
>> Exactly.
>> Yeah. So now that we've seen
the events are being sent out from Visual Studio,
the next step really is for you
to head over to Azure and start to analyze it.
I do want to call out that we have a great doc in Microsoft Docs,
just very simple, does what it says on the tin,
add telemetry to your bot,
and that's going to walk you through how you wire
up all of the things that I've just shown you.
It's a very detailed article,
but it really is very straightforward and we have
a full sample in our samples repository as well.
>> We'll make sure to put a link down below.
>> That would be great. Yeah. The other docs that
we've actually produced as well is
analyzing your bot telemetry because telemetry is great,
but there's no point throwing all of that data
into Application Insights and
then never doing anything with it, right?
>> Right.
>> So this article,
it's going to give you the ability,
it's going to show you how to use the log analytic feature in
the Azure dashboard to
actually allow you to interrogate that data.
So we have examples of what are called Kusto query.
So this Kusto language,
which allows me to interrogate the data in Application Insights.
So here we've got a number of users per period,
and it's got a sample query here where I can just grab this.
If I copy that and then jump over here,
this is the Log Analytics section
of my Application Insights resource.
>> So your Application Insights, Log Analytics.
>> Right. Yeah. Then, we're here.
Now all I've done here is I've said
show me all of the custom events that have been pushed open.
You can see those bot message received, bot message sent.
There's also additional data,
things like somebody started
a particular piece of the conversation, have they finished?
But I can take the query that we just found in that doc,
I can run that straight out of the box,
and it's going to go and give me
a really nice sort of rendered charts
showing me the number of users in this particular period, right?
So if we jump back to the doc just
to really call out a couple of things.
So we've got users,
we've got activity per period,
that could be the number of messages,
the number of dialogs that have been triggered.
Then you get to some really interesting things further down,
so how many times is a particular user visited.
One of my favorites is dialog completion and dialog abandonment.
So you definitely want to understand if you've got a dialog
that's showing a very low rate of
completion and a high rate of abandonment,
why is that, where are people dropping out?
It's the same analogy as we use when we're
tracking things like form dropoff on the web.
Why is somebody not submitting
this form when they've spent 10 minutes filling it out,
and it can hopefully allow you to pinpoint things like UX issues.
Because just as we have UX on a website,
user-experience issues on a website,
we have the same conversational UX issues in a bot.
So this doc is super useful for giving you some examples.
The nice thing to call out, right at the bottom,
we actually have the entire schema that
we push up into Application Insights,
which means you can then go and
customize those queries for your own needs.
The next step is we can take this query and we
can actually pin it to a dashboard, an Azure dashboard.
So this is great,
the fact I can dig into my data,
but I don't want to have to come in and
copy and paste these queries all the time.
So what does that look like if we switch over to the next tab?
This is a basic dashboard that I built with all of
those queries that we have out of the box in that doc.
You can see here that I've got my conversations per day,
my users per day,
which in this case are pretty
similar because I was using the emulator,
the number of messages that have been received.
But then we started to see things like, okay,
I can see my waterfall dialog has got a high rate of abandonment.
So I have not been finishing that when I've been testing the bot,
but in a real-world scenario,
that would potentially point to a problem.
Then further down, as I indicated earlier,
you can also hook this into things like LUIS calls.
So you know what are the most popular intents?
Why are people using your bot?
>> Yeah.
>> So that's awesome,
but then what if we wanted to take it one step further?
We actually have a Power BI dashboard that's available.
This is available in GitHub.
Again, hopefully we can put a link to the dashboard.
This is really information
that we've taken from lots of customers that
we've worked with in terms of what ideally
would they like to see in analytics.
So I've got my overview,
very similar information that I've just been seeing,
the breakdown of channels that are being used,
but then I can switch over to things
like if I jump over here to demographics,
then I can actually see where is my bot being used,
which languages have been detected from users?
So it really does give you
some incredibly valuable insights
into user behavior when they're using your bot.
>> This is pretty amazing because I mean, I'm an engineer,
I love to know if the stuff that I am writing is being, one, used;
two, if it's actually doing what I
think it's going to do and this gets you to that.
>> Exactly. With very,
very little work, you can actually have this.
If you've got about there today,
you can go and add this code in like two minutes and
start capturing that data and then you can analyze it.
If you only need to dip in and out,
you can use the Log Analytics dashboard with our doc.
If you want to have something more permanent and updated for you,
can use the Azure dashboard.
Then maybe if you want to share it out and
use the power within Power BI,
then you can do that as well.
>> It's actually cool because on Azure,
you've got multiple dashboards,
you might have one for your bot health,
you might have one for some other bot,
there's tons of stuff you can do there.
>> Exactly. You could potentially aggregate these things as well,
so you can see data from lots of different bots.
So the opportunities are pretty endless.
We've just tried to put in the framework for you to capture
the data and then give you the tools to analyze it,
but really not constrain you.
The schema is there, you can use it how you like.
>> So you showed an example in C#,
is this available in other languages?
>> Yes. It's available in all of our SDK languages.
So you can go and plot the telemetry in JS,
in Python and Java,
and you get the same data,
it's the same schema, although
the implementation might be
different in terms of wiring up the bot,
everything else that we've seen today,
in terms of the Azure dashboards,
etc., it's all the same.
>> Awesome.
Well, thank you so much for spending some time with us.
>> Thank you.
>> Thank you so much for watching.
We learned all about telemetry and your bots.
Thanks for watching, we'll see you next time. Take care.
[MUSIC]