JEREMY: Now we're going to go
into the mechanics of
real time Intelligence and Microsoft fabric,
and I'm joined by somebody that knows the topic
very well Will Thompson from the fabric Team. Welcome.
WILL THOMPSON: Thanks, Jeremy nice to be here.
JEREMY: Why don't we
start with explaining what
real time intelligence actually is and
what it brings to Microsoft fabric?
WILL THOMPSON: Yeah, so,
real time intelligence is
part of Microsoft fabric that helps bring
real time and event streams
together for you to ingest that data,
process it, and prepare it so that it
can be analyzed, transformed,
and then you can act on it as well,
either automatically or with human intervention as well.
All underpinned by, I think called real time hub,
which is a place for you to go to manage
and govern and secure all of these streams of data,
but also for people to go and find them as a catalog.
It's easy for people to go and
find the streams and the information they need,
which has been really hard to
do a real time data in the past.
JEREMY: Yeah, I know we have
a great way to demonstrate this.
Why don't we lead into what we've
got cooked up and ready to go?
Or we use something like this in our daily lives.
WILL THOMPSON: Exactly.
We're going to play the part of a
direct to consumer food retailer, where it's a hot spell.
We're thinking, is there anything we can do,
maybe to help increase sales around ice cream,
increase some customer loyalty?
Maybe offer some promotions.
But do that with some signals from our real time data
to help get ahead of any problems
that might occur as we do that.
JEREMY: That's right cause
sometimes the marketing teams
and sales teams don't make the best decision,
but we need data to keep everything on track.
WILL THOMPSON: Data can
help us make the right
decisions to take the right actions.
What we've done is we've brow this data together
from across multiple different systems
into a real time dashboard.
I've got information from our sales and inventory system,
which maybe only gets updated once an hour or so,
along with real time data that comes from
our IoT sensors about the freezers around our stores.
I've also got information from
our mobile delivery application
showing the number of orders that are coming in,
the number of orders being waiting for pickup,
and how many drivers are available.
If I focus on that freezer temperature piece,
you can see that it's trending up over time.
Now, increasing freezer temperatures
when there's ice cream involved, not always a good.
JEREMY: Not a good thing.
But there's a lot of
other products that we deal with beyond ice cream.
Can we narrow it in
to the ones that really matter for us?
WILL THOMPSON: Yeah, we certainly could.
I could go
and just write some queries against this data,
bust up some code, but I can use Copilot as well.
I can say show the average temperature
by department, there's a column chart.
Copilot is going to help me generate the right query.
I can drop into my workbench,
hit Go, and I'll get a chart straight back out of it.
It's really useful piece of information that I might then
want to pin back to my dashboard and say, hey,
I'm going to track this over time so I can keep
ahead of it and just have it there
ready for me updating in real time.
To move it around, resize it, put
it in alongside everything else.
That gives me that up to
date information as it's changing.
JEREMY: But we want to
get into not just department level,
but actually the individual groups
of freezers and selves.
WILL THOMPSON: Yeah, maybe we can
drill in a little bit further.
As well as writing queries,
we could also do this through some visual exploration.
For many of these tiles, I can say, hey,
go and dive into that and
show me in a little bit more detail.
I can hit this Explore button,
which will give me a workbench where I can say,
hey, I want to slice and dice this in different ways.
I might take out this summarization cause at the moment,
we're looking at the overall average temperature.
That give me tons of information.
Let's reaggregate it in a different way.
I'm going to look at our average temperatures
over time and group it by department.
Here, you can see that, yeah,
it's definitely those frozen dessert freezers
that are creeping up the temperature.
JEREMY: Ice cream seemed
like a good idea,
but we're seeing some indicators of
things that might not be ideal.
WILL THOMPSON: Yeah, exactly.
JEREMY: What do I do next?
WILL THOMPSON: Obviously,
getting ahead of this means
being alerted rather than manually going on,
and I can set alerts on any of these dashboard tiles.
This case where it was the
overall average might not be the best thing to
get alerted on because it's only going to tell
me what's the total average temperature.
I'm not going to spot the individual details.
But to do that,
I need to go and find the individual
IoT data from those freezers.
I can do that using the real time hub.
Real time hub is a place where we can
go to find all of our event streams,
all of our streams of data
from across the whole organization.
I can filter it down by the different types
of information or the location of it,
and I can search for it as well.
If I search my IoT data,
I'll find that IoT sensor
stream right at the top of the list.
When I click into this, I get to
see information about what else is using this stream.
What else is it connected to?
I get a preview on the volume
of events that are coming in from those freezers,
and I can even see a preview of the actual scheme
of the details in the events
as they come through in real time.
But what I probably want to do on
top of this is set some alerts,
and I can do it right here from that summary page.
In this case, I'm going to say, take all those events,
split them out by every single freezer,
by this freezer ID field.
Then for any of those freezers,
if the temperature value goes over a certain threshold,
let's say if it becomes greater than a value
of 29 degrees, send me an alert.
We're looking at really granular data
coming in for each of those individual freezers.
I know when there's a problem.
It's going to send me a message in Teams.
Choose a place to save the trigger back into
a workspace and get that thing created.
JEREMY: Now we've got basically
the IoT stuff more or less in order.
WILL THOMPSON: Exactly.
We can see in that trigger that
created a summary of
how many times that trigger would have fired,
and even the details of individual freezers,
how they're trending over time.
I can see D1 and D2.
This looks like those are the ones that are
creeping up causing those problems.
JEREMY: We're ahead of our freezers now,
but I saw some other things to keep track of as well.
I know our mobile delivery app
also saw some wait times and other things spiking.
How do we tap into some other databases
maybe to ensure that we get real time feeds from those?
WILL THOMPSON: Yeah, so
back on the dashboard,
we had that information from
a mobile ordering application.
But if I go and search for those orders,
I'm actually not going to find anything.
Real time hub doesn't have that information yet,
because I haven't gone and connected it.
What I can do, I'm going to put
the hat on now is like a data engineer,
go and call out and reach out to
those other systems and bring those
in for other people to use.
What we do is by choosing this get events option,
I can use a whole bunch of connectors that are
available to pull in data from confluent,
from Azure services,
from Google Pub Sub, Kinesis,
and Amazon, all those different connectors,
including CDC connections to let us reach into
operational databases and streamify
the changes going on in those
databases that we can then react.
I'm going to use the Postgress connector here,
because that's what our ordering application runs on.
I put in the details about the server name,
the database that I'm going to connect to.
I use this event stream item
to manage the events as they're coming into fabric.
Last thing to do is put in
the name of the table that I care about.
I'm listening to this delivery orders table.
Now that's going to create this event stream,
which tracks all of
those changes to that table and brings them in.
I can see the details in this payload
feel that before and after as those changes happen.
This event stream lets me
transform events wherever they're coming from.
I can aggregate it, filter it,
group it in different ways.
Even join it with other information as
multiple streams come in to get it prepped ready to use.
I'm going to choose this managed field option,
which lets me say, what's
the information out of
that table that I really care about?
We're looking at the order of wait time,
so the order ID,
the customer ID, and that wait time field.
Those are the things we care
about that we're going to react to.
I see this little preview of
what are those bits of information that's coming out.
This looks good, nice and tidy out.
I'm going to publish this back out
as a stream that anybody
else in my organization can go
and discover through the real time hub.
I want to hit Publish,
I'll go back to real time hub again.
Now when I type in orders,
it's going to find that stream for me.
Just as I did before, I can click in.
I can see some previews about the connections,
about the volume of events coming in,
and I can create alerts directly on top of this data.
I've just brought in from an external system.
Now, rather than taking
you through the process
again of setting on one of these alerts,
I'm going to jump the same.
They're basically the same process.
Click through on what field you want to alert on.
This case, I set up in an e-mail alert,
so it's going to look for when those triggers,
when those wait times cross
above nine minutes and then send me an e-mail.
JEREMY: E-mails and
Teams messages are pretty
good as well as mobile phone out notifications.
But I know a lot of people want to try to
automate the fix or the remediation.
Can we do that too with the message?
WILL THOMPSON: Yeah, you're going to love
this bit workflow automation.
Exactly. Pet lovers. In this case,
what we have is a power ultimate workflow.
We thought, what better way to get ahead of
these ice cream sales by saying,
maybe we could offer some promotions that would
increase the number of the volume
and the value of those sales,
but not the volume of those?
We have higher value sales,
but we're not tying up delivery drivers,
sending small orders around.
We have a Power Automate workflow that
listens for that data activated trigger.
Creates an approval workflow to say, hey,
I'd like to start this promotion and post
the Teams message for somebody to go and respond to.
We can say, hey, maybe the marketing team need to
just make an approval on this before the promo starts.
But by doing that, we're automating this between
the events coming in and the workflow starting to happen.
Back in that reflex item,
I'm going to change the action that I take away from an
e-mail to this campaign approval action.
It's going to kick off that workflow
automatically as that data arrives and changes.
JEREMY: Nice.
We've seen the IoT devices.
We've also seen how you can connect to
a CDC database for your mobile app.
But what about the other thing, the inventory system?
That was updating hourly or something.
Can we make that faster and also realty?
WILL THOMPSON: Yeah, we're actually doing
some system automation there because
that orders an inventory application
is coming from an Azure storage account.
The application is writing into Azure storage,
and we can connect to that as well.
Basically, any of the events that might be driven
as you create things in fabric or create things in Azure,
we can listen to that as well.
I have a pipeline that pulls
my data from the Azure storage account into fabric.
I can automate this by saying,
I want to trigger this, not on a schedule,
not on an hourly or four hour basis,
but in real time, as those events are happening.
I'm going to connect to my
Azure blob storage account and say,
this is what I want to listen to.
These are the events that I care about.
I put in the subscription.
I put in the storage account that I want to listen to,
and that's what we're going to subscribe to
here to then kick off this pipeline.
I choose my subscription.
Put in the name of the storage account.
Again, this is usually one of these events streams.
I'm going to a name as well.
Then I get to choose what are
the events that I want to listen to.
I just care about when the blob gets created,
don't care about the rest of these. I'll deselect those.
Now that subscription is created,
and it's going to start kicking off
this pipeline as soon as those events happen.
If I go back into my monitoring hub in fabric,
I can see those pipelines starting to kick
off as the events come in in real time.
We're automating that processes well,
system level automation as well
as things on those business events.
JEREMY: Amazing stuff.
We did real time intelligence.
We saw IoT sensors.
We also saw how you can do this with
your CDC databases behind the apps and even make
data that is not real time into real time data
using eventing and making
that a real time source as well.
I'd encourage everyone to
check out aka.ms/RealTimeIntelligence,
and there's a lot more coming at
the show. We'll see it soon.