>> We've got our fourth speaker here.
Say hello to Sarah Maston. Sarah, how are you?
>> I'm doing well. How are you?
>> I am really enjoying my day, it's flying by.
I can't believe that we're already to the fourth talk.
I've been taking notes like a
random raccoon over here going crazy,
because there's so many things
I want to go look up and checkout later on.
Anyway, I don't want to hold you up too much.
I'll just let you take it from here.
You can share with the audience who you are
and what you're going to be talking about?
>> Hi there. I'm Sarah Maston. How are you?
I am a senior IoT solution architect here at Microsoft.
You may be able to see my cat,
Dennis is joining the presentation today.
Hopefully, you can still hear me.
I've had something happen with my headset, but here we go.
I'm going to talk to you today about Big Data 2.0.
That sounds like the buzzword,
big data, I know it does.
But I'm going to get into this a little bit because
IoT is your new operational data source.
I find this topic really interesting
because I come from a Data Warehousing background.
I spent 20 years doing Data Warehousing,
building Data Solutions,
and really watched the evolution
of how it just became an enterprise database,
connecting local systems and then
up into the Cloud that we are in now.
I wanted to go over some concepts here.
I'm going to go over Big Data and data in general
for those of you who are new to the database side
and maybe working with some data warehouse architects
now that their field is new to you.
For the data warehouse architects out there,
IoT and embedded engineers may be new to you.
But at the end of the day,
you're all working with data.
If you really think about it,
an IoT solution architect,
you've got the Data Scientist,
the Data Engineer or Data Solution Architect,
and the IoT engineer, all working together.
This is an interesting combination that really learning
each other's language is going to
help bring those solutions to light.
For those of you that are new to the data scene,
I'm going to tell you about the three V's.
The three V's are volume, variety, and velocity.
Why they call Big Data,
big data in the beginning is that it was the rise of the apps.
What that meant was is it went from On-prem connecting
all these relational databases into one big place to do reporting.
Well, all of a sudden,
there was the web and there were all these apps
and people were clicking things.
This was such a huge problem that
it actually led to the rise of the Cloud
and how to compute that much data and how to store it.
The volume is how much is being stored to be analyzed.
The variety is that you're getting different types of data.
It's not just a row in a table now,
it's unstructured, it's semi-structured.
It could be files, it could be pictures.
The velocity is that it's coming in really quickly.
When you think about people clicking on buttons on apps,
the machines and the devices are able to bring in
the data faster and exponentially as much.
Just to bring in an example of how this is
a big data problem or what that is
if you're new to the IoT space is to say,
think of this satellite array.
This is a radio array that's down in Australia.
It is creating 50 terabytes of volume of data per day.
It's then sending signal and image data,
and not only signal and image data,
it's bringing in data around the radio telescope itself.
You're getting about 600 petabytes of data per year.
This is a lot.
For anybody who's coming in from a data warehousing,
architect background,
the eyebrows might have gone up at that point.
But this is an example of the three V's,
so the volume, the variety, and the velocity.
How are we going to deal with this?
But first, before we start talking about
what we're going to do with it,
let me just go into the nature of it a little bit.
JSON documents.
About five or six years ago,
I didn't have much experience with those myself.
I was busy combining datasets
that were pretty much tables and relational.
Let's look at this.
This stands for JavaScript Object Notation.
This is really the data structure of the web.
JavaScript is in the name,
but it's also the format that many,
if not most of the messages you're going to be dealing
with are going to come in and this is known as telemetry data.
You've got a TimeStamp,
you've got some measures here,
and you can tell by the TimeStamp,
but this is probably coming in fast
and furious if you're used to a shorter TimeStamp.
This is going to be a lot of data.
Let's look at the metadata.
Not only is my device going to
send me measurements that it's collecting,
it's going to send me data around the device itself.
Where is this device?
In this example, it's sitting in a building,
the building type is a school.
Great. Now, I can add new data to it and say,
"What's the weather?"
Now, we're getting into a topic
that many Data Warehouse architects
are going to be familiar with.
I'm combining data.
This is now getting more interesting.
It also sends pictures, let's say.
Now, this is interesting because on a slide like this,
you may be, I don't know,
I can't really think of an example.
But think of a vending machine.
Your vending machine traditionally knows
how much it's sold and it knows if it's out of inventory
and it might not be able to tell you that,
but somebody who fills that vending machine
is going to be the one to tell you that.
However, if you put sensors on this machine,
now suddenly you could get very exact information about
what product is out and then you're
very specific when you're restocking it.
Or it could be taking
temperature information of what's inside,
what's the humidity?
But if you add the weather data to it,
what's it like outside?
Could I save energy by using
the fact that this vending machine is
sitting in zero degree weather
and maybe I don't have to use as much energy
and waste energy keeping it cold
when it's already cold today?
Now, there are four main challenges
with IoT data that people talk about all the time.
The one that we're really dealing with today in
this module right now is
the combining of data with other data sources,
this number three here.
But we also need to understand
that this data lacks consistency.
You could have 10 different types of devices.
They're all sending temperature data,
or they're all different types,
different formats and that's a problem to deal with.
Where does this device sit in regards
to other devices around it? Where is it sitting?
That metadata is helping,
and that's really a concept we've been talking
about all day today around digital twins.
I can map this vending machine
which is next to my gas station, etc.
This was all me making the case
to you that IoT data is a big data source.
It fits this model of the three V's
that maybe you only thought about that
when it came to other types of data.
But this really is a new big data source
that we're dealing with and want to use
and leverage and bring into bigger stories.
Here we go. Are you still with me
because I know I'm going through a lot of things.
If you talk to anybody who's been doing Data Warehousing,
you're talking about ETL,
you're talking about ELT,
you're talking about what kind of
data warehouse structure I'm I doing?
There's a lot of stuff once you get into this world.
If you talk to anyone on the IoT world,
we're talking about connection,
we're talking about all sorts of things.
Where does it come from?
It's coming from all those devices
and we've heard in the other sister modules to this one,
that I invite you to watch if you haven't yet,
is that this is called Azure IoT Hub on
Azure as our Cloud gateway to bring in all this device data.
I've put the link here so you can go and learn about that.
Out of the box, IoT Hub is going to be able
to connect to a few other places within Azure
and this is really interesting
because you can route the data into storage
in a data lake or something called Time Series Insights,
which I'll get into in a minute if that
doesn't mean anything to you yet.
You can go in through an Event Hub or a Service Bus
and start getting events from that data
and doing something in an application.
You can process it.
The term real-time analytics may mean something to you.
It was a big challenge back in big data
1.0 and a big reason why we were trying
to figure out how to use that data.
But we've got it again and now we have services
that can connect to IoT Hub right out of the gate.
Stream Analytics is really fascinating because it can do
some windowing type functions, it can aggregate,
it can store it into tables,
or it can sit there and watch and say,
"Hey, you know that temperature got too hot or too cold,
send an alert, let's go do something."
There are four main buckets here
when you're talking about data landscapes, and types of data.
Since I started my career, this has doubled.
It started with two of these topics when I started a while back.
Now it wouldn't surprise me if next year we end up with five or six
and this will grow as time goes on, I'm sure of it.
We've got our relational databases
and these are your standard rows and tables.
Most people are very familiar with those these days.
The data warehouse now you're getting into conversations.
Do I want a star schema?
Do I want it relational?
Do I want to do something that's column or databases?
There are all sorts of deep dives into all these topics,
I'm just going to brush on them.
But if you're interested,
there's a 45-minute version of
this talk that you can double-click into
many of the things we're talking about today
and I'll give you links to that at the end.
Data lakes, we've got NoSQL databases.
I digressed, I apologize.
NoSQL came up, this was really an attempt to fix
what was happening once the Web 2.0 came up.
How do you store data as quickly as possible
and how do you get it around the world?
That is the rise of JSON
and there's all sorts of materials you can
learn about how NoSQL databases came into be.
I'm not just talking about document store,
I'm talking also about graphs.
If you're into graph databases,
super-interesting, and if you are a graph database person,
please go look at Azure Digital Twins,
you are going to really get into that and you may know Hadoop.
Hadoop is the MapReduce,
the big jobs on big data sets that have come up.
Those may be familiar and here's a mapping of
where the Azure services fall on those four buckets.
Still hanging on? I know it's a lot.
This slide is here so that
you can just map where it's coming from,
where it's going to and we have added this
to the materials for your reference.
I mentioned Time Series Insights,
and this is incredible.
If you've ever tried to do
a time series database, you're probably twitching.
It's not easy. But here we have a service
that you can just click into IoT Hub, again,
watch the longer talk on this,
and then go out to MS Learn and you can really
get double-clicked into this and learn about it.
You can augment it,
you can share views,
you can do all sorts of things,
and then it's connected to your data lake.
There really isn't enough time to do that justice, I'll be honest.
This is an ideological conversation,
we could spend the whole time talking data lake, data warehouse.
These two don't live together historically.
There is a new service from Azure called Azure Synapse.
I'm going to show a screen print of it in a minute.
But for those of you who come from
the solutioning side of data with ETL management,
managing your SQL scripts,
you're going to be really excited about that.
I was first time I saw it.
Questions, put them in
the comments and then we'll try to get to some.
If I don't get to your questions,
feel free to connect to me on LinkedIn if you'd like.
Here's a dashboard, we all know what a dashboard looks like.
We've got some things.
If you look at this with the eye of
a business intelligence architect
or data warehouse architect,
you'll see things that look like
they're pretty canned from a table.
I see some charts, I see some alerts here,
so that could be something coming from devices
and then I see something
that looks like that Time Series Insights that you ran past.
The answer is yes.
If you look at what's behind this layer of analytics,
you are going to see that
these measures are coming from a Postgres database.
Could be a dimensional data warehouse,
could be just an operational data store,
an ODS as we call it.
Then you see this lower quadrant here,
I've got stream analytics and it's aggregating
my device data in a time window
and you can set that all up in Stream Analytics
and then saving it to a table
and that's where that metric is being pulled.
I've got Time Series Insights integrated.
I didn't mention when I went right past it
that there is an API SDK layer
that you can bring the power of Time Series Insights
into your existing applications and dashboards,
including a Power BI integration there.
If you go to this link,
you can go out and double-click into this,
and type the demo that I've been
referencing here and you've been seeing screen prints.
This is Synapse.
Those of you who come from the data solutioning,
part of the house will see the pipe there on the left,
you may see that canister.
For those of you new to databases,
it looks like a soup can,
that is our international symbol
for a database, if that's new to you.
But right away, this is a place where your report writing,
your data scientists, your ETL developers can
all meet here on Synapse and manage your work.
This one, I'm throwing up just as an idea.
Many of you may have come to the session to learn,
"Hey, I want to do something really cool
and wow my boss and come up with a new idea."
Especially if maybe you're a data person saying,
"Hey, how do I get into IoT and how do I bring this up?"
I would guess that many executives,
many people of every level are interested in
how do I put my device metrics right up
there with my traditional like,
here's my sales and here's my web clicks,
and all of those metrics that are pretty standard.
This is really exciting to me that
you'd be able to just integrate,
whether it's Stream Analytics,
any of the IoT data right into your dashboarding.
For that, wow POC,
I'm putting this up there to just give you some ideas.
Now for something completely different,
this is Logic Apps.
When I was coding a lot of SQL,
I didn't have a lot of Java experience or Python.
It's a different coding when you're
doing stored procedures or SQL.
Logic Apps are a low code way to get into doing something pretty
cool and I'm just putting it up here to
spark your interest and get you thinking about,
if you didn't think application,
design, and development was for you,
there are a lot of new tools out there,
Power Apps being one of them,
Logic Apps being able to integrate.
This may be really exciting to you especially if you're trying to
bring that alert information
into something like a Teams group message,
that's where you're going to start to do that.
I have a QR code.
Here is the link to the 45-minute session
by my colleague Christopher,
where you double-click into all these concepts.
Thank you so much for keeping up with me.
I can't express how I know this is a lot of topics
in such a short period amount of time.
Here's some more resource links for you.
There's four other sister modulus as I call them,
that are going to bring you through that journey
of how do I connect the device to the Cloud,
what do I do with this data,
how do I make it do great stuff with data
I already have and then just IoT solutioning in general.
It's all set up to be 15 minutes to get the concepts,
45 minutes for anything you want to dig into
and with the accompanying demos.
Learn alert. At Microsoft Learn,
we have the actual MS Learn modules
that are paired to this session on the IoT learning paths.
Those are listed out to you as well.
I want to say that this is such a great example.
I didn't actually know about this
till a colleague told me and I just said,
"Whoa, what a lot of data."
Crazy. This is where you can
find more information on that project.
Thank you so much for your time, your attention.
MS Learn has so much to do.
I did want to call out the certification exam as well.
The AZ-220 for IoT, is phenomenal.
There's also a lot of other ones out there
as well if you're coming from different disciplines
and want to increase your certification footprint.