Are still here. I really appreciate it.
Try make it short and sweet.
I am dhruv goel and here to
Talk about ingestion in day a pipeline with kafka clusters
In azure hdinsights. A little into azure hdinsights
And what it is and how you can get
Managed kafka clusters with azure hdinsights.
We will talk about customer examples where we have seen a lot of success.
A quick demo on how this set up an ingestion pipeline with
A azure hdinsights. And a quick sneak.Preview of
One of the features we are working on and it will come
Out soon and time permitting, we will have q & a at the end.
Jumping right into it. First of all, how many of you
Heard of hdinsights? a couple of people.
Kafka? great, that awesome.
Out many of you have set up a kafka cluster before?
Okay. So you know what i am talking about.
How -- how hard or easy to set up your kafka clusters.
Doing it on prem or the cloud? on prem?
How about you? system set-up.
Okay. A wide priority veer.
So what azure hdinsights is a managed cluster service where you can have a lot of
Different type open source clusters created for you by a
Click of the button. You can have open source components like spark, x
Pwaition base, kafka, storm, all of those created in a very
Easy manner. What hdinsights provides is you a service.
A 99.9 Guarantee not just on the vms but also on the entire
Service. Out of box
Om s pollentation. Tools with visual studio,
Intelg and cool tool for data designers and engineers.
We see the enterprise security that they want integration and finally it is so easy to extend your clusters with apps
Like stream sets and many others that basically enable to you extend the
Clusters. So by the way, we just
Announced a price cut, and now it is 50 lower than where it was before and it is even
Easier for you and cheaper for to you justify that decision
To run the clustersers in the cloud which are much more
Scalable. And before we actually did the
Price cut, we had the study that showed that running
Hdinsights clusters compared to prem save you 60 to 70 of
Your management cost. A shot out to alina who did a
Great session about kafka and spark and that one goes a little bit more into details
Of kafka brokers and kafka topics. If you want a kafka one on one
On how those things work, you should definitely check that
Out. So what does kafka do for you?
What does it really en table? when you think kafka, you hear
This technology, what can it do.
What kafka is a streaming engine.
A streaming platform rather, and that enables a lot of different applications.
You can think of realtime fraud detention. Someone swiping a card.
Making sure it is your card or someone else's card. You can't wait for that
Decision to happen five minutes down the line or ten
Seconds down the line. That needs to be made real
Time. Iods things that you see today.
They are based on services to make sure you are gathering those insights in realtime
Quickly. Clickstream
Analysis from our customers. We have e-commerce customers that have their web sites based
On the behavior of the customers and want to give the right office to those
Customers and many others. So we can go deep near any
Scenario we want, but a bide variety of realtime analytics.
What kafka for azure hdinsights gives you get the
Service level, but you get other stuff like integration
With azure management. One of the biggest things you
Could have seen with the on on-prem compilation of day clusters, it is stored on the
Vms that kafka is running upon what that means more day
Coming in, you have to add more vms and that adds to the
Cost. With integration of azure
Managed disk, your computer storage is separate.
You can store it independently and reduces your cost
Exponentially. You can get gusters.
Really quickly. This kind of shows you on the
Right damage, shows you the ui that you will go through to
Kind of set up the kafka cluster really quickly.
We have for azure and comes with it.
D azure in the data centerself have update domain
And to give you even more and better availability.
We are aware of that and making sure that your vms
Never cross the same domain or fall domain.
With the oms and stream sets and different applications
Some of them listed over here with
Data, day iq, gives a lot of functionality to make sure
You are not getting a plain vanilla kafka cluster but a lot out of it.
We support the maker and one of the biggest things of
Hdinsights is you get your own secure v neck talk lock down
Wherever you want. A lot of our west percent have
Security requirements where nothing can be coming in.
They don't want to expose any public end points.
Stuff like that. The way you do that with
Hdinsights is through v-nets. And you can have other classes
Beside kafka in the same v-net. A storm or spark.
What that enables is you for to create this modern data
Warehouse that is real-time analytics where you are getting structured -- structure or unstructured data
Comecoming into kafka and then it can pass it continue to day
Bricks, spark or azure hdinsights and you process that and send is to a digitalization and down the
Line. That the second example that
Can be enabled through that. It?
You know, it is really easy. Set up a v-net and create a
Kafka cluster within that. The kafka cluster is easy to
Do through the portal on templates or scripts.
Once you set up the kafka cluster, what you need to do
Is create topics for data coming into those topics.
And in this case like -- a topic is basically how kafka
Distributes data across consumers and producers.
Likes, for example, if we talk about an event, you want
Topics that are carlos kateed in north america or in asia
And like different countries
And topics of those things and the biggest thing about kafka
Is the producer and the consumer are super simple. This is something written in
Spark and i am going to go through that in the demo. You really don't have to do many things. The only thing you have to do
Is kind of set the -- set the properties of the kafka
Cluster where you say, hey, this is my kafka broker, and
You send messages to it. Similarly reading from a spark application, you can say this
Is my kafka broker and then do read florida it.
You will see how similar it is from any batch job as well.
So with that, couple of examples.
Like i was talking about connected card architecture
And what we are seeing with some of our customers today,
They have these iod within the cards and sending signals when
The customers are running the cards, and then with the
Signals the companies with process how the cards are
Doing, how fast and preventive maintenance for those cars.
Send them signals, hey you need to come in for service
Maintenance and this data is coming into real time into
Kafka and protest with apache spark and storm written with
The same v-net and store it in cold storage and pass it on to
Realtime dashboards on power bi or anything else that you want.
Another big use case for us is something like siphon which is
An internal microsoft team, and what they do is they act as an ingestion service bus
For a lot of different things within microsoft such as
Office 365 and bing and the amount of what they are seeing
Through the kafka clusters is amazing.
They are seeing eight million events
Per peak interest.
And the biggest aspect of kafka is how scalable
And how -- how much trooper can handle
It very easily and it has -- super low latency within it.
With that let's do a quick demo.
Actually i am going to use a really simple demo that is
Available on our doc s web site.
What we do in this is basically reading from a data
Set, and then writing that to kafka.
That is what the spark actually is doing.
And we will have kafka then streaming that data back to
Spark. So this is kind of the set-up
Of that. And you can try out these
Demos super easily on microsoft docs web site.
What we have is a custom demo or template and all you put in
There is the spark cluster name and the kafka cluster
Name and the log in user name and create that for you. I got that part set up before
We started here so i can just show you what happens after
The cluster set-up is there. This resource group as you can
See, it has that deployment set up for the kafka cluster
Which is this one right here and the spark cluster which is
This one right here and a bunch of other things just to
Make sure. When you set up the vms.
They are a part of your cluster.
You go and look into the kafka cluster, you see that it has nine nodes.
It has -- it has a few head nodes, worker nodes and
Zookeeper snodz. Once we have that up and
Running and the spark cluster up and running this is
Embedded with spark clusters in hdinsights that help you
Basically write simple spark ways or do a data scan
Scenario. What i did was -- put in your
Cluster name/jupiter and upload the node that comes
With it and that is something that is provided in the demo
And go and upload it into your spark notebook, the jupiter
Notebook in the spark cluster and open that notebook up.
The first thing you are doing is setting some configurations
Up to load the right laboratorylaboratories for spark a nd
Kafka. In that notebook and then
Setting up a topic on which -- like i was talking about
Before, any data that comes in gets loaded in into topic.
A few key things that come into play when you are
Defining a topic is the -- the replication factor.
The number of partitions that you want to have, and the --
And -- and the topic name. The remaining stuff like
Zookeeper you just provided so i can connect to the right
Zookeeper right kafka cluster. So just to show what you was
Created before, there are is the list of topics i have in
My kafka cluster right now so you can simply equate that
With a simple script. And you can see how the topics are distributed.
Since that created a application -- an application
Partitions. Each partitions is replicated
Twice across the kafka cluster and gives you super high availability and puts on a
Different vm in case any vm goes down, your zookeeper
Selects the next for that partition so streaming
Application will keep working. It does not go down.
What it will do now a azure will get a new vm.
Replace the vm and it will copy over the data from one of
The existing follow-up fish son who.
Once it sets up the topic it looks like underneath.
Let's go back to the notebook. In the
Notebook retrieve the data from taxi trips in new york city in 2016 and a simple
Spark of data flame gets loaded
And connects to this url. Loads that data up and show
That -- so you can see that the kernal is spinning and you
Have the job in the background and you have that data in the
Day frame and you provided the kafka broker and the topics
And do a little bit of that set-up and send data to kafka.
Once i have done that -- i have done that four times
Before and always data in the kafka topic.
This will add to the topic. You can see the data is
Already there. And just to show you what i am
Going to do is actually read from the kafka topic.
A simple consult consumer. You can also read data from
Spark which we will do in a bit and will show how quickly
It has gotten all that the data already. And this is a streaming job
And don't know where to end. 6,000 Messages.
Before. I can run it again and send
The data again to kafka and we can see how quickly it is
Getting that data. So just -- by the way, this
Was a spark cluster we had running and now it is sending
That day to kafka and kafka already has ha data.
So, again, 7,000 messages it already got all the new messages in there. So what we are going to do now
Is read from kafka in this part cluster.
We define a schema for the cluster and two different
Types of queries in the notebook: the batch and the
Streaming. I will show you the streaming
One because that is more relevant.
And so -- just to show an example of what a streaming
Query looks like here. It is really simple.
Actually really similar to what a spark batch wave may
Look like, but instead of doing a read, you do a read stream.
And you see the format is server.
And you subscribe to that kafka topic.
Starting off basically special fiz how far back you want to
Read so read from any point in that kafka topic and load that
Streaming day frame. Now that you have the spark
Data frame, you can just select and what this
Questionery is actually doing is writing to cold storage and
Could be one your scenarios connecting to any cold storage
And writing back to that. Once it does that, it will --
It will give me some idea of when that is done.
And then we can look at the files in that -- the file
System location to see if we hose files.
And you can see the spark kernal is busy and has 7,000
Records that it needs to write in there and actually the time
It takes because of desk, 30 streaming query it doesn't
Know when to end. End at 30 seconds.
I hope that made sense and if you have questions, i will
Stick around if you want to ask questions afterwards.
These are files that wrote to your cold storage and since i
Have done this a few times before, it has more files than
What we wrote this time. That was just a quick demo.
At i also wanted to show you. Let me switch to the input
Here like one of the things that
They are working on and challenge with our customers
Is kafka clusters are usually long running and you want to
Make sure that any hard fixes, batches that come up you don't
Kill the cluster and start it again.
And the managed pass service is kind of what you expect.
What you get now is this rolling upgrade feature with
Which any -- if anybody -- if ever a new version comes out
Backwards compatible and this showing you the ui for that
And go in your portal and say i have a kafka cluster running
On this version and need to update it to the next version
And stop one broker at a time. One zookeeper at a time and
Check and make sure that everything is running and go
And do a service check and then it is on the kafka.
If that kafka broker is running fine, it will roll and
Do the upgrade on the next one and gives you an option to
Finalize the upgrade. It doesn't force to you go
With the upgrade. If things are not working correctly.
Hey, hold on, let me roll back to the previous version of
Production and not break my production pipeline.
So that's something -- that is in the works and will come out
Soon. Th that, i -- i think i
Ended up really quickly. So i have three more minutes left.
There is a lot of resources over there that you can check
Out and find more information about kafka, about hdinsights,
W to create streaming pipelines, how this learn more about different streaming
Sources. And like i said, you know,
Check out the presentation from yesterday which was about
Creating pipelines with kafka and spark.
Cool. Any questions while i am here?
Anything that i can answer right now?
Otherwise i am also available afterwards.
Great. Thank you all for your attention.
Really appreciate your time. Have a good one.