>> Hey friends. Do you know just how easy it is to simplify
the management of your Apache Kafka event infrastructure.
Lena Hall is here to show us how you
can use Azure Event Hubs behind
the scenes to help you focus on
your code instead of wasting time maintaining infrastructure.
We're learning all about it today on Azure Friday.
[MUSIC]
>> Hey friends, I'm Scott Hanselman.
It's another episode of Azure Friday.
I'm going to learn all about Apache Kafka and how to make things
simpler with large-scale data processing. How are you, Lena?
>> Great. How are you, Scott?
>> I'm very well. Thank you for bringing the big data
and lots of cool stuff with Azure Event Hub.
So what do you have for me today?
>> Definitely yeah.
I have here Apache Kafka.
This is one of my favorite open source projects ever.
This is really great, because it
allows us to ingest massive amounts of
events and distributed between
publishers and subscribers really easily.
It has a very rich community around it and
so many libraries and
connectors that we can use to get data from various sources.
So one of the things that can get a little
complicated is cluster management.
Kafka can be deployed anywhere, on-premises,
in the Cloud and managing the cluster can become a full time job.
>> Really?
>> Yes.
>> Wow.
>> There are various configuration settings that you need to
make sure are correct to ensure good performance,
efficient cluster management and sometimes when you add
new topics or scale of the cluster and need to
do rebalancing and various operations.
>> So keeping it optimized and healthy is really important.
>> It is and people spend a lot of time doing this
in addition to programming applications for it.
So there are some things that we can do to optimize it.
Sometimes people switch to using managed services to do that.
But in many cases managed services
require rewriting your application to target a new service,
and sometimes company don't want to do that,
to depend on a new service where Kafka
already has so many connectors and libraries.
>> If we're creating a huge amount of
information maybe it's coming from like what IoT Hub.
>> Exactly.
>> Little tiny devices. So I have
millions of devices like let's say I got millions of light bulbs.
>> Yes.
>> They're going to be sending this data
and they don't care whether I'm ready for it or not.
>> Yes.
>> They just take that data.
>> Yes. It can be IoT devices,
it can be financial applications,
it can be any kind of input source.
>> Okay.
>> Apache Kafka is great for aggregating it and
distributing it between many consumers.
So one of the things that we can do with Azure and in
particularly Azure Event Hubs is we can use
our existing Kafka applications and clients and
talk to Azure Event Hubs and
it will understand the Kafka language.
>> Okay. So is it pretending to be a Kafka ingress point?
>> It understands Kafka protocol.
>> Understand the protocol interesting.
>> So instead of talking to
VMs behind the Kafka cluster that are called brokers.
>> Okay.
>> We are talking to a public IP endpoint of Event Hubs,
and all we need to do to change
our solution is to change the connection information.
We don't need to change any libraries that we use.
We don't need to change any code, just configuration.
>> Interesting. So the things that I know how to use,
the things that are already part of
the Kafka ecosystem continue to work.
>> Yes.
>> They are all speaking Kafka.
>> Yes many of them are supported.
Yeah. So I have a real-world example that I can show.
So let's switch to this.
So very often Kafka is used with Apache Spark.
So Kafka can act as
an event ingestion system and Spark as an event processing system.
So here we have Kafka producer.
As you said it can be data coming from
sensors or it can be a financial application anything,
and there is a Kafka consumer that would receive data
from the event log and process it in real time.
So if we want to simplify our cluster management and
use Azure Event Hubs with our Kafka producers and Kafka consumers.
All we will need to do is take a look at
our connection configuration sunk
off creates IP addresses of brokers.
In an Azure Event Hubs is a public endpoint,
and we can still use
the connectors like Apache Kafka connector for
Spark and just point
our producers and consumers to the right end point.
So let's take a look at how it how it looks like in action.
So imagine a company is already using
a Kafka cluster on Azure HDInsight.
These are my broker VMs.
>> Okay.
>> We have a Pachase Spark cluster using Azure Databricks.
>> Okay.
>> So when we navigate to the Data Brick cluster in Spark,
we have a library attached for Spark streaming with Kafka.
That's the only library we need,
and this is the Kafka producer.
So in real world it can be sensors or any information,
and here we generate messages that are coming
in every 10 seconds- 10 messages every second.
>> Okay.
>> So we send it to Kafka brokers.
What happens when the consuming end is we connect
to those stripes servers to our topic.
>> Right.
>> Perform our processing logic.
>> Is this Java I'm looking at or is it a Kafka specific language?
>> So this is Scala.
>> Scala.
>> So this is a good language to work with Spark.
It's supported in Azure Databricks Notebooks and
it's a really good interactive way of
seeing how we can work with events.
>> You brought in a library with Maven.
So everything thinks it's talking Kafka.
>> Exactly.
>> Because it is.
>> Because it is right now, yes.
>> Okay.
>> So we're seeing a lot of events coming in through our system.
>> Okay.
>> So now if we want to switch to an Azure Event Hubs,
we would need to create an Kafka enabled
Event Hubs namespace and
a namespace is equivalent to a cluster in Kafka,
except there's no cluster,
it's just a container for topics.
>> Okay.
>> We have a test topic here we named it the same
way as our Kafka topic,
and they have the same absolutely unchanged to producer logic.
The only thing that we change this configuration.
Bootstrap.servers now points to
the public end point of our Event Hub,
and we're adding the connection string which we can get
from Azure portal settings in shared access policies
>> Our processing logic is completely unchanged as well.
So only thing we change is our connection configuration and was
completely unchanged processing logic we are receiving
same events through Azure Event Hubs using Kafka API.
No change in libraries, no change in code.
Just change in configuration.
>> Does that mean that I would then no
longer have to run Kafka for that,
and then I don't have to think about maintaining it
optimizing it keeping it healthy and happy.
>> Yes. You're not running Kafka,
you're using Kafka libraries
and you're running Azure Event Hubs on the backend.
>> Interesting. So I have a serverless,
effectively serverless ingress point for all of my Kafka messages.
>> Exactly, and that's very useful in a lot of cases
where you don't want to take care of infrastructure management.
One of the other benefits in this scenario is
that we can combine
and mix and match Kafka libraries and Event Hubs library.
Because if I'd have had such a good integration with Azure,
we can have our producers right to Event Hubs using Kafka APIs.
But we use Event Hubs APIs to talk with Azure Functions
or Azure Storage and benefits from both those systems.
>> I see. So once it's into
this ecosystem I really have the benefits of to ecosystem.
>> Yes.
>> So that I can use the Azure ecosystem
of services that are available to me
as well as a number of the Kafka connectors both in and out.
>> Exactly, yes.
>> Very cool. Okay, now that people
have this now is just in preview is just a checkbox.
>> This is available.
You can try it out right now.
There will be links under
the video that you can check it out. It's available.
>> Are there any downsides,
there's any reason why I wouldn't want to do this.
>> Sometimes you might not want to do this
because you might want to run your Kafka cluster on-premises,
in your own data center.
Or you might want to manage all of this specific nuances
of your broker configuration and do a lot of fine tuning.
>> But with Event Hubs you won't have to do any of
the configuration except for planning for your through output,
and if Kafka you would have
to set up a lot of the configuration settings for this.
In Event Hub we have throughput units which once the unit is
basically for one megabyte
ingress per second or 1000 events per second,
and if you don't really have time to
think about what your workload is going to be,
you can rely on that auto inflate
feature which will automatically adjust to spike.
>> Excellent. That was my next question.
The fact there's an auto inflate is really good to know.
>> Yes. It's very useful in practice.
>> Fantastic. All right,
I am learning all about large-scale data processing with
Azure Event Hubs for Apache Kafka.
Thank you so much Lena Hall.
>> Thank you.
>> I'm learning all this stuff today on Azure Friday.
[MUSIC]