Thanks for joining us on
Solving with AWS Solutions,
your deep dive into reference
architectures built by AWS that
are ready to deploy instantly
in the cloud.
I'm Ashley and I'm here with
Mukta from the
AWS Solutions team.
Here to tell us about the AWS
Streaming Data solution for
Amazon MSK.
Thank you, Ashley.
Amazon MSK is a fully managed
service that's responsible for
Apache Kafka
infrastructure and operations.
This solution helps you
accelerate your lifecycle by
minimizing the need to
provision resources for
multiple streaming use cases.
Let's get started.
All right, can we start with
the landing page?
Yes, sure.
So here is the landing page
for the solution.
It has a link for
implementation guide,
which have all the information
about the solution.
If you go a little bit down,
there is a link for
the source code.
If you need you can modify it
according to your need.
And also, there is a link for
CloudFormation template,
which will help you in
deploying the solution.
OK, so I see that this
implementation has
four different options.
Do I need to launch all of
them?
No.
All four options are
independent of each other,
and you can launch it and you
can use according to
your use case.
For this particular example,
I will be using Option one,
which creates Amazon
MSK Cluster, which one of the
best practice for
monitoring and encryption.
The solution also creates
client configured for all
necessary libraries, which will
help you in creating the topic
and producing and
consuming the data.
OK, so this solution uses
Apache Kafka, right?
Can you tell me a little bit
about that?
So Apache Kafka is an open
source platform for real
time processing data.
It decouples applications
producing data
from applications,
consuming data.
I can show you how to create a
topic and how to
produce the data.
For that, let's go to the
systems manager session,
which I've already created.
And I, as you can see using
this Kafka topic to decide you
have created a topic and then
to produce the data according
to your need, according to
your options, you can use a
different kind of
script available here.
The one I'm using here is the
verifiable producer
data search, which will help
you in producing the data.
All right, so we've
got the data.
And what if I want
to visualize this, my workflow?
Is there a dashboard included?
Both Apache Kafka and Amazon.
MSK gives you a lot of option
for metrics The solution itself
gives you the option for
monitoring different type
of monitoring, for example,
default or power broker basis,
so that you can have additional
metrics that can show you in
the dashboard here.
This is created once you
produce the data will
consume the data.
And what do we get to see here
in this dashboard?
So, so here you can see
everything like, for example,
the CPU usage by broker on
number of partition
across all brokers.
So when you produce the data,
you will see you can see all
the metrics here.
Wonderful.
So I know you said you were
exploring option one.
What about the other options?
When would I use those?
So Option two for Option two
users Lambda which uses the
existing topic to
process the data.
And also, you can use this if
you want to go over this same
option option for users.
Kinesis Data Analytics,
which is a better use if you
want to use Apache Flink.
Apache Flink is a popular
framework and an engine for
processing data streams.
So it sounds like there are
lots of options
with this solution.
Yes.
Wonderful.
Thanks so much for walking
me through it.
We hope you've enjoyed this
exploration into the AWS
Streaming Data Solution
for Amazon MSK.
Check out the solution and many
others on the
AWS Solutions webpage.
Thanks for watching.
See you soon!