In this video, you’ll see how to perform common operations on an Amazon MSK cluster.
With Amazon Managed Streaming for Apache Kafka (Amazon MSK), you can
efficiently launch and expand a cluster, configure auto scaling and security settings,
and update cluster configurations.
We’ll start from the Amazon MSK console, where we are logged in with an account
that has AWS Identity and Access Management (IAM) permissions to
launch an Amazon MSK cluster.
Let’s create a new cluster.
We’ll use the Quick create method, which creates a cluster with
recommended best practices settings.
Let’s give our cluster a name.
Under General cluster properties, we have the option of changing the Apache
Kafka version, the broker type, and the Amazon Elastic Block Store (EBS)
storage volume to allocate per broker.
We’ll leave the default settings selected.
Let’s quickly review all the cluster settings.
These settings cannot be customized using the Quick create method.
Some can be changed after the cluster is created.
Let’s create our cluster.
The cluster creation process takes about 15 minutes.
Once our cluster is created and active, we can start modifying some of its properties.
Let’s edit the broker storage value.
Currently, we have 100 gigabytes of EBS storage volume per broker.
This value can be raised as high as 16 terabytes per broker.
For this example, we’ll scale up to 500 gigabytes.
We'll save our changes.
While the cluster is updating, we can set up auto scaling for our EBS volumes.
Let’s return to the Properties tab.
Here, we can configure auto scaling for storage.
Let’s increase the maximum capacity our EBS volumes can scale out to.
We’ll set a maximum capacity of 1000 gigabytes per broker.
We'll also add a scaling policy.
A scaling policy contains the settings required for dynamic capacity scaling.
Let’s give our scaling policy a name.
The scaling metric measures broker storage utilization.
We’ll set a target value of 60%, so our storage capacity will scale out automatically
when the current broker storage utilization is at or above 60% capacity.
Let’s save our changes.
We have successfully configured auto scaling.
Next, we’ll expand our cluster by adding brokers.
To do this, we’ll use the AWS Cloud9 IDE.
First, we’ll connect to an Amazon Elastic Compute Cloud (EC2) instance where we
have a Kafka client installed and create a topic with 10 partitions and 3 replicas.
With our Kafka topic created, we'll now return to the Amazon MSK
console to edit the number of brokers per availability zone.
There are two options for scaling at the broker level.
We can edit the broker type or increase the number of brokers.
We'll increase the number of brokers.
Let’s increase the number of brokers per Availability Zone to 2.
Our cluster will now have 6 total brokers distributed evenly across 3 Availability Zones.
Let's save our changes.
Now that our cluster is back in an active state, we need to rebalance it
by assigning partitions and replicas to the new brokers.
We’ll return to our Kafka client to perform this task.
We’ll use the describe topic operation to view the current broker partition mappings.
The Leader values indicate that our partitions are currently located on
brokers with an ID of 1, 2, and 3.
Now, we’ll create a json file to describe the topics that we want to move.
We’ll input the topic we created earlier and save the file.
To generate a new proposed layout for our partitions, we can use the
kafka-reassign-partitions tool, which is included with Kafka.
We’ll pass along a list of brokers that we want to spread our partitions across to.
The command outputs the current partition assignment and proposes
a new reassignment configuration.
We'll copy the proposed partition reassignment configuration and save it into a new file.
We’ll now execute the kafka-reassign- partitions command using our newly
created file as input.
Once the partition reassignment is complete, we can see its state by
describing our Kafka topic again.
Our partition leaders are now spread across all 6 brokers.
Next, we’ll return to the console and update the security settings of our
Amazon MSK cluster.
Our cluster currently uses IAM role-based authentication.
Let’s edit the security settings to enable other access control methods.
For our example, we'll enable SASL/SCRAM authentication.
All we need to do is select the check box and save the change.
Let’s return to the Properties tab to confirm that the security settings have been updated.
As expected, SASL/SCRAM authentication is now enabled.
Finally, let’s see how to create a cluster configuration that can be
applied to individual clusters.
We’ll give our configuration a name.
Let’s edit the default configuration properties.
For this example, we’ll add a log.retention.hours property,
with a value of 72.
We'll also modify the auto.create.topics.enable
property by setting its value to true.
Let’s create our new configuration.
Now let’s return to our existing cluster and edit its properties to use the new configuration.
From the dropdown, let's select the new cluster configuration we just created.
A new dropdown indicates that this is Revision 1.
We can continue to add configurations to our cluster and choose which revision
to use, as needed.
For now, let’s save our changes.
With our cluster configuration updated, let’s return to the Properties tab to
confirm that the changes were applied.
The cluster configuration changes were successfully applied.
You’ve just seen how to perform common operations on an Amazon MSK cluster.
You can learn more about this topic in the description and links for this video.
Thanks for watching. Now it’s your turn to try.