In the previous demo,
you saw how we downloaded a bundle of Kafka tools and scripts,
and we use some of them to start a Kafka Server.
Then we use a few of the scripts inside to create a topic,
read from a topic, and write from a topic.
Now, it's great for demonstrations,
but if we want to use this inside of Python,
we would have to use some Python libraries in order to interact with Kafka.
That way, our applications can interact with Kafka programmatically.
To start off, I still have the Kafka server running,
and I still have the Zookeeper broker running.
This means that Kafka is still running and available on my local machine.
To start off by using Python to act as a consumer or a producer,
I have to install the package for using Kafka in Python.
You see that I already have this installed, Kafka-Python.
But if I were to install it from scratch on a new environment,
I would type in pip,
install kafka-python, and this will install it.
Again, this is already installed,
so nothing happened on my computer.
Now, I have two files prepared on this machine.
The first one is called producer.py,
and the second one is called consumer.py.
As you can imagine, one is used for reading messages from
Kafka topics and the other one is used for writing messages to Kafka topics.
So I'm going to be starting the first one.
Consumer.py.
Let's take a look at what's inside of this file.
This file is very simple.
We take Kafka and we import the KafkaConsumer.
Then we have the topic name.
Our topic name is named items. We have a consumer.
The consumer is equal to KafkaConsumer connecting to a topic.
By default, it will connect to the correct port on local host if nothing is specified.
Now this will start a loop and read the messages as they come in from the consumer.
This will essentially be an infinite loop and we'll read messages as they come in.
As the messages come in,
we will be printing them out to the console.
Now that we've seen what's inside of the file, let's run it.
To run it, I call python consumer.py.
This launches behavior very similar to the script that
came in with the Kafka files that we downloaded.
It will wait here for messages to come in and print as they arrive into the topic.
I've opened up a new terminal.
It's in the same directory and we'll use this terminal to look at the producer.py file.
Since the previous tab that we had for the terminal is now being blocked by the consumer.
Going inside this producer file.
Again, this seems pretty straightforward.
We have the Kafka library and we're importing KafkaProducer.
We've set up a topic name and again,
the topic name is items.
The Kafka Server we've defined explicitly here is at local host on port 9092.
Then we actually instantiate the KafkaProducer and pass the Kafka Server.
This line will set up a producer that connects to the server.
Following, we have the line producer.send with the topic name and the message.
This line is telling us that for the topic name items,
we want to pass it in with the message.
One thing to note is that you'll see that there's a string b,
before the Test Message string.
This b is a python-specific syntax
telling us that we want to treat this string as binary.
This means that when we pass values into Kafka,
they should be structured as binary messages.
In this example, it's pretty straight forward where we simply
have to include this character before our strings.
Lastly, we have the command producer.flush.
The reason we have this is that the Kafka producer library that we're
using will not send messages right away after you've sent them.
Instead, it tries to be more efficient by batching
messages as groups and sending multiple messages in one request.
That way, the producer isn't wasting valuable resources by
making many HTTP requests with small amounts of data.
It's more efficient to make a few HTTP requests with large amounts of data.
Therefore, when we run producer.flush,
it's telling the producer to send the message immediately.
If we don't have this,
we might have to wait a few seconds before we can see
the message show up inside of our consumer.
Now that we've gone over what's inside this file, let's run it.
Going to run with python producer.py.
When I run it, nothing is outputted
because nothing has been specified to be printed out here.
However, if we switch to the tab with our consumer,
you'll see that a message came into our topic.
This is structured as a Kafka data structure,
but we can see that inside of topic,
topic equals items, which is the name of our topic.
It provides some additional metadata that we might use if we have
some more sophisticated designs where we have to
optimize for certain use cases such as the offset,
or the headers, or the checksum.
Some other pieces of metadata such as the timestamp.
Overall, what's important is that inside of the value,
we see the value that we passed in test message.
So we've taken a python library,
and we used it to act as a consumer,
and as a producer so that we have a Kafka server
that's running and we can use Python to programmatically interact with it.