Christopher Blackden: Hello, today we're going to
talk about how to use Amazon Kinesis data streams
and data fire hose for real time analytics. So to
start off, let's look at some of the documentation
here. We get this data into data streams from an
application or some process we've set up that
produces data. In the Amazon terminology, it's
called a producer. It will take that produce data
and put it out into a lambda function or an EC2
instance, or a Kinesis data analytics application,
which is what we're going to be taking a look at
later. If you look at the page for data firehose,
it's very similar except it will put it in, it
still gets the data from a producer, but the
consumer part of it, instead of invoking a lambda
or putting it in an EC2 instance, it can put it
into Elasticsearch redshift or s3. So, aside from
that, these two are pretty similar. Definitely
want to take some time to evaluate, which is
better for your use case. But as I mentioned
earlier, I'm going to focus today on AWS's data
analytics service. So here I am on the Amazon
kinesis dashboard. As you can see, I already have
a data stream up and running. So I'll go into that
a little bit more later. But I want to show you
guys how to create a data stream as well.
So to start off here, let's start with create our
data stream. And I'll call this tech snips demo.
And if we scroll down a little bit, we see that
it's asking us for one input and that's the number
of shards. Now a shard is the rate at which Amazon
will consume data. So if I put in one shard, that
means I can write up to one megabyte or 1000
Records per second. And I can read two megabytes
per second. And I can scale this as high as I need
it to go for my application. But each shard will
come with its own cost. So keep that in mind as
well. For this demo, I'm just going to use one
shard and then click Next. Great, so my stream has
been created. I'm going to go ahead and access it
now. And we'll see we have different monitoring
and tags that we can take a look at while we're
using this. We also have our ARN for if we want to
reference this in an API or a script or something
like that. And we have producers that we can put
action to send data to the stream.
Now depending on your application, there are
several different ways to do this. I can show you
one using the AWS CLI by This is going to be one
of those things, you want to look up the
documentation and see what fits your use case
better. So let's get started on the CLI. So I've
got my WSL terminal open on my Windows 10
terminal. Let you know I've already installed the
AWS CLI and configured it for use with my account.
If you haven't done that yet, go ahead and take
some time to do that now. I'm going to use AWS
kinesis list streams to see what streams I have
available. And it shows my Kinesis analytics demo
stream, which I'll be going more into in just a
moment, and the tech snips demo stream. Let's go
ahead and look at our tech snips demo stream by
using AWS Kinesis describe stream, and I'll pass a
parameter of stream name, tech snips demo. Great.
So I see I have some information on my shards and
my ARN stream name, as well as some other metrics
I can use here. Now keep that shard idea in mind,
we will be going back to it later. For now let's
put a record into our stream. I'll use AWS Kinesis
put record, and I'll pass the stream name, again,
partition key, which is basically the idea of the
shard that we want to to write to. This is not
something you have to worry about for the time
being. So just put one. And the data I want to
write is Hello tech snips. Great. I've got a
confirmation that I've written to the shard. So
now let's go ahead and get my record. I have to
make sure I have the sequence number.
So let's go ahead and copy that down for now. And
then unfortunately, there's no way to get records
from Kinesis directly from the stream, you have to
access it from the shard. So what we're going to
do, first get our shard iterator and that will let
us know which shard we're accessing with our
record in it. So if I use Kinesis get shard
iterator, I'll need to pass this three parameters.
That is the stream name, the shard iterator type,
which there are several of these. So go ahead and
check the documentation. In this case, since I
know my sequence number, I'm going to put my shard
iterator type as at sequence. And then I will
include the starting sequence number. And that
will be my sequence ID. Excuse me, that will be my
sequence number from the previous step. The next I
need my shard ID. And I also have that from the
previous command. So let's go ahead and paste that
in there. And now we can run it. All right, great.
We've got our shard iterator now. Let's go ahead
and get the record that's in that shard. So this
command is AWS Kinesis get records, and then we're
going to pass it the shard iterator.
All right now we have our sequence number, the
timestamp and the partition key. Great. We don't
recognize what's in data though. This is not
exactly what we wrote in there to begin with. But
that's because part of the Kinesis is that it uses
base 64 encoding for everything it writes. So if I
go ahead and run an echo with this data, and then
pipe that into base 64, decode it, I can see Hello
tech snips. I don't know why it's at the beginning
of my prompt here. But there it is Hello tech
snips in our shard. Now let's go back to our other
stream. So as you may remember, I have two streams
in my demo here, tech snips demo which we use to
describe how to put things into a stream using the
CLA. And we also have the Kinesis analytics demo
stream. So for the rest of the demo, I'm going to
be working in the AWS console, and show you how to
use the data analytics application to parse that
stream as it's already got stock ticker data
streaming to it.
Okay, so I'm back in my console, here, my AWS
Kinesis console and under data analytics on the
left selection bar, where I can create an
application. Now we'll call this application
stream parcer. And what this is going to do is
it's going to consume data from the stream, and
then aggregate it in a way where we can analyze it
in real time. So let's go ahead and connect our
stream. And the reason we're using the Kinesis
analytics demo stream here instead of our tech
snips demo is because that is already set up by
Amazon for you to be updated in real time. If I
were going to have to do the same thing with
techsnips, I would need to write a whole
application around that. So let's pick this one
for now. And we see we have some options with
pre-processing some of these records with lambda,
or changing our access. We don't really need that
right now. So let's go ahead and skip straight to
discover schema. There we go. Alright, so it's
detected our schema. And it looks like we have a
ticker symbol sector change in price. This is a
stock ticker after all. And this is how it looks
formatted in a sequel-like way, and if we go to
raw, we see what's actually in the stream, row by
row. So keep that in mind as you're writing your
applications. Let's go ahead and Save and
Continue. Great. So that's set up. Let's go ahead
and go to our sequel editor. And I'll start the
application because it is a lot easier to run this
way. This is going to take a few seconds to start
up. So let's wait.
Alright, our application started and we see the
data coming in, real time from our stream. Let's
go ahead and scroll here we see there's a lot of
different sectors here, a lot of different
organizations, we don't really care about all of
them. So let's see if we can filter out just the
technology sector. So if I go up here to the real
time analytics tab, it gives me different
in-application streams. And these are defined from
the sequel queries that you run here. So I've got
one here, that's going to create a new stream out
of all of our data in the other stream, and then
filter that based on whether the sector has the
the word TECH in it. So let's go ahead and run
that. Great. So now I have my new in-application
stream called destination sequel stream. It's only
got records coming in from our Kinesis stream with
the sector-like tech or technology. All right.
That's how we get started with data analytics on
Kinesis. Thank you for watching.