hi everyone in this video we will look at creating a data stream with aws kinesis
so first we'll go ahead and search for it in the services bar data streams are becoming really
popular it allows you to actually capture data in near real time so let's start here we'll go ahead
and click create data stream and like everything else the first thing we'll do is give it a name
i'll call it my demo stream we'll set the number of shards which is
really the throughput that it's going to allow we're going to keep it at one so that
for the sake of the demo it will reach that throughput quicker and we can see results
so now let's take a look here we are it's now given an arn we can monitor it so once we start
to push data through we can monitor it so what we are actually going to do for this demo is
create an application a python application that's going to just consistently push
dummy data to our stream from our local environment assuming you have python already
installed the first thing you need to do for this is install the required aws package which is boto3
boto3
i already have a python file saved here that i'm going to use i'll provide this in a github link
for you to use as well and basically it's just going to go through and loop through a thousand
times and say hello world we need to put the name of our stream here and we can just see
it's all it's doing is using this put record function to push a piece of data
that just says hello world to our stream let's go into this file and then try to run it
python kinesis.py that's the name of our file and it should error here
right so it's saying you must specify a region
there's certain defaults that are not configured here it's basically saying
you know we recognize what you're trying to do but we don't have any idea where you want to put this
or what username to log in as etc to handle this we will go to our user account and create
a dot aws folder that's going to hold our configuration and credentials so dot aws
we'll navigate to this folder and create these two files one is config with no file type just blank
and the next will be credentials also with no file type so what do we put in here for config this is
going to be where we say which region do we want this is where it's going to look for that
so we'll type region equals in our case we're creating this stream on us east 2 so we'll put
that here and this would be wherever you are creating yours wherever your account is based
and in credentials also write in default and we'll need to say our user credentials our
account credentials so it's able to log in and the name of these variables are aws access key id
as well as aws secret access key and to find this you actually need to create this in your account
so if you go under your account information and select my security credentials there will be
a section here to create access key and this is what we want so go ahead and click this obviously
i'm going to delete mine after i do this go ahead and copy your access key as it shows there
and then also your secret key again this is mine you can't use these you'll create
your own and i'm going to remove these after this video so we're all set here now let's
try this python script one more time and it should now recognize the credentials
and it's working here we can see it's consistently looping through and just pushing data
that hello world text to our stream so let's hop back in and see if we can monitor this and see
the data coming through select monitoring it may need a second to process we'll refresh a few times
okay
a little something
so it looks like it's recognizing it may just be a couple seconds behind on
on everything coming through
there we go now we're really able to see it in the monitoring which is great so it will
continue to push this through so this data from our local script is getting pushed up through
to our stream and the next step which we will handle in the next video is now using this data
that's in the stream and putting it somewhere else in our case we will take a look at using
kinesis data fire hose which will allow us to take the data from this stream and land it into
an s3 bucket so thanks for watching hope you found this helpful and we'll see you at the next video