[Music]
hello and welcome to solving with aws
solutions your deep dive in
architectures built by aws
that are ready to deploy instantly in
the cloud i'm your host sean
and i'm joined here today by fellow
builder and mario kart champion daniel
daniel's going to be talking to us today
about a new streaming data solution
hi sean thanks for having me customers
can use streaming architectures for a
variety of use cases
such as iot telemetry data and
e-commerce applications
an aws has several servers that can be
used to build them however
customers tell us that configuring their
service to work with each other
while following best specs for
monitoring security can be time
consuming
the solution tries to help with that
let's jump into it
all right daniel take me a little bit
deeper into the solution what's it all
about
absolutely so this is the landing page
of the solution and right now there's
two options
for each option there's a link to the
source code on github the confirmation
template and the implementation guide
there's also a button that customers can
use to launch the solution directly into
the aws account
ah so they can get up and running pretty
quickly yeah absolutely okay great
and i see right on the landing page
there's two options here
uh do i want to do one the other or both
how do i decide that's a good question
customers can choose whatever whichever
option best fits the use case
and only launch that template i see and
so
well let's let's pick the first template
what's this pattern all about
this pattern is designed for use cases
where you want to collect data from
several sources
including non-aws environments and
prepare that for downstream processing
okay uh i see you chosen to put api
gateway at the beginning of the
architecture what was the
design thinking behind that we decided
to use api data as a proxy to kinesis
because api gateway allows us to set up
custom authentication approaches so
let's say i have my own existing
identity provider
you can set different quotas for
different data producers so you might
have one device that sends a lot more
message
messages than the others and it also
allows us to change the backstreet
the backend data stream without
modifying existing clients
oh that's pretty helpful um so let me
get this straight so
let's say maybe i've got a factory floor
with sensors maybe temperature sensors
i want to ingest all that data into the
cloud this is the pattern i want to use
yeah absolutely they will send the data
to api gateway and once the data arrives
in kinesis
you'll be picked up by a lambda function
that by default only logs the records
but you can easily modify that for your
own use case
oh so i could take that lambda function
and maybe perform an action let's say if
the
temperature is getting too high or too
low yeah absolutely all you have to do
is customize the limit function i see
let me just jump into the console real
quick and show you what happens when a
recognition is received
okay let's jump in so i'm using the test
method
of the api gateway console but you can
easily do that via the sdk as well
i see so i'm sending some sample data
and once the data arrives in kinesis
it's picked up by the lambda function
and it ends up in cloudwatch as you can
see on the console
but like i said before if you wanted to
customize that all you have to do is
change the lambda consumer function
i see this is going to make it easy for
me to get up and running with the
streaming data app
and well let's move to the second
pattern when would i want to use that
one yeah absolutely
so earlier i mentioned e-commerce
applications right so let's say i had an
e-commerce website running in the bus
and i wanted to feed a real-time
dashboard with customers information
i see this pattern would be a good fit
okay so i'm looking at the architecture
here
how did you design this so in this
specific use case
where our application is running on ec2
so we're sending data to kinesis via the
with the kinesis producing library also
known as kpl which is ideal if you have
more control over the code that's trying
to kinesis
once the data arrives in kinesis it is
processed by kinesin analytics
and in this specific option we're
leveraging apache flink did you choose
apache flink because it's a managed
service within kda yeah that was that
was one of the reasons
kinis data analytics takes care of
several tests that otherwise i would
have take care of such as
snapshots and backups and apache flink
also has several features that are
interesting
for our use case such as exactly one's
processing
oh i can see why exactly once processing
is important when you're dealing with
data coming in from all over the world
and into the cloud
yeah absolutely let me just jump into
the console and then see how
this this option could be used okay so
i'm logged into the ec2 instance and i
will run the command manually but again
you can do that programmatically as well
and this will generate some sample data
once the data arrives you can use the
streams picked up i can use data
analytics
ends written to s3 nice but again apache
flink has
so many features that you can leverage
and you can use almost all of them
in case that analytics oh okay that's
that seems pretty helpful
um i noticed you included cloudwatch as
part of the architecture
what am i getting out of that so
regardless of the option you choose
the solution will create a dashboard to
help you monitor the overall health of
the architecture
and also alarms with thresholds that are
predefined according to the service
documentation
let me show you one of them okay so this
is the dashboard for the first option
which uses kinesis and lambda
these metrics are available elsewhere
but you have to go into multiple places
to find them
this provides a single view into your
application health ah so have it all in
one place right here yeah absolutely
and you can easily see if something's
wrong if an alarm the function is taking
too long
if if it cannot read from kinesis for
some reason it's here
and in addition to that we also have
alarms which will automatically notify
you of any problems
so that you don't have to constantly
monitor the dashboard manually i see so
i get all this out of the box so i can
really focus on my app
rather than sort of the alerting
monitoring
and such so how do i kind of get started
with this
and get up and running so all the
components are available on
github and customers can use them into
existing workloads outside of the
solution
oh man daniel this is a great way for me
to get up and running with my apps
thanks for walking me through no problem
thank you for having me these are two
great patterns to help you get a jump
start into building your streaming data
applications on aws
they've got alerting security and
monitoring built right in
you can either use the pre-configured
defaults or customize them for a
specific use case
check the solution out and many more on
the solutions website see you next time