Hi, welcome to 'Back to Basics'. My name is Seb,
and today we will talk about event-driven architectures with Amazon Kinesis.
When writing applications, many parts of the applications talk to each other,
usually in response to some form of a change or an event.
And I often find that one service is waiting for an event
to occur in another service.
If I have ordered a taxi and am waiting for it to arrive,
in a traditional architecture, I will need to query my taxi booking microservice
every few seconds to ask, has my taxi arrived yet?
This design pattern is not most suitable when dealing with events like this.
It generates a significant amount of additional requests,
and also means that we will only find out that taxi has arrived
after we ask again, not when it actually arrives.
But what if the taxi could tell you when it has arrived, by sending an event.
With invention of the event-driven architectures, there are better ways of handling these scenarios.
So let's explore these scenarios in more detail.
Instead of frequently polling to check whether my taxi has arrived,
I'm going to re architect my application so that my taxi booking service
sends an event when the taxi arrives.
My consumer service will now listen for incoming events.
And my taxi booking service can simply push a message to it once my taxi is here.
But what happens if my consumer service becomes unavailable?
My taxi booking service will not be able to reach it and the message will be lost.
And we could have our taxi booking service simply keep retrying delivery of the message.
However, as our taxi service increases in popularity,
the service will get very busy simply trying to deliver all these messages.
This is not a scalable architecture approach to follow.
Instead, what we can do is we can add an event-streaming service
such as Amazon Kinesis data streams.
This service allows the producer of the events to write the events to the stream.
These events are then persisted on the stream in the order in which they arrived in,
for anyone to take action on them.
So now when the taxi arrives, and even it says onto the stream,
and the consumer who is listening to the stream would receive this event in a matter of milliseconds.
And as we add new features to these applications,
we realize there may be other services that are interested in those events.
Kinesis is a massively scalable and durable real time data streaming service.
So it's actually well suited for this task.
One of the key advantages of using Kinesis in this example,
is that we can have multiple consumers reading from the stream.
We simply add another application to start reading the events from the stream.
And the application can perform another action,
such as gathering real time stats about how many taxis are currently booked.
And there are other interesting questions we could try to answer with this data.
What if we want to predict future demand for our taxis,
based on trends and how our taxis have been used in the past?
Amazon Kinesis only stores the events by default for seven days,
and it cannot store it for longer than one year.
So for this example, we need to store these events somewhere else.
We could simply write another consumer that reads from the stream
and saves this data to Amazon S3 buckets or Amazon EBS volume.
But I recommend that instead we use Kinesis data Firehose,
which will allow us to stream events to Amazon S3,
without having to write or manage any code at all.
Kinesis data Firehose natively integrates with Kinesis data stream
and Amazon S3 to seamlessly deliver these events to the bucket.
Once those events are in the bucket, they can be sent to Amazon Glacier for long term archive.
Or we can use Amazon Athena to query the information in this bucket.
For example, as I already mentioned, we can run a query to get stats
about how many taxis were booked each week for the past three years.
And later on, we could look at using machine learning to predict demand
for our taxis into the future, but that's a topic for another time.
So there you have it. Today we have learned how we can use
event-driven architectures when building our next applications.
How you can implement them in AWS using Amazon Kinesis.
And how Amazon Kinesis allows you to expand your use cases
as your application grows.
Check out the links below for more details.
Thanks for watching 'Back to Basics' and see you next time.