- Welcome to "Back to Basics".
In this episode, I'll share an architecture
to gain near real-time analytics on data stored
in Amazon DynamoDB using Amazon Redshift.
Real-time insights are critical with many use cases,
including personalization, fraud detection,
and churn prevention.
The key enabler for the real-time use case is getting access
to the data very quickly, gaining insight,
and taking action at the right moment.
Let's understand this with the help of a use case.
I have worked with customers in the healthcare
and insurance industries
in which they have developed
incentive-based health challenges,
or quests for their members.
Examples of challenges include run in a 5K event,
bicycle 15 minutes daily, or walk in a charity event.
To understand effectiveness of these challenges,
they needed near real-time insights.
Getting insights quickly helps to understand
how members are signing up for quests,
what cohort of members are signing up,
and also blending this data with other enterprise datasets.
Let's see how you go about designing for this use case.
You have a user-facing health fitness application
with well-defined access patterns
like creating new challenges, new member signups,
and to track member activity
as they progress through challenges.
We selected DynamoDB as an operational data store
as it provides fast and consistent performance
at any scale for online applications.
Access patterns are well defined,
and DynamoDB table designs can be tailored
for those specific query patterns.
But how about analytical queries where you want to slice
and dice across different dimensions?
Or you want to do analysis by taking quest member data
and joining with other enterprise datasets?
For this, you need Redshift,
a fully managed data warehousing solution that uses SQL
to analyze structured and semi-structured data.
So you identify data stores,
DynamoDB for the online health fitness application and Redshift
for analytics on the data generated from the application.
But how do you continuously refresh data from DynamoDB
into Redshift?
Let's look at a pattern
that uses DynamoDB's integration with Kinesis
and Redshift's streaming ingestion.
You have an application built on DynamoDB.
You enable Kinesis Streams at a table level.
With this, the table sends out a data record
that captures any changes to the data in DynamoDB.
This data record includes specific time an item was updated,
item's primary key, images of the item before
and after the modification.
These data records are captured
and published in near real-time to Kinesis.
Next, you use Redshift's streaming ingestion feature
to ingest data directly from Kinesis into Redshift.
With this, you avoid the intermediate step
of staging data in S3,
and thus reducing overall latency in data refreshes.
Let's see how streaming ingestion works.
Here, Redshift works by acting as a Kinesis Stream consumer.
You create a materialized view that is the landing area
for data.
Auto-refresh can be turned on,
which loads data from the stream as it arrives.
When the materialized view is refreshed,
Redshift compute nodes allocate each data shard
to a compute slice.
Each slice consumes data from the allocated shards
until the materialized view attains parity with the stream.
You don't have to land stream data in Amazon S3
because streaming ingestion data goes directly
to the Redshift materialized view.
Once the data is refreshed in the materialized view,
you can perform downstream processing
and transformations of streaming data using SQL,
or use your existing BI and analytics tool
for real-time analytics.
With this pattern, you get three key benefits.
Redshift can process several gigabytes of data per second
from Kinesis Data Streams.
This allows you to experience low latency
and high throughput when consuming streaming data.
So, you can derive insights
from your data in seconds instead of minutes.
Also, it's a straightforward setup.
With few clicks,
you get data continuously replicated to Redshift
without managing complex pipelines.
Finally, you can perform rich analytics on data stored
in DynamoDB with Redshift
and using familiar SQL without needing to learn new skills.
Tying back to the health fitness application,
when a new member registers
or as a member progresses through a challenge,
data is made available in Redshift
in near real-time for analysis.
This pattern has the ability to be applied
to many other use cases across other industries.
For example, in the gaming industry,
you will see real-time use cases
to analyze telemetry data generated from games,
or in the financial sector to determine
if a credit card transaction is fraudulent.
Hopefully, you are able to tie this architecture back
to a use case that you may have.
Check out the links below for more details.
Thanks for watching "Back to Basics".
See you next time.