(upbeat music)
- As a recap, I'll have a batch of use case.
It's a production in my S-Q-L database,
and I want to ingest it into my data lake.
We want it in the data lake, as we want to do
longer term predictive analytics,
or measure our existing performance accuracy.
But we don't want to impact our production services,
so for this use case, I'll use A-W-S Data Pipeline.
Accessing A-W-S data pipelines for the first time,
you'll see this screen.
You can click Get Started Now
to create your first data pipeline.
First, we'll provide a name,
then we'll choose a source,
and here we're going to use existing A-W-S managed
and maintain template to take a full Copy of R-D-S
My S-Q-L Table and export that to Amazon S3.
At this point, we'll be prompted for the metadata
relating to the database.
So I'll provide My S-Q-L password.
(classical music)
I'll provide the S3 output folder,
which my table will be exported to.
I'll provide the database username.
(classical music)
I'll provide the name of the table
that I want to transfer to Amazon S3.
I'll Choose the appropriate A-C to instant start
and I'll share the R-D-S instance I-D
(classical music).
All of these resources are temporary,
so there'll be deleted after this demo.
We can choose to run this pipeline on a schedule
maybe once a day for a set period of time.
But for this demo, we'll run it only on activation.
We've also got more granular options for logging and
security.
For logging I'll disable for the purposes of this demo and
frugality.
And I'll use the default service role
which I've already provided with access to my resources.
Hear then I'll choose activate
and we'll get standard validation warning about locking,
not being enabled.
Which I'll accept and activate.
(classical music)
Once I'd refresh,
you'll be able to see that my task runner is initializing
and about to execute our first data pipeline.
For my second lightning demo,
I have an I-O-T device.
Maybe it's in the truck of a wine distributor.
Maybe it is part of the transportation unit.
To ingest that data into our data lake and make broader
use of it.
I'm going to create a kinesis fire hose.
Access in Kinesis for the first time you'll see this screen.
Select the option for a Kinesis data fire hose
and click create delivery stream to get started.
Initially,
you'll be asked for a name for your delivery stream and
you'll be prompted for a source.
Today we're gonna choose the default option for adding
objects directly to our stream and we'll click next.
(classical music)
At this point,
we could transform our source records with A-W-S Lambda
or convert the record format.
Maybe to parquet for query or storage optimization.
We want today though, we'll leave those options default.
Next up, we could choose a destination S3 Redshift
or any other options.
We already have a destination bucket that we used earlier
on.
So we'll use that.
Here we can also add an S3 prefix to partition our data
or an error prefix for handling error conditions.
(upbeat music)
On this screen,
we can optimize our caching, compression, encryption
logging.
We can enforce tagging,
and we can add any relevant permissions which we might need.
Once you have the optimum setup, click next.
Review that set up and click create delivery stream.
Now using the A-W-S, S-D-K or the command line interface
like me,
you'll be able to put records directly onto Amazon S3 using
your Kinesis fire hose.
(classical music)
In my final example, A-W-S data exchange is my source.
I want my third party data which is weather forecasting data
downloaded, as soon as it's released on A-W-S data exchange
to Amazon asteroid.
For this I'll deploy A-W-S data exchange subscriber
coordinator.
First I'll find the subscriber coordinator implementation
guide on the A-W-S website and I'll choose to launch in the
A-W-S console.
The date is set that Ivonne is located in Oregon.
So I quickly changed my region from Northern Virginia
to Oregon.
(upbeat music)
And I'll choose next.
I'll provide my stack with the name automate data exchange,
and I've already chosen the product that I'm going to use.
So I'll provide the dataset I-D.
I'll provide a Lambda I-D to notify me when new downloads
have been completed.
And I have the ability to change the logging level.
I can also choose my destination bucket.
And the subscription prefix just in case the bucket is
a shared resource.
Once that's complete, I'll click next,
and you can see a general summary.
Click next, the game to review all of the parameters
that you've provided.
And at the bottom of set these notifications
for the creation of I-A-M and privileges.
Once that's complete, click create stack to fully automate
the ongoing download of your data sets.
(classical music)