Okay, so let's jump straight into a simple example
of parsing some raw json
into highly optimized Parquet format
to query on the AWS Athena Data Lake.
So I click connect data source.
You can see there's a series of data sources supported here,
some basic ones that are just free to use,
you can sign up online and have a go with these,
and then we've got some more advanced integrations
for streaming and batch loading.
So I'm just going to upload a file,
and I'll show you the contents.
It's very simple, there's just one record in it,
but it's a nice example of a piece of raw json.
You can see that it has a nested data structure,
so there's an enclosing array with the object itself,
which represents a customer, some customer attributes,
and then a sub-nested element which contains the name field.
So we'll upload that file,
and I'll just call this job customer.
And if we save that, Skippr's now analyzing that raw data.
There's only one record but there might as well be billions,
it doesn't make any difference to Skippr.
You can see that it's parsed any validation errors,
discovered the schema, and here we have
our nested representation of that data.
And just to remind ourselves, there's the raw data.
We have customer, which is a nested record
of some attributes, and then within that we have another,
again, that nested map of name.
So we might decide, at this point,
while Skippr is purely a loading tool
and doesn't really make any attempt to transform data,
that's for other platforms,
we may at this point be interested to note that actually
there's some personal identifiable information in this.
And so it's entirely possible that we might say
for phone, actually, and email,
we wanna drop those fields,
we don't want to load them into our Data Lake.
You could argue address as well,
but I'll leave it there for the sake of demonstration.
Equally, an entire nested array structure,
it's possible we wouldn't want any of these fields,
and so we could drop the entire nested structure.
In this case, just by way of example,
I'll just hide the last name field.
So if I go ahead and just mask that,
and, well, we're okay with that, so we'll save our schema.
And the final step is to simply
add an output destination,
and in this example, we're going to use AWS Athena.
So we'll create a new destination,
and we'll call that customers.
And I'm just going to name the workgroup customers,
which is just the Athena workgroup
where our query's operating.
The database, I'll name customers,
and for sake of simplicity,
I'll just name the glue table customers as well,
or customer, rather.
The bucket that we're going to land the output data in,
I'll use our test bucket,
and rather unimaginatively,
I'll save it in the output path of customer as well.
The region for this test bucket happens to be in Europe.
And then, in this case, again, just my test credentials.
And we're set.
So we'll test to make sure we got all that right:
yep, we have permissions, and we've entered the details,
and so if we just save that.
That's it, that job's now started.
It hasn't started, apologies; now it's started.
So Skippr's now running.
It's parsing the data that we've uploaded via that file,
which in this case is just that one record.
But you can probably imagine, if that was a Kafka connector,
you might have billions of records streaming in now.
Skippr's discovered the schema,
so it's now serializing that data,
into Parquet format, in this case.
and outputting it into an S3 bucket,
which we can then query via the AWS Athena service.
And just to show you that S3 bucket,
you might recall that we entered skippr-sample-data-output.
You can see this bucket's currently empty.
It's still empty.
I've got it configured to output the Parquet data
in one minute batches.
You'd probably want to increase that in production
to something like 30 minutes, an hour,
depending on your tolerance.
The larger the time period,
the larger the Parquet files, that you'll have output to S3,
and that's going to help optimize your query performance.
But in this case, we just want fairly quick results.
In about a minute's time, our data will appear here.
While we're waiting for that, we can jump into Athena.
This is the AWS Athena console, we see it's empty.
If I just refresh, we can see that our customer database
and our customer table is being created,
and if I expand that, there's our schema.
We'll have just a quick, closer look at the schema,
using the generate DDL command,
and we can see here, we've got an internal timestamp
that Skippr's added, and we have our customer record,
which is a struct, and contains our address,
phone, email, company, and then finally the name field,
and which again is structured data.
So not much fun to create these schemes manually.
I don't know anybody who enjoys it, I certainly don't.
So it's quite nice that Skippr just does this for us,
by parsing the raw data.
So let's pop back to our S3 bucket,
and see if our Parquet data's appeared.
Yep, there we go.
Okay, so we output to the customer folder,
and we have some time partitioning,
which in these case isn't relevant,
and finally, our Parquet file's here,
which is just the 2.1 kilobytes.
You'd normally expect this to be 100MB or so,
which again would optimize your query,
but for the purposes of demonstration,
we've just got the one record in here.
So let's prove that we can query out
this Parquet file via Athena,
and we can do that very simply just with preview table.
And there it is, our one result,
and you might recall that we said we were going
to drop the name and email field,
and the last name we were going to mask,
which we did on the Skippr schema page,
there's our drop phone and email and mask in the last name.
So let's have a look at what it's like
to query structured data in Athena.
We have a customer record,
and let's have a look at the address.
And we could equally say, well,
we want select the name,
which is a map of first and last name,
and then finally, let's have a look
at the hashed last name.
And there we are.
So this will work with Gigabytes,
Terabytes of raw json data,
all auto-optimised into Parquet columnar storage.
You can see that the nested data has,
the schema's discovered, and it's all parsed out
to Parquet nested data structure.
So no flattening of json
or slow chunking through long json strings.
We can take full advantage of the performance
that comes with the Parquet format
and any Data Lake engine
that itself takes advantage of that Parquet.
Be that AWS Athena, as in this case,
or Snowflake, or your Datelake of choice.
So I hope that was interesting and useful.
Thanks ever so much for checking out Skippr.
If you'd like to try it for yourself,
just sign up to the beta at skippr.io,
and please feel free to reach out via the chat
or hit me up on LinkedIn if you'd like to discuss
how Skippr can accelerate your data projects' time to value.