What's up, everyone?
If you're looking to create your first big data application,
stick around, this episode is for you.
Today's episode comes from Kesha based in Atlanta, Georgia.
Kesha is a software engineer, tech instructor, active speaker,
and loves all things machine learning.
She's passionate about mentoring and empowering women, youth,
and underrepresented groups of people in tech.
Now, she wants to use her skills to shed light on income inequality.
Let's hear what Kesha has to say.
I've been in the IT industry for 25 years,
and I'm a strong advocate for diversity and inclusion, and equity in tech.
I'd like to build an app called Salary Overflow,
think, Stack Overflow, but for salaries,
that focuses on tech equity and uses artificial intelligence
to democratize tech salaries and drive equal pay in the industry.
Imagine if you had a job offer, if you could pull up information
to see what other software engineers like you earn at this company,
or what people who aren't like you earn at this company.
That would help with your salary negotiations.
Or what if you could see the salary projections
or predictions for people like you
over the course of a career in a particular role.
Kesha wants to build an application that allows anyone
to understand their salary potential and shed light on income inequality.
I love what Kesha is doing here,
which is one of the reasons why I had to have her story on the show.
Along the way, I'll also give you tips to build the foundation
for your first big data application.
The first question to ask when starting a big data application is,
"Do I have a data set?"
Right now, Kesha doesn't,
so she'll need to build applications to collect data,
and then build the infrastructure to ingest and analyze it.
To gather as much data as possible,
I recommend that Kesha create surveys on her website and on Facebook.
The mechanisms for her to collect data might change over time,
but the process to prepare and analyze them will stay the same.
To create a survey on her site,
Kesha can use APIs created with Amazon API Gateway
and AWS Lambda.
API Gateway is a great service that simplifies the process of creating,
managing, and securing your APIs.
Lambda allows you to run code without managing servers.
The responses to the survey will write to Amazon DynamoDB.
DynamoDB is a NoSQL database
that delivers single-digit millisecond performance at any scale.
The next step is to prepare this data for analysis.
AWS Glue is a fully managed service to extract, transform,
and load data.
Use AWS Glue to crawl your survey answer DynamoDB tables.
The AWS Glue extract, transform, and load library
natively supports partitions when you work with dynamic frames.
Partitioning is an important technique for organizing your data sets
so they can be queried efficiently.
Dynamic frames represent a distributed collection of data
without requiring you to specify a schema.
A Crawler in Glue will run through your data
and do three things.
One, classify your data to determine the format, schema,
and any associated properties.
Two, group data into tables or partitions.
And three, write metadata to the data catalog.
This data catalog is your single pane of everything related to extracting,
transforming, and loading your data.
With Glue, you can write scripts in Python or Scala
or just like Glue figure out how to crawl your data.
If you don't set the Glue Crawler right the first time, don't worry,
even I got stuck with Glue the first few times I tried.
The data that Glue extracts will go into Amazon S3 for analysis.
In other episodes,
I discussed how S3 can store front end code and images,
plus even host a static website.
S3 can also store data which is useful when you don't know what structure
or you're collecting it from multiple sources.
S3's unlimited storage capacity makes it great
for big data applications.
This architecture pattern for storing and analyzing data
from multiple sources, is referred to as a data lake.
Now that you know what data lake means,
you can feel more confident
dropping this buzzword into conversations with your boss.
Kesha also wants to include her Facebook survey data
in her analysis.
To do this,
extract the Facebook survey data into a Microsoft Excel file
and drop it into S3.
Then, run a Glue job to extract, transform,
and load this data for analysis.
Now you can query and even join the two datasets
with Amazon Athena.
Athena combines the strength of Presto,
with the serverless and self-managed capabilities of AWS,
a combination that rivals peanut butter and chocolate.
Oh, yeah. Oh, yeah, so good. I need more of that.
Yeah, definitely, peanut butter and chocolate, mmhm..
to store information and schemas
about the databases and tables
that you create for your data stored in Amazon S3.
The best place to secure access to your data lake
is with AWS Lake Formation.
Lake Formation provide a single view for managing all of the services
of your analytics workflow,
including S3 buckets, Glue jobs, and Athena tables.
For those who enjoy analyzing big data with Python,
I suggest using Amazon SageMaker.
Amazon SageMaker is a machine learning service
that allows data scientists and developers
to quickly and easily train, build, and deploy machine learning models.
You could take advantage of SageMaker's fully managed infrastructure
to quickly run statistical Python packages
like NumPy and Pandas.
You'll also be set up to use SageMaker for machine learning in the future.
After analyzing your data,
you definitely want to share visualizations of it.
To include jaw-dropping charts in your app,
I recommend using Amazon QuickSight.
QuickSight is a fast cloud-powered business intelligence service
that makes it easy to deliver insights.
QuickSight seamlessly connects to your data in S3 or Athena.
Be sure to choose Athena as your data source
if you created new tables with it.
Use the QuickSight JavaScript SDK to embed charts in your app.
For example, you can show a chart of the top five cities in the US
with the highest average salaries for senior software engineers
broken down by gender.
Kesha also wants Salary Overflow to answer specific user questions, like:
what do black women software engineers earn in Atlanta?
I have a job offer, is this offer comparable
to what other software engineers of the same education and experience
make at this company?
These specific questions can be answered by utilizing an API,
just like before when Kesha created the survey
with API Gateway and Lambda.
The Lambda functions for this API will process requests by querying Athena
and sending the response back to the front end.
The results from the Athena query are stored in Amazon S3,
which can be used to respond to repeat requests
instead of re-running the query.
Now, it's time for a recap!
Collect data using Lambda and DynamoDB to survey users on your site.
Next, move the response data into Glue for data transformations.
The result of that transformation and all other data
will land in your data lake on S3.
Run queries on that data with Athena.
Use QuickSight to embed charts or dashboards on your app.
To integrate that data into your application,
make a request to API Gateway,
which will invoke Lambda to query Athena.
Secure and manage workflows in your data lake with Lake Formation.
That's how you build a big data application.
It doesn't matter when you start or how much data you have,
as long as you just get started.
Thanks for hanging out with me. See you next time.