Hello and welcome to 'This is My Architecture'.
Today I'm joined by Josh from Ookla.
Hi, Josh. Hi, thank you for having me.
So let's start off, can you tell me a little bit about what Ookla does?
Yeah, so, Ookla's flagship product is Speedtest, which is the best way for consumers to
measure their internet connection.
Awesome, well, I know I've used it before, I'm sure many people watching have used it as well.
But I'm really excited to hear more about the architecture we have today,
so, can you dive in for us?
Yeah, absolutely, so, as users take Speedtests, we receive reports
about their internet experience to an API, we receive about 4,000 of those per second.
Those come from consumers to our API that drops them into a Kinesis stream.
This Kinesis stream really enables us to have an async architecture from here on out.
We also get 24 hours of data retention
in that Kinesis stream, so if we have any slowdowns or issues further in the pipeline
we can always catch up, so we have the data retained in Kinesis.
Awesome, so why Kinesis?
Yeah, so, Kinesis gives us a lot of robust scale opportunity.
We previously used to do a lot of work in line in the API, connecting to RDS, that sort of thing.
And this has allowed us to scale far beyond what we could do with that system.
That's great. So once we're in Kinesis, where does the data flow from there?
We then hit our report enrichment step, which is where we build links between
the measurement data that we see in these reports
and fact tables that we have in RDS.
So these facts might be things like
the mobile carrier, the device manufacturer, the geographic place that we want to associate these measurements with.
Well that's great, so, we have the data flowing to RDS here
and I see we have Kinesis Firehose up there.
So what happens from there?
Yes, so once we've enriched these reports with the dimensions from the fact tables,
we write those enriched JSON reports to Kinesis Firehose.
Kinesis Firehose gathers several thousand of those
and compresses them and writes them as a newline-separated JSON into our S3 data lake.
So I'm guessing with 4,000 reports per second,
how much data do we have in that data lake?
So, we have a little over 5 petabytes of data
uncompressed in that data lake.
Wow, that's a lot.
So, the data moves to the data lake, where does it go from there?
After the data lake, we really want to pull through to our data warehouse,
which is Redshift, for our further aggregation and processing.
To do that, we use a Lambda job, it's triggered on new objects landing in the data lake.
That Lambda job normalizes the data into a CSV that we can pull into Redshift tables.
And so, down here at Redshift we have this data flowing to customers,
or is this for internal use?
Yeah, so Redshift really gives us the power to run aggregates over long time frames
and come up with the insights and analysis that we want to write back to RDS
and ultimately expose back to our customers through the Speedtest Intelligence product.
So I know that we have some near real-time data happening.
Is there any way to go back and use some past data?
Yeah, so we actually use Athena to explore the data,
the full breadth of data that we have collected in the data lake,
it allows us to write SQL queries and analyze any field in the JSON that we collected.
Once we've identified a new piece of data or a new analysis that we'd like to do and expose to our customers,
we are able to actually trigger this Lambda job retroactively
on the historical data in the data lake, we can go back 6 months or a year
and pull those insights through, and it's as if we've been collecting them for a year.
Yeah, so we have a lot of data coming in, we also see a lot of serverless happening in here too,
which is awesome.
This must be really great for your customers, to have the ability to get this past data,
near real-time data, and then provide these insights,
so, thanks so much for joining us, Josh, I've learned a lot.
And thank you for watching 'This is My Architecture'.