Welcome to 'This is My Architecture' in New Zealand. I'm Viral.
And joining me today, we have Justin from PredictHQ. Welcome, Justin.
Hi, Viral. It's nice to be here.
Justin, could you tell us a little bit about PredictHQ and what you do there?
So I'm part of the platforms team at PredictHQ.
What this means is it's our responsibility to ensure
our online services are up and running optimally.
So PredictHQ is a demand intelligence company.
And what this means is we tell our customers where the demand is coming from.
So where people are going to be,
things like concerts, food festivals, sporting events, things like that.
So we're very much a small team,
but we've got that New Zealand number eight wide culture
of getting things done at our core.
That sounds like some great kiwi engineering at its best.
Could you give me an example of a customer that use your services?
So Uber are one of the customers we help predict demand for.
So imagine you're in San Francisco on a Saturday night.
We tell them where all the events and people are going to be,
so you never have to wait long for a ride.
That sounds like it's going to be an interesting solution.
Could you tell us about the solution that you've built on AWS?
So we run a modern microservices based architecture.
Most of our core services run on top of EC2.
So this includes things like our Kops-based Kubernetes clusters, Kafka and ElasticSearch.
We do also consume a wide variety of Amazon services.
This includes S3, DynamoDB, IDS databases and even Redshift.
So if we were to take a look at it one level deeper, how does it all work behind hood?
So we generally divide our pipelines into three main areas,
and that's our fetcher services, pipeline services, and our customer facing services.
So if we start here on the left,
our fetcher APIs are responsible for going and fetching raw data from a variety of places.
So this could be paid for datasets, public APIs, or even manual data entry.
So on a typical day, we do approximately terabytes of data that we pull in and we pass.
And then what we do with that is we push it into one of two data stores,
either Kafka or ElasticSearch for later processing.
So you've told us about how the fetcher services pull seven terabytes of data.
That's a lot of data. What happens next after?
What do you do with all that data?
So next we move on to our pipeline APIs
and they're responsible for pulling in all that raw data
and transforming it into that meaningful information that our customers need.
So an example of one of these APIs is our data deduplication service.
What it does is it uses natural language processing
and machine learning to group together the related raw events that are in the data stores.
How do you ensure your microservices to be effective and cost effective and scalable?
So we rely on auto scaling groups heavily in order to scale out
to meet our peak customer demand
and scale in to save money during the quiet times.
The other thing we do is we aggressively cache our processed data in one of two places
either Redis or DynamoDB.
So at any one time in DynamoDB, we'd have about 90 million records cached
and what we do is we heavily used the expiring TTL feature of DynamoDB
to purge old data out of the system after a set period of time.
So we've seen how you collect all the data,
process all the data, use of machine learning,
and take it and generate insights for your customers.
How do your customers get access to these insights?
So there are a variety of ways our customers can access our data
and the first one is through our Control Center Web Application
and that sits on top of our raw APIs.
That's a really convenient way for our customers to visualize and manipulate our data.
The second way is through our raw API
and we typically serve about 25,000 requests a minute.
About 95% of those requests are serviced in under a second
and an increasingly popular way that our customers
access our data is either through Snowflake
or the Amazon Data Exchange.
We find that very often they are already consuming
these services so it's very easy for them
to integrate our data into their existing pipelines.
Justin it's really awesome to see how the solution has come together
and how PredictHQ is putting New Zealand on a global stage.
Thank you for sharing your solution with us today.
Thank you for watching 'This is my Architecture' in New Zealand.