Welcome to 'This is My Architecture'.
I'm Gautam, and today, we have Tom from HBO with us.
Tom, do you want to share a little bit about your work at HBO?
Absolutely.
I'm head of experience operations.
Experience operations is an engineering team
within the larger operations group.
What we do is we build internal tooling, things like account management
for customer care agents.
We build tooling for operational visibility,
as well as fraud mitigation tools.
Sounds cool.
I see we have canaries up here.
Can you talk a little bit about the overall business use case
for HBO? Sure.
As part of operational visibility tooling, we were asked to solve,
how do we have one place where an internal stakeholder
could go and look and see the status of the overall experiences
up or down or issues in certain parts of it.
To solve that, we built a validation framework for outside-in
monitoring of the experience from the customer's point of view.
We used canaries for this. We built canaries within CloudWatch.
The canaries themselves run every one to five minutes.
Some can run ad hoc.
When the canaries run, actually, all the artifacts are saved in S3.
The artifacts include all the HAR files from the test runs, all the screenshots.
That way, we can go in and debug any issues as they occur.
Now, as I said, internal stakeholders need to use it,
and sometimes they don't have access to the AWS console.
So we built a status page, an actual front end that we can expose
and share with our internal stakeholders.
The status page uses app sync, which provides a GraphQL interface
for the Lambda we're using.
The Lambda, again, has AWS SDK, which allows us to pull in
all these information from the canaries, including last run, historical runs,
status, get all the HAR file information, get the screenshots,
and expose all of that to the status page for internal consumption.
Sounds really cool.
It looks like the canaries themselves are serverless.
There's kind of a serverless architecture.
Have you deployed to multiple regions as well?
As you might know, HBO Max recently just went international
to LATAM regions.
Part of our solution in front of that was actually deploying our canaries
in different regions, so we can simulate the experience
in other countries as well, which was a huge, huge win for that.
Sounds great.
In terms of the status page itself, can you talk a little bit
about who are the internal stakeholders, who are the teams you're working with
that you give access to this status page?
Absolutely.
For these folks that are monitoring the status page
are our internal response team, our customer care teams,
our internal on-call teams.
What they really care about, other than the status page itself,
is the status as each one, but also the alerts.
We actually built alerting within CloudWatch to notify those teams
if there are any issues happening.
The status page itself can actually go and get the status of the alerts,
associate them with the canaries that are run.
At any given point in time, if alerts are fired, those internal teams
can look at status page, see which ones are actually firing
the alerts, see what's breaking, see what in the actual network calls
are breaking.
That's great, Tom.
Any lessons learned or future projects you're working on with this architecture?
I know these things are always a work in progress.
Lessons learned, everything works beautifully,
it's the soft things.
It's socializing the tool with different teams.
It's creating thresholds for the alerts that aren't too noisy,
identifying patterns to avoid false positives,
and things like that.
The next steps, once we roll this out, this was received really well
by the other teams, and everyone's excited
to take part in this.
We're working with teams like the accessibility team to build
canaries around accessibility testing and things like that.
Also, HBO Max runs on a huge amount different devices.
We're looking to expand and to use device farm
to expand the amount devices that we're testing on as well.
Great.
Thanks, Tom, for sharing this architecture of how you're using canaries
and amplify to have a status page which shows you what the user experience
is like, and you get to improve it.
Thanks for watching 'This is My Architecture'.
Thanks for being here with us, Tom. Thank you.