Welcome to another episode of 'This is My Architecture'.
My name is Aarthi, and joining me today is Eric from Hudl.
-Eric, welcome to the show. -Thank you.
So, Eric, tell us what Hudl does.
Hudl is a sports video and analytics company that works with over 60 million customers worldwide.
Great. So, Eric, talk to us about what we are going to see today.
Yep, so, with our video and data, data is obviously nothing without the content itself.
So, at the core of what we do is ingesting our coaching video,
and returning that back to our coaches in a playable format as quickly as possible.
Okay, can you walk us through this architecture?
Absolutely. So, our coaches have one of many ways to get content into Hudl.
They can do it through our mobile app, our web platform, or our Hudl Focus cameras that we provide ourselves.
Programmatically, then we initiate the upload and get that content into our S3 bucket.
At the same time, we take that message
and we create an SQS message in one of the given queues within SQS.
So, do you have multiple queues here, and how are these messages prioritized?
Absolutely. There's kind of two factors that play into that for us.
One is performance, and time sensitivity.
Some users just have different demands than others.
The other factor is just how long that job is going to take, given the size of the content that we're ingesting.
We don't want that clogging up another queue that could be in its own separate one.
Great. So, once that prioritization is complete on these SQS queues,
how do you transport this data now?
Yep, so once we have those messages, we're able to deliver it to its appropriate Auto Scaling group.
Leveraging Spot Fleet then, we have over 20 instances that we have defined as being appropriate for transcoding,
and we're able to toss those jobs through our EC2 instances, to then deliver that content back to our users.
You talked about millions of users when we started this conversation,
what scale are we talking in terms of the transcoding jobs that are running on these EC2 instances?
Yeah, even on our most recent season with American Football,
you know, Friday, Saturday nights are very big nights for us,
at scale, we're looking at something around 30,000 messages per minute,
ingested across 10,000 instances at scale.
That's big numbers, how are you optimizing for this scale here?
So, going back to the two factors, one is performance, and one is cost effectiveness.
So, what we've done is initiate our own lambda that leverages CloudWatch metrics,
to see the number of SQS messages that we have in a given queue,
and compare it to the number of instances that we have.
It's target-tracking scaling.
Once we have that current ratio, we're able to inform the Auto Scaling group.
And so that way, it can make the decision on whether to scale in
and produce more instances for a workload or scale out and hit that cost effectiveness piece, like I said.
That also helps with the seasonality as well,
keeping only instances up and running during the seasonal period.
Absolutely.
You talked about cost.
I'm curious to know, what was the cost reduction with this architecture?
Yeah, at the end of the day, given the infrastructure that we were currently running ourselves,
we saw a total of 28% decrease in our total compute costs for transcoding services.
Another area of interest, Eric, is performance.
How long does it take for your end-to-end,
like transcoding from the time the user uploads for the entire process to complete?
And going along with that is, how do you handle failures as well?
Yep, so once we have receipt of that content in our S3 bucket,
end-to-end, introducing that content to our ESGs
and bringing that back into our S3 buckets in a playable format, has never taken us longer than two minutes.
And you mentioned interruptions and failures.
It's always a concern with Spot.
Two things that we do is watch for those messages alerting us
that an instance is going to scale out
so that way we know that we can't send any more jobs to it.
That said, too, we've also leveraged Spot Advisor.
Knowing which instances are going to be the most available at a given time,
We've gone through one of our busiest nights,
like I mentioned, 30,000 jobs a minute, 10,000 instances plus,
and we'd never saw a single Spot interruption.
That's impressive.
Eric, thank you for sharing this transcoding architecture, where you've optimized for both cost and performance.
And thank you for watching 'This is My Architecture'. See you soon.