DALE MARKOWITZ: Some people are born athletes,
other people are born Mathletes.
And me?
Well, I've been failing gym since the second grade.
So I always thought it would make sense
to take something simpler, like software engineering.
And this basically describes my lifelong relationship
with sports until a couple of months
ago when I learned that my friend, Zack Akil, who's
both a software engineer, but also
a competitive rugby player, was working on a project
to analyze soccer kicks using machine learning.
So in this episode of "Making With Machine Learning,"
we'll talk to Zack to see how he did it.
And then I'll try to adapt his techniques to my tennis serve.
Stick around to see how I did it and how you
can do it with your own data.
[MUSIC PLAYING]
So I wanted to talk to you because I
heard that you built a really cool demo at London
Next this year, using machine learning
to analyze how people were able to kick a soccer
ball for a penalty shot?
Did I get that right?
ZACK AKIL: Yeah.
Everything except for the soccer bit.
It was actually a football demo.
DALE MARKOWITZ: Football?
It didn't look that way to me from the video.
ZACK AKIL: No, it was football.
We call it football over here.
DALE MARKOWITZ: Yeah.
So Zack, why did you decide to build this demo?
ZACK AKIL: Well, we have a partnership with the Football
Association, and we were just sort of spitballing ideas
as to what would be a really cool way of highlighting both
the sport and some really cool things you
can do with machine learning.
DALE MARKOWITZ: So what exactly were you doing with that demo?
I wish I could have been there.
ZACK AKIL: Yeah.
So we were using three different cameras
to track people's accuracy, power, and the very style
of their kick.
They go on to the pitch, they line up the ball.
We give them the instructions-- three kicks,
and they all get recorded.
I was there with a whistle.
The people who were organizing it
didn't know I was going to bring a whistle,
but I brought it anyway.
I'd blow the whistle.
They would kick the ball.
They would try to hit one of two targets in the top left
and top right corners.
After three kicks, they would see on a live dashboard
the results of the power, the accuracy, and the style
of each one of their kicks.
After that, they would get their own player profile card.
DALE MARKOWITZ: So tell me about how
you took all of this machine learning technology,
and you put it into an actual phone app.
ZACK AKIL: Ooh.
So that's a good question because we
did have three mobile phones recording
from different angles.
We have one camera placed down on the ground.
Its only job is to track the ball on its way
to the goal from the side.
And by tracking the ball from this angle,
we can easily see the speed of the ball.
And then the next camera we have is positioned above head height
and behind the kicker, looking directly at the goal.
And this camera tracks where the targets are on the goal.
And then the last camera, that's positioned
kind of at a 45 degree angle, facing the kicker.
And this is purely for the Video Intelligence API
to run its pose estimation on.
The API will pick out the coolest shot,
and then that frame is used on the player profile card.
For tracking the ball's speed, we
did some high school mathematics,
where it was the difference in space over time
to work out the speed of the ball.
And then to work out accuracy, because we also
had binding boxes for the targets
within the goal returned, we just
did like a Euclidean distance to the nearest target.
So that was accuracy and speed pretty easy.
And then the last one was style.
I mean, how can you measure style, really?
So we had to just come up with our own metric of what
was the most stylish kick.
DALE MARKOWITZ: So what went into it?
What was the technology stack that powered your demo?
ZACK AKIL: For the kind of really cool stuff, the machine
learning, we used AutoML to build
the tracking models of the ball and the targets and the goal.
And to analyze the posture of the player,
we were actually using a feature that's
soon to be released, called Pose Estimation.
DALE MARKOWITZ: The feature that Zack's talking about here
is actually called Person Detection.
And it's now available through the Video Intelligence API.
ZACK AKIL: When you give it a video
and there's people in the video, it
will estimate the skeleton, and where it is,
and where each joint is.
DALE MARKOWITZ: What are all of the different machine learning
problems that you faced as you were building this?
ZACK AKIL: The first challenge that we knew we wanted to solve
was just tracking the ball.
We figured, OK, a football is a pretty generic object.
Maybe there's a pre-trained model that we can utilize.
And what we found was when you take a picture or a frame
from a video of a kick, the ball is too blurry
to then be picked up by a generic pre-trained model.
And that was the first challenge, was OK,
we had to train our own vision model
to detect a blurry football.
So I built two auto ML models.
Here's the first one, which is the tracking of the accuracy.
And you can see here, if I click on image,
you can see where we've labeled the images.
Using the tool, we draw boxes around things.
So if I want to draw a box around that,
we drew a box around it and told it what we wanted it to track.
DALE MARKOWITZ: So you have two targets
that are on the goalpost and the ball annotated in this model?
ZACK AKIL: Yep.
Exactly, yeah.
After we've labeled some data, to train a model
it was really easy.
You just go to the Train tab, and there's a button.
And you can see here we actually trained a couple of models.
By clicking Train Your Model, it will give us
some questions about it, like, how long do
we want to train the model for?
But it was effectively just that one click,
and we would get a new model.
So here's an example of the predictions it makes
and how it ranks itself.
So we've got this camera looking from the side view
to track the ball going across.
And you can see in this instance,
the ball's a bit blurry.
And that's why we needed to train a custom model,
because a generic pre-trained model didn't
recognize this as a football.
DALE MARKOWITZ: So do you play soccer, slash, football?
ZACK AKIL: No, I hate football.
I'm more into rugby.
DALE MARKOWITZ: Now that I understood Zack's technique,
I went to the tennis court to see
if I could use it to improve my tennis serve.
I used the Video Intelligence API
to track my skeleton, which let me calculate
the angles of my arms and legs.
And I trained an AutoML Vision model
to track the trajectory of the tennis
ball and calculated speed as it zoomed into the net.
The math revealed what I'd always sort of known
about my tennis serve--
it stinks.
But check out my blog linked in the description
to see how I'm using these stats to get better.
That's all for now.
See you next time.
[MUSIC PLAYING]