I think there's always a debate on how much harder are outdoors compared to indoors. There's
all kinds of opinions out there, but I think this data-driven approach is really awesome
because it actually kind of shows that outdoor grades are typically sandbagged by... Grading climbs
is very difficult. There is no concrete standard of what makes a climb V4 and what makes a climb
V5. It’s very difficult to take different types of routes, different individual climbing styles,
and different individual body types into consideration in grading climbs. Indoor
climbs and outdoor climbs are also very different, so it’s hard to grade them by the same standard
as well. Is there a way to revolutionize how we grade climbs? Today I am very excited to
have Nathaniel from Climb On join us. He came up with a very interesting data-driven approach to
grade climbs with data science. Hey, I'm Nathaniel Okun. I'm a climber in Northern California in the
Bay Area. We got really lucky because we have access to this treasure trove of data through
this climbing app called Kaya, and Kaya is this app where folks can try climbs inside or outside,
and they log how many attempts they took, and this actually tells us a lot about my opinions
on each climb. If I take seven attempts to do the V6, and three attempts to do the V5,
and one attempt to do the V4, I'm indicating my preference or my opinion that the V6 is harder
than the V5, which is harder than the V4, and so thousands of people are using this app Kaya, and
this allows us to look at climbs in just a totally novel way. What we looked at is, one, how accurate
are the grades that setter set, and lastly we benchmark indoor climbs with outdoor climbs. The
nearest outdoor place where folks are climbing a lot at both my gym and outdoors is Bishop,
and so we look at a bunch of climbs from Bishop and we say: "Hey, you know Serengeti. That's a V5.
What is it at Dogpatch?" How do you actually use the data driven approach? We do what is called
tournament ranking, and it's also used by Google in their very early prototypes of a search engine.
Let's take a look at this example, so we say that we have two different climbers. One climber said
that climb 3 was harder than climb 2, and climb 2 was harder than climb 1, and then another climber
just climbed climb 4 and climb 2, and they thought that climb 4 was harder than climb 2, and so what
we do is we go through rounds of this tournament, and each round a boulder problem earns points for
being harder than another boulder problem, so in this example climb 3 is harder than climb 2,
and climb 3 is harder than climb 1, and so it's going to earn two points, one point from
each of these climbs. Then if we look at the next stage, so now when climb 3 earns points,
it earns more points from climb 2 because it is defeated in this tournament, and so it's going
to earn two points from climb 2 and one point from climb 1, and we see that climb 4, which in
this round is of equal difficulty to climb 2, is going to earn more points than last round
because we've now realized that climb 2 is harder, and so if we look at the last round of this
tournament when things resolve, we see that climb 3 has established itself as by far the hardest,
and climb 4, which the data shows harder than climb 2, has finally established itself as the second
hardest problem. And so for every single climber, we look at all the climbs that they've climbed.
We look at all the attempts that they've taken. For Dogpatch at least, which is the largest
data set about 700 climbers, about 350 climbs, and that turns out to be something like a bit
more than 10000 data points, and so we create this enormous tournament. We run this sort of
tournament for about a hundred rounds. Question, so is the how much harder, is that relationship
captured in this algorithm, like meaning let's say, you know, say climb 3 it took him 10 tries,
climb 2 took him 9 tries, and let's say climb 1, it only took him one try. Is that relative difficulty
of the climb been captured in this? Yeah, so uh we totally do take it into account. It's unclear
if that's the right decision. Look, if a climb takes me 10 tries, and another one take takes
me five tries, is it twice as hard? Probably not, and so what we do to try to account for that is
we use a log scale. And so if one climb takes me 10 tries and the other one takes me 5 tries,
I have a, you know, a formula that I tuned on the data. You know, instead of giving one vote, we give
that harder climb 1.3 votes. Let's say you know climb 3 is a V7, climb 2 is a V6, climb 1 is a V5,
but climb 4 could totally be a V10. For this case, climb 4 would be ranked lower than climb 3.
That means V10 would be ranked lower than a V7, so that doesn't seem to be right, right? Yeah, so
that's a great point. The lucky thing here is that we have about 10000 data points, and what
we do is we exclude climbs like that V10. That's only been climbed by one person. That is just not
having enough interaction with the graph. I see, so you actually pruned out the outliers. So the first
thing that we tried to look at was are the grades that setters give to problems in a bouldering gym,
are these actually accurate? And what we found is incredibly cool. We found that a grade actually
is plus or minus one, pretty consistently, so if you climb a V3 that was set by your gym, it's
anywhere between a V2 and a V4. And we found this consistently for every single grade, and notably
the hardest climb at every single grade level is harder than the median climb at the next level,
so if you climb the hardest V4, that's going to be harder than the average V5. At least to me that's
a really empowering notion. Is the high variance inherently that's going to happen no matter how
good of a job that the setters do? Or is this something that potentially that the variance could
be lower? I think that it's quite hard, and even if you did it perfectly, you would still see the
range be about a grade level from like say V4.5 to V5.5, and we're seeing it as just a little larger
than that from V4 to V6, so they're doing like a a good job, but not a perfect job. One really cool
thing is that we can take a look at an indoor gym, and then we can take a look at an outdoor
bouldering area, and we can see which climbers climbed at both and use those to rank both outdoor
climbs and indoor climbs, and sort of benchmark an indoor gym based on outdoor climbs. Also caveat,
there's not that much data here. I think we're looking at something like, you know, 500 data
points, and so take this all with a grain of salt. This is really just for fun. For Bishop, we see
that grades have a really, really high variance, and that's partially because we don't have a lot
of data, but it's also because outdoor climbing is just different than indoor climbing. If we take a
look at this graph, like look at Buttermilk Stem, I don't know if anyone in the audience has done
Buttermilk Stem, but it's kind of inscrutable, right? You know it's a bit of a tricky problem,
and maybe when you get it dialed, it feels like V1, but I don't think it's the sort of problem
that flashes like other outdoor V1s. The reason why this is rated so high is that there are two
folks at Dogpatch who project v9, and it took them a lot of attempts, and so the tournament thus
compares this V1 to all of the V8s that they flashed at DogPatch and says: "Wow! Buttermilk Stem
really hard." Something that's much more accurate is taking a look at Solarium, which is a V4,
and we actually have a lot of data for Solarium. About 20 folks at DogPatch have gone and climbed
Solarium, and so this is one of the few regrades that we're actually confident about. We think that
Solarium is something like a really hard V5, or an easy V6 at Dogpatch. This is super interesting
because I think there's always a debate on how much harder are outdoors compared to indoors.
There's all kinds of opinions out there, but I think this data driven approach is really awesome
because it actually kind of shows that outdoor grades are typically sandbagged by two grades
compared to indoors. Yeah, I don't know. Maybe more accurately, indoor grades are kinder. I think one
of the benchmarks for V3 in Hueco Tanks and it's Sign of the cross. I've climbed Sign of the Cross,
and I've climbed a couple of V7s at DogPatch. I don't know. Maybe Sign of the Cross is like
physically easier, but it's sharp, and it's cold, and the landing is from 20 feet up, and I'm a
little spooked when I get 20 feet up, and I think all of those contribute to the fact that it took
me way more attempts than any indoor climb. Is V3 V7 at Dogpatch? I don't know, but I think for
me, who's an occasional outdoor climber, it kind of is. For the Bishop climbs that you list over here,
so the only one that I actually did before is the Heavenly Path. You know I feel like the fear
factor up there makes it more than a V1. I even heard about stories about people regularly falling
off the Heavenly Path V1 and have to be sent to the hospital, so the fear factor is definitely
real. I remember when I was climbing it, but I was like the shaking. I was like, I can't fall.
I'm in a no-fall zone. That was so scary. Yeah, I feel invincible climbing indoors. I'll take any
fall, and outdoors it's a different animal. This is really, really interesting, and I really hope
that there's more data for other gyms, and also other outdoor climbs too. I find that there isn't
a lot of action on Kaya on my home gym, and also the outdoor climbs near me basically. See because
I remember I was projecting a V7 like before I injured my finger, it felt way harder than indoor
V7, so you know it'll be awesome if I can have a similar analysis. According to this, if it's a V9,
that's like, you know, why am I even, you know, projecting that, right? That's like way out of my
abilities. Is there any more insights that we can look at the data that we can think about? Yeah,
so I think one really interesting thing about this data is looking at slabs, and so in this dataset
what we do is we say if something takes you more attempts, it's harder, but that's not really true
for slabs because I'm sure that we've all had this experience where we find this inscrutable slab,
and we have a lot of fun, and we figure it out, and it takes us 20 attempts, but then we do it
again and again, and we dial it and it doesn't take that much physical strength necessarily,
but it's a reflection of our maturity and climbing technique, and it's really fun,
versus we have another climb that we're able to flash and we try really hard on it,
and we use our forearms a lot, and we use our back a lot, and it is just hard to climb every time.
Which of these climbs is harder? That's a great point. The slabs are not necessarily harder in
this situation. This is a really interesting food for thought. I think what this analysis has really
taught me is that climbing is just this very unique joyful thing, and like the grades are fun.
There's just so much depth and uniqueness to every single climb. Yeah, I just came out of this more
excited about climbing than ever. Yeah, I think climbing is definitely a really complex sport.
Let's say compared to like weightlifting, you lift how much weight; you lift how much weight.
But then climbing there's so many variations, different types of climbs. It's really hard to
standardize the grades, and then now we try to use the data-driven approach to get a more accurate
grade. But even that, there are still a lot of points that's hard to capture into the data,
so, yeah, climbing is really a complex sport. You also have a blog where you post all these
analyses, right? Yeah, basically I'm just trying to write more about climbing and data,
and, you know, just really interesting things. So what kind of future works that you planned?
I think something that's really interesting is we can take a look at individual climbers and
we can make suggestions as to where they can work on their weaknesses. We can take a look
at my own climbing for example and identify that I really like powerful crimps, but I struggle on
compression problems. I would definitely love to see like my climbing profile. Like, hey, you really
tend to use crimpy climbs, and you struggle on things that use a lot of other muscles,
and we can generate this report automatically for you. I think that's a really fun corollary
to this work that I plan on writing on. That sounds amazing! I'm pretty sure all the viewers
will be super excited about your future work, and I really appreciate you coming on to the channel
and present your work, super, super interesting. I had a lot of fun. To anyone listening, like please
comment let me know like what would you like to see with this data. You know what mistakes did I
make? I would love to improve and interact with the community, so, uh, please just let me know.