Rachid: Thank you for coming back to the "Made by Google" podcast.
I'm your host, Rachid Finge.
And today is all about falling.
Happens to everyone.
I fell off the stairs, actually, last December
and found out I broke my foot.
And still, I got to say, I got lucky because I was able to get up myself.
And, of course, that's not a given.
What if you're alone or cannot get up?
Wouldn't it be great if you could get help in such a desperate situation?
Well, the good news is that Pixel Watch now can help,
thanks to a new feature called fall detection.
Pixel Watch can detect when you fall and get help.
It could actually save lives.
So let's talk more about fall detection in today's episode
of the "Made by Google" podcast.
Our guests know everything about it.
I mean, they made it.
So with us are Paras Unadkat from the Fitbit team
and Edward Shi from our safety teams.
[upbeat music]
Paras, Edward, welcome to the "Made by Google" podcast.
Please tell us a little bit about your roles at Google
and how you ended up here.
Paras, maybe we can start with you.
Paras: Yeah, so I'm a product manager on the Fitbit team.
And I was a product lead for the fall detection feature.
I work on a number of our sensors and algorithms-based features,
basically using the sensing technology on wearables
along with our amazing machine learning capabilities at Google
to create great features for our users.
Rachid: Amazing. And what about you, Edward?
Edward: Yeah, I'm also a product manager on our safety team,
on the Android and Pixel safety team.
And I've been working on products that aim
to help users feel and be safer, such as car crash detection and,
in this case, fall detection.
And when I heard about the feature with fall detection,
Paras and I basically thought it'd be a great way to collaborate,
given the cross section between the algorithm
and the safety themes of the feature.
Rachid: Today's guests work together to improve your personal safety
using the latest in technology.
Paras Unadkat joined Google six years ago,
working on machine learning in the Google Ads team,
but then joined Google Earth, and now Fitbit,
where he works on the cross section between sensors and health,
making sure those sensors do helpful things.
If Paras' work is making sensors detect falls,
then it's Edward's work to make useful things happen when a fall is detected.
Edward Shi works on our safety teams.
You might have seen the personal safety app on your Pixel phone.
That's Edward's team.
So often, the whole is greater than the sum of its parts.
And the work of Paras and Edward is one of the most powerful examples.
Let's hear from them how fall detection was developed
and how Edward went above and beyond, taking one for the team.
I hope you'll enjoy our conversation.
Let's bring this a little bit into reality.
I actually fell off the stairs last December.
Now, let's say that would have happened somewhere this week.
I was wearing my Pixel Watch, and I have fall detection enabled.
Edward, what happens when I fall and I'm wearing my Pixel Watch?
Edward: Well, first off, I hope you're OK.
I think you're OK, and I'm glad to see that.
Rachid: Yeah, pretty much.
Edward: When Pixel Watch first detects a hard fall,
it'll wait and see if it detects you moving.
Now, if you don't move for about 30 seconds,
it will start to display a fall detected on-screen notification
to check in and see if you need help.
Now, during this time, it'll also vibrate and sound an alarm,
so you know that Pixel Watch has detected a potential fall.
And you can, of course, respond accordingly.
So in the case that you need help, you can tap, "I fell and need help,"
to be connected immediately to emergency services,
or if you're OK, and you don't need help, you can tap, "I'm OK,"
on your watch face to dismiss that notification.
Rachid: All right.
Edward: Now, otherwise, if you're unable to respond for about a minute,
Pixel Watch will automatically attempt to call emergency services
and play an automated message that lets them know that the watch detected
a possible fall, you didn't respond, and also share your location,
so emergency services can know where to send help.
During this time, if you recover or you're able to speak,
you can also speak to an emergency operator yourself and, if possible,
let them know that you need help or not.
Rachid: Amazing, OK, and we'll get back to that 30 seconds
and additional 30 seconds a little bit later.
Now, Paras, it's, I guess, your job to make sure
that the watch detects the fall in the first place.
So how does it do that?
Paras: So the fall detection feature uses an algorithm
that works based on the motion sensors on your Pixel Watch.
And it uses machine learning to differentiate
between the motion signatures of fall events--
like you were saying, falling down a flight of stairs--
compared to things throughout your day-to-day life
that might look like falls, so things like exercising,
doing burpees, kettlebell swings, and things like that.
There's no way to build a perfect system,
but by collecting a lot of data and training our models
across all these different data sets and configurations,
our model does a pretty good job of differentiating
between different types of motion patterns.
Rachid: I find it interesting because, like you mentioned, burpees,
which give me a sort of headache, I guess for you as an engineer,
that gives you a completely different kind of headache,
where you need to be able to distinguish it from an actual fall.
So how do you keep the two apart?
Paras: So it really just comes down to the data sets that we've collected.
And by collecting a large data set of real and simulated falls,
along with a huge data set of real-world activities--
things like burpees, people driving in cars,
different types of vehicles braking hard,
all sorts of things like that--
we've been able to build a model that does a really good job
of being able differentiate and learn the difference
between those different motion signatures.
Rachid: OK, and we'll get back to how we train the model a little bit later.
But, Edward, we were just discussing the 30 seconds notification
and the 60 seconds where, actually, emergency services will be called
if you're unresponsive after a fall.
So how do you decide that it's 30 seconds at first
and then an additional minute to have these things kick in?
Edward: Yeah, and to start, I did want to say,
any time when a fall is detected,
users can immediately contact emergency services
by tapping the "I fell and need help" button on the in-screen notification.
And that's really important to us.
But that being said, we wanted to ensure that users have the option to respond
if they are able to after a fall.
For instance, we know sometimes when a user falls,
they're able to recover themselves, and they don't actually need
to call emergency services.
So we wanted to give them that opportunity
to dismiss that call during that 60 seconds,
essentially, if help isn't needed.
We also know at the same time,
unintentionally calling emergency services can be
a really stressful experience for both the user,
and it can be burdensome to emergency call centers.
So with the 30 and the 60 seconds, this gives us an opportunity for users
to indicate that they're OK and cancel and stop automatic calls,
while algorithmic at the same time,
if we detect their significant movement,
we can also stop that automated call from going through.
So this is our approach, and it's, in essence,
to balance getting users help quickly while minimizing stress
caused by unintentional calls.
Rachid: And how does that go like during the development of the product?
So at some point, you're gonna say, it's not 25 seconds, not 35.
It's gonna be 30. So how does that come about?
Edward: Yeah, so I think one of the big things
that we were trying to do with our feature was sort of make sure
it's providing as much value as possible,
being able to detect when our users actually need help,
and when they're just doing burpees,
or when they're just getting up really quickly after an event happens.
So because of that, we kind of built in the stillness threshold,
where we sort make sure that, hey, before we place any emergency calls
or do any of that, or even kind of just bother the user
and say that, hey, it looks like you fell,
we just make sure that they're actually motionless or they're
underneath a particular stillness threshold after that fall event.
Rachid: Right.
Edward: And the idea is that, if the user actually needs help,
and they're actually unable to make a call for themselves,
we'll be there for them in that situation,
but we won't spam them with any alerts or anything like that otherwise.
Rachid: Right, so let's say I'm playing football, and--
I actually mean soccer, so that's football here in Europe--
and I'm wearing a Pixel Watch, and I fall,
but I stand up immediately to continue the game.
Then I will not get any notification because Pixel Watch noticed, like,
OK, it seems to be, you're fine, you're moving again.
Edward: We know you're OK, exactly. We know you're OK.
We don't want to take you out of your game.
We don't want to cause a bunch of chaos, so we just won't trigger at that point.
Rachid: Perfect, now, for the people who don't know, within Google,
we have this tradition of what we call dog-fooding,
which is where we test products that are about to be launched
with the whole Google population,
so the product teams get as much feedback as we can.
Now, I'm just wondering, Paras, did I miss any, like,
dog food invitation for maybe, I don't know,
volunteering to fall and train your model?
I'm just so curious how you gather data about falling
likely without-- hopefully without hurting people.
Paras: Yeah.
No, you definitely didn't miss any dog-fooding communications
about us collecting data from Googlers.
We wanted to collect--
just make sure we collected all of our data in a way that was safe
and nobody got hurt.
So no Googlers were harmed in the creation of this product.
Rachid: All right, and what was in the actual way of doing it?
How do you gather that data?
Paras: Yeah, it's a really good question.
So obviously, the problem of gathering fall data is quite difficult.
You can't exactly ask people to just go and take falls for you.
Rachid: Sure.
Paras: And there are a lot of different things
that impact how a fall looks to a set of sensors, right?
So if you ask somebody to go on a crash pad and fall there,
that doesn't look like a real fall because there's a crash pad there.
The level of impact is significantly attenuated.
The thing that causes harm is the thing that you're actually trying to detect,
which means that it's really difficult to get people to actually do that.
So we had a few different ways of going about this.
Initially, early on in the program, we worked with a number
of different university labs and external fall labs
to really just understand what a human fall looks like.
Collected a ton of data that way--
a lot of harness arrested falls, had people come into the lab,
set them up on a bunch of different elaborate rigs
to kind of have them lose balance, see how they react to that,
figure out what that all looked like, and really understand
what the different components were to a fall,
and different types of falls, and that sort of thing.
So we collected a lot of data in that way.
We also had other kind of mechanisms for collecting
real-world and simulated data.
And one of the biggest problems that you run into in building a model like this
is the problem of data set variability, which is you collect a number of falls,
but you can't collect every single type of fall in the world.
And there's so many different ways and configurations in which
people might fall and might be harmed.
So one of the ways that we approached this was that we started a partnership
with the Google research team to use AI computer vision to take videos
that we'd collected of people falling and use those videos to map them
into a simulation space and simulate the physics of different fall types
to really augment our data set of real-world falls.
So we still had that big data set of real falls.
But to make it even bigger and just collect this really massive variety
of different falls to really make sure that our algorithm was picking up
on all the different varieties across different scenarios,
we were able to tune things like different lengths of limbs,
different body types, different weights of different parts of the body,
different surfaces of people falling on, slopes,
just different ways of falling to really make sure that we had broad coverage.
And then one of the more interesting ways, I think,
that we were able to validate that our algorithm was actually working
on real-world data, we worked with a team
of stunt doubles for a week.
Rachid: Oh, really?
Paras: Basically just directing this team of stunt doubles to say, hey,
here's, like, different types of falls that we want to validate against.
Rachid: That's amazing.
Paras: And it was like-- it was incredible.
But I think a combination of all these things helped to give us
a lot of confidence that our algorithm and our feature
is gonna be working really well and actually being able
to detect these types of events when they happen in the real world.
Rachid: I think we can, hopefully, safely add a disclaimer
that no people were harmed during the development...
Paras: Yes. Rachid: Of this feature, right?
Maybe that's a good thing to say.
Paras: Done very safely and with all the right kind of safety precautions
in place, but really with an eye towards collecting
as much real-world, realistic data on this as possible
in the safest possible way.
Rachid: Excellent, now, Edward, you also worked with the safety team
on something like crash detection in the car.
I'm wondering, like, from a product perspective,
when do you decide that the system is good enough to go to real-world people?
Is there something you can say about that?
Edward: Truthfully, the balance is a little bit nuanced,
and we're always looking at the data in particular.
And so a lot of the different criteria that Paras looked at for fall detection
also applies in similar ways to crash detection.
We really look at, OK, what are the different scenarios
that we want to capture.
We want to make sure we capture or we optimize
for crashes above a certain speed, for example, et cetera.
And so when we have our data, and we look at those particular scenarios
that we're really optimizing for, we want to make sure both ends--
of course, an actual crash is detected,
and then we also, at the same time, want to make sure
anything that may look like a crash, but isn't a crash, is also mitigated.
So it's striking a good balance between those two.
And when we look at the data, we want to see that
before we actually release to the public.
Rachid: Amazing, and, of course, fall detection new on Pixel Watch
for people who own the watch.
I'm just wondering, from maybe your personal experience testing it,
have you already experienced a benefit of it,
or have you seen people maybe testing it that already were happy
that they had the feature on the device?
Edward: It's funny that you mention it.
When Paras said that no one was harmed,
I did take some voluntary falls just out of my curiosity.
And I may have had like a bruised elbow, but it was worth it.
In that sense, I've been fortunate that I haven't had an actual fall,
but I'm definitely relieved.
And I have that extra peace of mind to know that if I did take a hard fall,
I trust and believe in our product to help me if I need it.
Rachid: Amazing, definitely you took one for the team,
so that's absolutely great.
Now, we close every episode of the "Made by Google" podcast
with a top tip for our listeners.
[upbeat music]
How can they get the most out of fall detection?
Paras: I think one big thing with fall detection in general
is that a lot of times, people have this image of falling
as a thing that is really built for people at risk of falling.
So you kind of imagine, like, those old Life Alert ads,
like the "I've fallen and can't get up" type thing.
But really, falls are a much, much broader,
widespread problem, to your point about tripping
and falling down a flight of stairs or doing household chores,
and you're climbing up on a stool for something,
and the stool tips over, and you fall and hurt yourself.
So there are all sorts of situations that you can get yourself in
that I think a feature like this could be really valuable for.
We're very focused on making sure we kept all that
in mind developing this feature.
So it really, I think, is a feature that's built for everyone
and can really add this element of peace of mind and safety,
no matter who you are.
Rachid: Amazing, and indeed, I think especially useful for people
who might have a harder time getting up, definitely the case.
And, Edward, to use fall detection, anything you need to be mindful of?
Do I need to enable anything on the device or in settings?
Edward: Yeah, tactically, I'd say, for fall detection specifically,
you want to make sure you've granted personal safety location permission
on your Pixel Watch.
We can still operate without that, but in the case that you do fall,
we want to make sure we can share your location with emergency services,
so they know precisely where to send help.
So that's my top tip for fall detection.
And then separately, at the same time,
I know we talked about crash detection here.
Please feel free to check out our other safety features available as well.
We have a number of them to help give you peace of mind,
from crash detection to emergency location sharing and emergency SOS,
so all to hopefully help everyone be a little safer.
Rachid: Excellent, Paras, Edward, thank you so much for your time.
I hope you stay safe out there while you undoubtedly test
new personal safety features for Pixel Watch.
Thank you so much for talking to us.
Edward: Thank you. Paras: Thank you.
Rachid: A big thank you to Paras and Edward.
Check out the personal safety apps on your Pixel phone and Pixel Watch.
And yes, great advice to grant the personal safety app
a location permission so that, if it has to call emergency services
on your behalf, it can tell them where you are.
We have some more great episodes of the "Made by Google" podcast coming up.
So if you don't want to miss those, best to subscribe to our podcast.
Next time, we're talking about talking on the phone.
Catch you next time. Take care.
Talk soon.
[upbeat music]