Hi, I’m Young-Ho Kim.
I’ll present the MyMove project,
a collaboration between the University of Maryland at College Park,
Microsoft Research, CJE SeniorLife, and the Pennsylvania State University.
Activity tracking devices with sensing technologies can automatically capture activities from sensor data.
However, such technologies are usually built using younger adults’ activity datasets,
and may not be applicable to older adults.
The physiological characteristics and movement patterns of older adults are highly different
from those of younger adults.
Therefore, the current activity recognition models may yield inaccurate results
when applied to older adults.
To develop activity tracking technologies that are accurate for older adults,
we need an activity dataset about these people,
including sensor data that reflect bodily movement,
and the ground truth labels that describe the performed activities.
Since it is challenging to collect these labels in an automated manner,
we advocate the involvement of older adults
in the process of collecting data within their own environments.
As a means of an effective activity labeling method for older adults,
we focused on the speech input on a smartwatch.
Smartwatches incorporate many sensors that can record various aspects of human activity,
and notifications are easily delivered to the wrist with vibration.
Speech input is beneficial when combined with smartwatch;
it allows people to specify complex information by phrasing as they wish,
and requires little or no visual cues on a small smartwatch screen.
We designed and developed a smartwatch app called MyMove.
People can always start verbal reporting on a watch face.
On the recording screen, people are asked to describe the activity, associated timespan,
and perceived effort level, in a single utterance.
Considering individualized preferences, we allowed people to use
both the screen and the physical buttons.
Also, the watch sends a signal at random time, to prompt people to capture the current activity.
Let’s see the demo video.
This is a prompted reporting scenario.
While the user is reading a book, she receives a notification signal on the watch.
She touches the mic button, and says,
So I've just been relaxing, reading, since about 2 p.m.
Then she finishes the recording.
Now a voluntary reporting scenario.
The user came back from taking a walk.
She pushes the side button to start reporting.
Just went on a brisk walk for about 45 minutes.
Then she finishes the recording.
Using MyMove, we tried to answer these two research questions:
How do older adults capture their activities using speech on a smartwatch?
And how useful are their verbal reports as an information source for activity labeling?
We conducted a deployment study with 13 older adults who are using smartphones.
We deployed two devices, an Android smartwatch with MyMove installed,
and the activPAL activity monitor that is attached to a thigh.
activPAL served as a ground truth source for locomotion activities.
During the first 4 days of the study, we asked participants to get used to wearing the devices
and recharging the watch at night.
For the following 7 days, participants not only wore the sensors,
but also collected verbal reports.
Here, I will share some of the key findings from the study.
First, I’ll cover participants’ engagement in the data collection,
Characteristics of the captured activities,
And perceived strength and challenges based on the participants’ feedback.
In the interest of time, please refer to our paper for other interesting analyses.
Participants were highly engaged in the data collection.
Each of them collected about 13 verbal reports per day.
Also, participants wore the devices covering most of their waking hours.
Even most of them kept an activPAL monitor on their leg
throughout the entire data collection period.
This is an example verbal report from participant P6,
“Spent the last 12 minutes,
eating breakfast, seated in front of the TV.
Minimal level of effort.”
It contains the time cues,
and two simultaneous activities, eating and screen time.
She also added the effort level information.
We identified 9 activity types and 29 semantics from the verbal reports.
The most prevalent one was housekeeping such as cleaning, preparing food, or gardening.
Interestingly, participants explicitly mentioned brief movements
that are not a deliberate exercise, such as moving around the kitchen and going up and down stairs.
As I showed in the example, participants tended to describe multiple activities in a single report.
About 40% of the verbal reports contained two or more activities that happened sequentially or simultaneously.
For example, screen time activities were often combined with other activities,
such as Housekeeping or exercise.
For example, participants often engaged in the online exercise classes on Zoom.
Our participants favored the flexible ways to combine two different reporting methods,
or to choose between the screen and physical button modalities.
For example, one participant remarked that “it was very efficient watch.
It was nice that you could just either touch the screen or the physical buttons.”
Participants had different preferences on the interface, which supports our design choice.
Although our participants reacted positively to the free-form phrasing of multiple information components,
they also expressed challenges.
For example, being cognizant of the verbal reporting, they had to be conscious to remember the start time when they started a new activity
so that they can report it afterward.
Through this study, we demonstrated how speech input facilitates older adults
to collect activity labels on a smartwatch.
Our older adult participants showed a great engagement in the data collection process,
producing a rich and valid dataset.
We believe developing activity datasets with older adults will open up new opportunities
for building personalized activity tracking technologies for older adults.
For more details, please refer to the paper and our website.
Thank you for listening to this talk.