Hi, I am Young-Ho Kim, a postdoctoral associate at the University of Maryland.
I will present the Data@Hand project, which was done in collaboration with Bongshin Lee,
Arjun Srinivasan, and Eun Kyoung Choe.
Smartphone is a commonplace to view and access the self-tracking data,
as fitness trackers are usually accompanied with smartphone apps,
which provide data visualization to explore the collected data.
Due to the small smartphone screens, it is challenging to explore personal data on a smartphone,
by comparing different time periods or navigating through the timeline.
For example, typical fitness apps allow us to choose the scope size from a given set.
In addition, navigating to specific dates is surprisingly limited,
...often making us infinitely scroll.
To overcome this limitation, we incorporated natural language.
Expressing time and ranges in natural language is flexible and expressive.
...and people already know diverse ways to express date and time.
We combined speech with traditional touch interaction,
to leverage the synergy of these complementary modalities.
Touch and speech have different strengths in the smartphone context.
The novelty of this work is that we supported the combination of touch and speech,
in the context of self-tracking data on mobile devices.
Let’s see the demo video.
I open Data@Hand, and it shows the past seven days of my data.
I will extend the scope.
“January 1.”
Oh crap, I see my step counts dramatically dropped since mid-March, the COVID-19 lockdown.
“Days I met my step goal.”
The highlighted days have gotten sparse since March.
What about other activities?
The sleep ranges also dropped since March, similarly. I will take a detailed look at it.
First, I will check my recent sleep patterns. “Sleep range of this month”
“Days I woke up earlier than 7:30 AM”
Only five days? How terrible.
Now I’m curious how my sleep differs from last year’s.
“Compare with last August”
The average sleep schedule of this month is shifted by an hour late,
and the bedtime and wake time are more irregular this month.
What about right before the lockdown?
“February 2020”
S imilar difference.
Now I want to see the impact of the lockdown from the monthly trend.
“Show 2020 by month”
I can see an upward trend since May. I’m going to bed earlier,
and waking up earlier. It’s a good sign.
What about other activities?
We conducted an exploratory think-aloud study with 13 long-term Fitbit users
across the U.S.
Participants freely explored their Fitbit data using Data@Hand for about 20 minutes.
We found that participants strategically chose the modalities in different cases.
Using speech was generally received to be fast and intuitive.
Paticipants actively used speech for switching time periods and executing long and complicated commands.
Participants combined touch with speech when they fine-tuned the parameters on the screen,
...such as when they modify one period while comparing two.
For more details on the results, please refer to our paper.
With Data@Hand, we demonstrated that supporting flexible time manipulation can powerfully
facilitate the visual exploration of personal data.
Also, our participants could quickly learn and adopt multimodal interactions
...in exploring their data on smartphones.
In closing, we invite the HCI community to engage in the multimodal interaction research
for personal data visualization.
For additional details, please refer to our paper and the website, Thank you.