Hello everyone, my name is Yuhan Luo. I'm a PhD candidate at UMD's iSchool.
Today I'll be presenting this work, FoodScrap: promoting rich data capture and reflective food
journaling through speech input. This work is in collaboration with Dr. Young-Ho Kim, Dr. Bongshin Lee
Dr. Naeemul Hassan, and Dr. Eun Kyoung Choe. Food journaling is not something new these days,
it has become a prevalent approach for us to collect their food information,
gain awareness on their diet, and make informed food decisions. There are multiple ways to
practice food journaling, for example, we can use text input, barcodes, and photos.
Additionally, researchers have made a great effort in developing automated systems to capture our eating episodes
such as using proximity sensors and microphones. While these approaches mainly focus on capturing
food items, calories, and nutrients 11 0:00:59.200,0:01:03.840 12 such as condiment and preparation method 13 are important information for nutritionists to access patients' food intake --> 14 researchers also find that capturing eating contexts such as time --> social environment, 15 and feelings about food can help promote situated reflection on one's eating behavior, 16 In food science research --> understanding food decision-making is important for 17 developing a personalized diet and promoting mindful eating habits --> therefore, researchers often rely on 18 surveys and interviews to capture people's food decision-making in different scenarios in a retrospective manner. So how can we capture
this important information more efficiently? Using automated approaches, it is difficult
to capture all the details, but manual approaches, such as typing require too much input burden.
So how about speech input? 22 0:02:00.240,0:02:06.800 23 0:02:06.800,0:02:11.040 24 their mobile device --> and half of Americans 25 have used the voice assistant, In addition --> half of the consumers will 26 adopt speech-based shopping by the end of 2022, More importantly, we see the potential of speech
input for facilitating data capture in two aspects. First, we speak faster than we type,
so speech input can lower the data capture burden.
Second, we tend to be expressive 30 0:02:36.640,0:02:41.280 31 0:02:42.480,0:02:47.440 32 difficult to edit the data on the fly 33 and it often comes with privacy concerns, Although theoretically --> we know the pros and cons of speech input 34 little research has actually examined how speech input can be helpful in self-tracking contexts,
especially for capturing unstructured personal data such as food practice
In this light, we're interested 37 0:03:10.640,0:03:18.320 38 0:03:18.320,0:03:24.240 39 0:03:24.240,0:03:28.800 40 that captures everyday food practice, 41 The app first asks people to log their meal type, eating duration,
and a food photo, so that they can remember to log other details later after eating.
We draw from previous food science literature to design a set of guided prompts.
The purposes are to uncover food 45 0:03:48.800,0:03:54.480 46 0:03:54.480,0:04:00.080 47 0:04:00.080,0:04:05.440 48 0:04:07.120,0:04:12.880 49 0:04:12.880,0:04:16.800 50 also have specific and different eating 51 goals such as losing weight --> reducing sugar intake, or developing mindful eating habits, 52 At the end of the study --> we measured participants' perceived data capture
burden using the User Burden Scale, and did a debating interview to understand their overall experience.
Here I'm covering some interesting findings from the study.
If you are interested in knowing more , please refer to our DIS paper. 56 0:04:43.440,0:04:52.560 57 0:04:52.560,0:04:58.000 58 0:04:58.000,0:05:02.160 59 their skipped meals, This is because 60 for skipped meals --> they don't have to answer the same questions as for menus and snacks, 61 In addition --> each meal and snack entry included over 140 words.
Our participants provided rich details regarding food components such as dish name, ingredient types,
items, sauce and spice, and etc. They also described the preparation type such as toast,
bake, and steam, and detailed preparation methods including steps.
Additionally, they would provide contextual information such as the weather and taste of the food. For example
in this transcribed entry from P8, she mentioned the dish name: focaccia bread for her breakfast,
together with all the detailed ingredients that she used to cook this food, and she mentioned
spice and sauce such as olive oil, ranch dressing, and then she talked about the preparation type,
and additional contextual 70 0:06:07.520,0:06:14.320 71 0:06:16.240,0:06:22.560 72 0:06:22.560,0:06:27.840 73 0:06:29.360,0:06:34.960 74 0:06:34.960,0:06:41.840 75 0:06:41.840,0:06:46.480 76 judgment on his previous meals 77 saying that they were "junk food" and decided to seek something healthier
with fresh veggies, which kind of reflected his eating strategy relating related to health belief.
Moving to their capture burden: The results from the User Burden Scale shows that the data capture
burden was relatively low and our participants acknowledged this during the debriefing interview,
saying that speech input was easy and convenient. Specifically, P7 highlighted that if he were
to type the exact same information in text entry he would barely make a sentence or two.
However, participants brought several limitations of speech input, including re-recording effort,
which happened during the moment when they lost their chain of thoughts a nd had to start over
the entire recording. Sometimes speech input would cost extra mental load because there's
no visual feedback, so participants had to be extremely focused on what they were talking about.
Social constraints sometimes influenced participants' ability to use speech input,
for example, they don't like 89 0:07:58.880,0:08:06.000 90 0:08:06.000,0:08:12.240 91 0:08:13.920,0:08:20.400 92 0:08:20.400,0:08:25.600 93 0:08:25.600,0:08:31.440 94 0:08:31.440,0:08:37.120 95 0:08:37.920,0:08:44.640 96 0:08:44.640,0:08:50.160 97 0:08:50.160,0:08:54.720 98 0:08:54.720,0:09:00.320 99 0:09:00.320,0:09:05.840 100 0:09:05.840,0:09:12.560 101 0:09:12.560,0:09:18.320 102 0:09:18.320,0:09:24.640 103 0:09:24.640,0:09:28.640 104 visualization such as word cloud, 105 Given that some of our participants did listen to their audio recordings --> another opportunity 106 would be enabling a more efficient audio search, Additionally --> to complement the limitations of 107 speech input we can leverage multiple input modalities combining speech --> text, and photo 108 across different devices --> so that people can choose when to use which modality for what type of data, 109 With that --> I would like to end the presentation and take any questions, Thank you for listening.