I am glad to present our work entitled “Investigating Preferred Food Description Practices in Digital
Food Journaling”
Prior research as well as commercial apps have sought to support several food and health
related goals.
Broadly speaking, food journaling can be used to support more quantitative goals, such as
monitoring calorie and nutrient consumption to manage diseases or weight;
as well as awareness and mindfulness goals, such as to make healthier food choices and
identifying food intolerances
Food journaling has typically been supported by technology through lookups to food databases
and barcode scanning of food packages, with a focus on the nutritional information feedback.
However this can incentivize eating more packaged foods and searches can be cumbersome, inaccurate,
or unsuccessful.
Various efforts have then sought to support more open modalities, or input methods, in
order to make journaling less burdensome and more descriptive, such as taking a picture,
recording a voice memo, or saving open text descriptions.
Other efforts have sought to leverage automation to detect eating moments through sensors,
as well as natural language and image processing to identify food ingredients in photos and
open texts.
Given the diversity of modalities that have now become available, as well as the gamut
of devices people use throughout their day,
we sought to understand how people would ideally prefer to describe foods when given more flexibility
and under fewer technology constraints around recognition, accurate entry, and desire for
recall.
This understanding will enable us to leverage these varied modalities to design food journals
which better align with how people wish to use them
So to answer this question we built ModEat, a flexible and lightweight cross-platform
prototype for journaling on phones, computers, and voice assistants, that supported different
input modalities.
The ModEat on the phone allowed for scanning of barcodes, photo taking, open text input,
simulated database searches, saving website URLs (such as for a recipe), and voice memos.
ModEat on the computer supported the same modalities. The Alexa and Google assistant
supported simple conversations, such as asking to record a food description or to read the
previous entry.
We deployed ModEat with 15 participants that had a mix of quantitative and awareness food
related goals.
They used ModEat for two weeks and answered a survey for each entry to detail the context
related to the meal. We also conducted interviews before and after deployment.
Our analysis involved a mix of quantitative and qualitative methods of the interviews
and participant’s 1008 food logs.
Finally, to understand gaps between description and automated recognition, we further investigated
how participant’s logs were and were not interpretable by commercial natural language
processing (NLP) and food image classification services.
In analyzing food logs and interviews, we identified preferences and strategies our
participants used in recording their foods and that revolved around different ways of
describing granularity, specificity, amount and context.
In particular, this revealed a high variance in how they preferred to describe their foods,
both between individuals based on goals and among individuals based on their foods and
circumstances.
Also, image classification and NLP services interpreted logs fairly well, but we observed
missed opportunities in their outputs.
I will now summarize some of our findings, but encourage you to read our paper for more
details, such as how context was referenced in some of the logs.
Participants varied in how granular they grouped food items. Most inputs were of single food
items, such as 1 cup blueberry, but some decomposed foods in their ingredients, while others aggregated
distinct foods, such as main and side dishes.
We observed that the type of input modality correlated with granularity levels, such as
database searches tending to be of a single food item or text descriptions having more
aggregated food inputs.
This was aligned with participants perception of modality use, such as P6 that said “Database
search was most useful generally because it’s keywords, so a lot of the time I put cutie
tangerine because we gave a ton of those”
conversely, participants often viewed text inputs as useful for describing lists, such
as P9 that said “I easily put down multiple items at the same time in text description”
We also observed that participants tended to aggregate foods more often when eating
with others versus when eating alone, perhaps suggesting that they tended to aggregate when
in social situations where they wanted to journal multiple items quickly.
Also, there was no significant impact of food journaling goals on granularity levels
Participants also described foods with different specificity levels of detail, even of foods
aggregated in a same input.
A minority of logs consisted of generic ingredient descriptions, such as “dumplings”…
Instead, participants tended to describe foods in ways which were specific enough to distinguish
from other foods or ingredients, such as chicken or rice. Other logs went further to describe
varietals of same food, such as “roast chicken breast”.
There was no significant correlation between goals and specificity levels.
Instead, participants reported that a meal’s circumstance influenced the level of detail.
For example, P13 said that “If you eat out and you are logging what you had, then more
than likely, all you can say is, ‘I had Irish stew,’ because I don’t know what’s
in it.”
Participants varied also in whether and how they described food amounts.
Some inputs had formal scale references, such as ounces or grams, while others referenced
a numeric quantity. Still others used non-standard measures that made sense to them, but might
not be interpretable by others, such as “small”, “big”, or “entire”.
Other descriptions referenced brand package sizes, or containers with variable sizes.
However, more than half of described foods had no amount clarification at all.
Participants with quantitative goals were more likely to clarify amounts than participants
with awareness goals.
They explained that to get more accurate feedback they would really need to inform the amount
measure, contrasting with participants with awareness goals that said it was easier to
just describe the food.
In regard to NLP services, food items were generally identified correctly, with about
80% of descriptive entries correctly being interpreted and returning relevant nutritional
information for every food item.
Only around 4% completely failed, not matching any foods or wrongly identifying foods. For
example P15 journaled “2 tablespoons salad topper”, and this was interpreted as salad.
Many varietal descriptions used adjectives to describe food names, which could lead to
misinterpretations and ambiguity, such as “salmon cakes” or “banana tea” being
interpreted as separate items instead of flavors.
Interpretation was also impacted by specificity and amount description strategies, which we
detail more in the paper.
Participants varied in how they used images to depict their foods, such as arranging for
aesthetics and clear amount compositions, use of stock images, or pictures of packages.
Food characteristics would typically be clear, but some pictures were of wrapped foods, empty
plates, or unclear liquid substances.
Classification services tended to correctly identify at least one food element in most
images with clear foods, but some services mostly identified background elements, such
as keyboard, furnitures, and appliances or food containers, such as a bowl or plate.
However, image classification services would typically ignore labels and words, such as
the salsa types in this first image. Also, shape and color influenced the recognition,
such as this third figure of a cheese bites package being interpreted as beer or chips.
Moving to what we learned from deploying ModEat, we observed that some styles of capturing
foods were not precise and could introduce ambiguity or unclarity. Which in turn can
lead to challenges when reviewing data or reflecting on one’s eating behavior.
A possible way to mitigate variance and unclarity could be to implement conversational journaling
to surface what was recognized and enable correction and incrementation.
Such a system could then encourage greater specificity, clearer amounts, or further details
to clarify ambiguities and diminish variance.
On the other hand, methods aimed to address variance could introduce burdens to the already
demanding food journaling domain.
Allowing for flexibility or even treating variance as a design opportunity may be particularly
beneficial for people with awareness goals.
In this sense, interpreting past ambiguous entries could be leveraged to provide deeper
reflection around circumstances under which the data was collected, potentially highlighting
the social and cultural celebratory nature of food.
Finally, we also observed an opportunity for classification models to be trained for not
only recognizing food in text and images, but also identify contextual cues, such as
references in text and background information in images.
Even if not identified with high detail or accuracy, these models could help point to
fun or personally meaningful experiences, such as surfacing restaurant names or household
objects visible in communal dining.
Further mining of information embedded in photo metadata or passively recorded (such
as location and time) could further provide context to complement reminiscence and reflection
for both quantitative and awareness goals