Hi My name is Ryan, and I’m a student at Northwestern University.
In this talk, I’d like to share with you a way to imbue context-aware applications
with a richer understanding of human situations.
Context-aware apps have the potential to shape how we think about our everyday situations
and the activities we can do there.
For example, these applications could help us find opportunities for shared experiences
across similar situations to connect with our friends.
It's becoming easier to create simple context-aware applications,
due to better sensors and algorithms for detecting context.
For example, mobile phones can sense a user’s location,
and use a Place API to know that a user is located at a restaurant serving coffee and tea.
However, one thing to consider is how we can get these apps to trigger
in situations where there’s an opportunity to have a human experience.
For “grabbing food on a cold day”, how would we express such a situation?
This is not a situation typically captured in the context-detectors.
There’s really no “cold day food” detector.
So, the challenge, essentially, is that we have ideas of human situations, but they are quite far
from the machine’s available context-features. And we need a way to connect them.
This paper is a step towards enabling that middle Expression Layer,
through providing designers the cognitive supports to do this bridging.
For example, a designer can think of warm liquids or broths,
or spicy foods to get their spicy sweat on.
These could then be linked to detectable place features, like Ramen, or Indian.
So this paper makes the following contributions:
Through pilot testing, we discovered three bridging
challenges when translating human concepts into machine features.
We then developed Affinder, a programming environment that provides
cognitive supports to overcome the bridging challenges.
In a lab study, we show how Affinder’s core features can support expressing human concepts
using machine’s available features.
So let me jump ahead and show you how Affinder works.
Affinder is a visual programming environment for defining logical
expressions made of context-features, which we call concept expressions.
Designers can flesh out intermediate concepts, like spicy food,
and then these are further connected to context-features like Thai or Indian.
Overall, this workspace is a place for linking between human-defined concepts,
and the machine detectable building blocks.
But what we found in early testing there is challenges in constructing these well.
First, Designers might find one context-feature — such as
a baseball fieldsfor the concept grassy field — but miss others that also match.
So to address this, Affinder uses Yelp reviews to create an unlimited vocabulary of terms
associated with the context-features so users can discover relevant contexts.
Second, Designers can over-fixate on their initial ideas for a situation.
For example, the first idea that comes to mind might be a park
for places to toss a frisbee. However, this could accidentally narrow the scope of their thinking,
and they might not consider others.
To address this, Affinder provides prompts to reflect on why an idea
is appropriate for an experience to help users expand to more concepts or ideas.
Third, Designers may use a context-feature that does not operate as they intended — such as
a garden that is recognized as a park but does not qualify as an open area to play.
To address this, Affinder provides tools to simulate on real-world cases,
and repair the expression to more precisely evaluate their concept.
Let me show you a demo of Affinder.
So a person has searched "fields"
and has found relevant context features like parks, discgolf, and football.
They use the reflect and expand prompts
to question why parks was an appropriate place for this experience.
They realize that "open grassy areas to play" is the more general concept they belong to.
However, "open areas" might also include "open sandy areas",
and this spurs an additional search for "sandy" places like beaches and lakes.
Okay, so after composing those, we skip ahead to them simulating this concept expression.
They find this case where a park which is also a garden
is being classified as an "open area to play".
But gardens are too formal, so they exclude that to make the concept expression more precise.
So, now I'll share a few key results from our lab study,
where we conducted a between-subjects test with 14 university design students
comparing their usage with the full version with all its core features vs. a baseline version.
We measured the breadth of place context-features that were included in each construction.
The median constructions had more context-features, when designers had the supports
for finding relevant context-features, and for preventing fixation on their early concepts.
People said that the cognitive supports, such as the unlimited vocabulary search,
helped with stretching their concepts of the situation they were trying to express.
In this first quote, one said that seeing context-features that were related to their
initial idea, but in a different way, helped them generate more concepts they wanted to search.
In contrast, in the baseline version, people tended to run out of ideas
since their simple text searches were largely unsuccessful.
Finally, simulation and repair tools helped
designers see the accuracy of their construction on real-world cases.
This designer found campgrounds and baseball fields as open grassy areas for tossing a frisbee.
While there were some cases that matched their prior notions — such as on the left,
where a campground had lots of grass and foliage — some cases revealed misconceptions — like
on the right, where campgrounds which could be mainly supporting water sports
like kayaking and rafting actually do not represent that "grassy area to play".
And so now I will describe our core takeaways.
Affinder's cognitive supports were key to enabling designers to bridge the gap
between human concepts and machine representations.
Given this, we expect that others who research cognitive supports,
for areas like design ideation, would be well-suited for tackling such bridging problems.
However, bridging problems are distinct from design ideation:
In addition to helping designers stretch their concepts,
designers are finding implementable forms (in this case, as machine representations) for their ideas.
Second, there are many more context-aware use-cases where a richer understanding of
[*correction*] human situations will be important.
For example, digital workplace assistants could
recognize when a person might be struggling to ask for help.
And in a virtual reality use case, technologies might help users explore
a variety of virtual environments that supports a personal activity.
To imbue machines with this richer understanding, better detectors of context,
powered by technologies like computer vision, will of course be a component.
But context-features may not directly capture what we care about as application designers;
and thus, we want to introduce bridging tools as a research area of focus--for
providing a language for communicating the human situations we want to detect,
that are not fully captured by the current machine features.