Good morning, ladies and gentlemen.
Thank you for coming to our talk.
I'm Michael from Carnegie Mellon,
and I'm presenting our paper Unakite Scaffolding Developers'
Decision Making Using the Web.
Let's talk about programming.
How do you program these days?
I would like to argue that programming today
all boils down to two essential things.
Remembering what to Google, and copy
pasting from Stack Overflow.
And you can buy these online.
Well, sorry.
Sorry about the-- yeah.
No, just kidding.
They're not real books.
But the problem behind is real.
Program today is all about finding solutions online,
and using them in your code.
In fact, research has shown that developers spend on average
19% of their time looking for information on the web.
And the number goes up to 41%, if you look
at the far end of the spectrum.
For example, novice programmers.
And indeed, there are lots of research tools
that help developers find specific information online,
such as finding APIs, finding example code,
and keeping track of where the information came from.
However, finding information is often just a first step.
There are lots of trade-offs when it comes to programming.
For example in 2019, there are a variety of different JavaScript
frameworks that you can choose to build web apps.
And each has its own advantages and disadvantages,
such as being superfast, or being having a large developer
community.
Just as one of our informative study participant
put it for programming questions like this,
there are almost always multiple answers.
And you need to collect these information about trade-offs
from lots of different sources, and do
compare and contrast, which is called Sensemaking.
And indeed, this is supported by prior literature,
such as this graph, the different stage
of sensemaking that in addition to finding information.
The next step developers will have to take
is to organize, and synthesize, and compare
the things they found in order to make an informed decision,
which brings us to the first research question we would like
to address.
How can we help developers organize information
about trade-offs?
In addition, organizing and understanding the trade-offs
are not just important for the first developer,
but also important for the next developers who often need
to know the design rationale.
Or why decisions are made the way they are.
So that later, if they need to change things,
they don't accidentally violate any requirements
or assumptions.
As research-- and research showed
that understanding design rationale is one of the hardest
questions for developers to answer,
which brings us to the next research question.
How can we help developers understand decisions?
Guided by these research questions,
we first carried out a formative study
where we interviewed 15 developers
to understand how they currently collect,
and manage information about trade-offs.
And there are three main findings I would like to share.
The first one is making decisions
is frequent and difficult in programming.
13 out of 15 developers reported being frequently swamped
in decision making tasks, such as,
which are often difficult to do?
Such as choosing optimization methods when
training neural nets, and picking cryptographic
algorithms to protect users passwords.
The second finding is that developers
need tools support for making decisions.
8 out of the 15 participants reported using existing tools
to organize information.
However, they also stressed how hard it is to do it.
For example, it's hard to collect content online.
A representative quote is, copy pasting is just too much work,
and I lose all the styling.
It's hard to maintain provenance.
In other words, keeping track of where
the information came from.
It's hard to synthesize new content with existing content.
And there are often too many contexts switches
back and forth between different applications and browser tabs.
But how might developers want to organize all these information
they found?
What are their models and structures?
Well, this is answered by our third finding, which
is that participants reported maintaining
a mental table comparing different options with respect
to different criteria, while making programming decisions.
For example, this is the comparison table
that a participant maintained in a Google Doc
on the topic of picking JavaScript frameworks.
Inspired by the results of the interviews,
we would like to investigate how often
do decision making in programming actually
involve trade-offs.
And the obvious place to investigate
is Stack Overflow, which is where everybody goes when they
have programming questions.
And sure enough, we were surprised to discover
that of the most viewed questions,
88% were actually representable as tables comparing trade-off.
And even half of the general questions
were also representable as tables.
The rest of them are mainly about facts, tutorials,
and debugging.
We also found out that some of the highly liked answers
are already using tables to capture the trade-offs
between various options.
And inspired by the results, we designed and implemented
Unakite, the Chrome extension that helps developer
collect and organize information,
while searching and browsing.
And you guessed it, of course is a gemstone.
An acronym that stands for a users
need accelerators for knowledge for implementations
in technology environments.
All the features in Unakite are designed
to serve these four design goals, which are go
over one by one.
Number one-- scaffolding, which is to help developers structure
information about trade-offs for better decision making.
One of the common barriers that developers reported
in the interviews is that, it is not realizing the problem they
are trying to solve is actually a decision making problem
until later on in the process, when they're
stuck with multiple options and criteria that
have to compare over.
So the scaffold is process.
And to encourage them to approach problems
in a more structured way at an earlier stage,
we developed the option criterion evidence framework
through a lot of iterations and testing.
So that to style easy for developers
to understand and use.
In Unakite, developers are encouraged
to organize information they collected into a comparison
table.
With options being the row headers,
criteria be in the column headers,
and evidence spread across the rest of the cells.
As I mentioned earlier, developers often
curate a similar, but less organized mental table.
Formalizing this framework provides
a concrete framing for them to think
about decisions in a structured way
that they're already familiar with.
The second design go for Unakite is
to have lightweight interactions.
This is guided by our prior research, which
found that, interactions for collecting and structuring
information or browsing needs to be quick, intuitive,
and low cost without taxing users with too much
cognitive workload.
Streamlined developers interactions with the system,
Unakite offers a sidebar on the right side of the browser
window.
It contains the comparison table that I just
mentioned in the snippet repository, which
houses all the information snippets the user collects.
It is designed, so that developers
can do the information organization
right within the page that they're currently on,
saving lots of lots of context switches to other tabs
or applications compared to traditional approaches
while you're maintaining a Google dot
document in a separate browser tab.
Developers can use Unakite to collect any information
from any web pages.
For example, one can select some content,
and click to save to Unakite button that pops up.
The content will be immediately it saved into the repository.
Alternatively, we can also draw a bounding box
around some counting, and click the button.
These interactions are carefully designed
based on developers natural habits of copy,
pasting, and taking screenshots without introducing
too much cognitive load of learning new interactions.
The snippets you collected will be
preserved in its original HTML format, which
helps developers quickly recognize it
among many other snippets in the repository.
To organize information, users can use drag and drop
to move any snippets they collected into the table,
as either an option, a criteria, or a piece of evidence,
which can either be positive, negative, or informational.
This one.
To make things faster, users can use the clip button popups
to quickly click content as options for criteria
directly into the table.
Or select some content, and then click anywhere in the table
to pull it in as evidence.
Together with the other features mentioned earlier,
users really have the flexibility
to capture and organize their knowledge
in a variety of different ways and orders.
The third go is summarization, which
is to synthesize and manage various pieces of information
developers found.
Unakite provides a couple levels of summarization
out of the box.
The first one, the comparison table
provides a high level summary of the trade-offs and the decision
making space.
It offers a clear and glassable picture
of the advantages and disadvantages of each option
without exposing too much of the details of the evidence behind.
This is really useful for both the people who
are making the decision, and the people who are trying
to understand those decisions.
Second of all, the individual rating icons
provide another level of summary of their corresponding
supporting evidence.
Different from other approaches where long paragraphs
are recursively summarized into shorter ones,
we attempt to summarize them in terms of developers
ultimate information need, identified, and interviews,
which usually is to tell whether an option is
good or bad in terms of the specific criterion.
To go another realm deeper, users
can simply click on those rating icons in the table.
This will bring up the corresponding supporting
evidence snippet in the repository.
The last design go is contextualization,
which aims to recreate the context from which decisions
were made.
For each snippet, Unakite retains the context
by automatically keeping track of its source, URL,
and a collection timestamp.
So that developers will know where and when
it was collected.
Each table and its corresponding snippet
repository can also be viewed in the Unakite web app, which
has a unique URL of course.
The user study participants later
suggested that they can share the URL with other developers
attached within code comments, or simply
use it as documentation.
Either way, this serves as the design rationale
behind a programming decision.
And code readers will be able to recreate the context,
and understand the decision making context to see,
what were the options and criteria?
What were the trade-offs involved?
To evaluate Unakite, we ran two studies.
The first one focused on authoring unicode tables
to evaluate its usability and effectiveness.
And the tasks were to collect information
from Stack Overflow pages, and organize them
into comparison tables.
The study was a between subject design,
and the control group used Google Doc to make the tables.
We chose Google Doc as a baseline
since it's reported in the formative study as a tool
that developers are already using,
and it's a tool that all the users study
participants are familiar with.
Recruited 20 people, each completed two tasks
in the same condition.
And after that, all the participants
then use Unakite for programming tasks
that they are trying to solve in real life.
And here's what we found.
First of all, Unakite is usable.
All participants are able to use the various Unakite features
that I talked about earlier.
For example, this is the table of one
of the participants who are building--
who are trying to choose the best JavaScript
frontend framework in the third task.
And here are some more examples that participant made,
such as picking cloud computing services
and hybrid app development frameworks.
Second of all, using Unakite is more efficient.
By analyzing the screenshots, we found out
that the overhead cost of Unakite,
which is the portion of time spent on directly using tool
features is only 25% compared to the 44% and the Google Doc
condition.
This is about a 45% reduction, and the result
is statistically significant.
The second study we ran focused on evaluating developers
stability to understand the trade-offs
and decisions with Unakite.
And the tasks were to explain code decisions made
by a previous developer.
Recruited 16 people, and each completed
one task in each condition.
So in the Unakite condition, people
use the table that the previous developers made.
While in the control condition, people
read the same set of directory sources
that the previous developer looked at while
making the decision.
Look at the results.
Everyone got the answers correctly in the end.
However, in the Unakite condition,
user saw a small table like this,
and they spend on average less than two minutes
to export a table like we talked about earlier.
While in a control condition, they just
had the source material.
And it took them roughly 5 and 1/2 minutes to get it.
So in summary, understanding decisions with Unakite
is a lot faster.
And now we'll talk about some of the limitations of Unakite.
So first of all, we've found in the user study
that users would like to apply some sort of weights
to the different criteria or columns.
Or be able to prioritize them of one
over another, which the current version does not gracefully
support.
In addition, the current design only
focuses on supporting a single developer's decision
making process.
Or a lot of times, especially in corporate settings,
making decision is a collaborative process.
Finally, our lab study has limited ecological validity,
so we would like to deploy Unakite in the field.
And examine how developers adopt,
and use it in natural settings.
And we'd like to address all of these in future work.
In conclusion, meaning programming questions
are not just about using the web to finding
specific information.
It's often about making decisions involving trade-offs.
Through designing and evaluating Unakite,
we demonstrate that if we help developers capture and organize
information in a lightweight way,
the resultant structure might be useful and beneficial
to themselves and other developers later on.
Last but not least, I like to think severely
thank the people who funded the research,
and the people who helped along the way.
With that, thank you so much for being here,
and I'm happy to take any questions you may have.
[CLAPPING]
Hi.
Thank you for the great talk.
I was-- while listening, I was struck that the tool you built
might generalize beyond the questions
that programmers ask themselves, right?
So I see a connection to the work
that, for example, Travis crippling
did on the living waters guide in Washington,
where people want to consider a complex trade-offs for how
they want to vote on a particular measure.
And I'm wondering if you could, maybe just give us
your thoughts on lot of this design is
particular to the needs of developers
versus the needs of other people who forage for information
on the web on complex issues.
Yeah, sure.
It's a great question.
And the question was, what's specific for programming
that when you two, let's say make a table out of.
I guess, the biggest thing about programming
is the need for keeping track of these design rationales,
as opposed to, let's say shopping.
When you have to make a decision,
everybody has their preferences.
So there really isn't too much desire or need
for making these tables.
But in terms of programming, you have
to preserve these decision making contacts.
So that later people will have to read your rationale in order
to implement their code.
Thank you.
Thanks.
So a question I had was like, when you go to a [INAUDIBLE],,
and selecting you want to add it and organizing the table.
Oftentimes, like, from my own experience,
I feel the need to like go, and plug it into my thing.
Execute it.
See if that's what I want before that I make trade-offs.
How do you think you might extend the system to kind
of support that intermediate step between like foraging,
trying out, and then saving?
Yeah, that's like an [? M2 ?] system.
We actually had a editor plugin in the earlier versions,
and we decided to take it out for the UIST submission.
And yeah.
So we also have like common stability,
so you can make commons.
For example, note down, what works?
What didn't work in the system.
Thank you.