I'm Michael Coblenz.
Today, I'm going to tell you about PLIERS: A Process that Integrates User-Centered Methods
into Programming Language Design.
As you can see from recent major bugs, it's still far too hard to write software.
Log4j was due to a code injection vulnerability.
Forcedentry was caused by integer overflow.
And a bug shut down Amazon Web Services for the better part of a day.
Part of the problem is that programming languages make it too easy for programmers to accidentally
insert bugs.
Suppose you're programming in an object-oriented language and you want to call foo() on x.
But it's not safe to do that if x's type doesn't implement foo(), and you might have a hard
time checking if you're not using a statically-typed language.
You also can't invoke foo() safely if x might be null or if x might be a reference to freed
memory.
Likewise, suppose you want to call setAddress() on a person reference.
But what if person references an object that is used as a key in a hash table?
Mutating that object could corrupt the hash table.
In this work, we view programming languages as user interfaces that programmers use to
achieve their objectives.
Our goal in this work is to enable language designers to create languages that make programmers
more effective by taking user needs into account directly.
We choose safety properties that are relevant to users, and then design languages that provide
those properties as effectively as possible.
Then, we evaluate to see whether we succeeded.
The goal might be fewer bugs when doing a task, or task completion in less time, or
making it more likely to complete a task correctly in a certain amount of time.
Or it might be to enable people to do the task with less training than was previously
required.
PLIERS, which stands for "Programming Language Iterative Evaluation and Refinement System,"
is a systematic method for making languages effective for programmers while at the same
time retaining key safety properties.
In a user-centered way, PLIERS involves users throughout the language design process.
We used PLIERS to develop and show benefits of two different languages.
PLIERS includes five steps.
First, we analyze the programmers' needs using methods such as interviews, corpus studies,
or contextual inquiry.
This leads us to develop some initial ideas for the language.
These ideas include a formal aspect, such as a core calculus and formal properties that
we would like the language to have.
We might also sketch proofs of those properties to become more confident that the language
will have the desired properties.
There's also a user-centered aspect, in which we design example programs and maybe implement
an interpreter or compiler.
Now, it's impractical to do user studies for every design decision.
To focus our user studies, we do a risk analysis using techniques such as cognitive dimensions
of notations, or a comparison with prior systems, to see which aspects are worth studying in
more detail.
Then, once we've identified key usability risks, we iterate on the design in a typical
HCI fashion.
We evaluate the system, either with user-centered methods or with traditional methods, such
as performance testing, and then change the theory and the prototype accordingly.
Finally, we do a summative evaluation to see whether we have achieved the goals we set
out for the language.
One project concerns blockchain programming.
Blockchains are used for high-stakes applications in which the users don't trust each other.
On these systems, it's particularly important to get the programs right the first time,
since money can be stolen from vulnerable contracts very quickly.
We've seen millions of dollars stolen through vulnerabilities that were caused, in part,
by traps laid in the Solidity language, which is commonly used for writing smart contracts.
We made two observations about blockchain applications.
First, they typically manage important assets, such as virtual currencies.
If an asset gets trapped inside a smart contract, the money is lost forever.
Second, which operations smart contracts support frequently depends on the contract's state.
The DAO hack involved invocation when the contract was in an inconsistent state.
In our language design, we used a technique called "linearity" to ensure that assets cannot
get lost inside contracts, and we used a technique called "typestate" to enable the compiler
to reason about which operations are valid according to a contract's state.
This is a new combination of features in a programming language, and user studies hadn't
evaluated the usability of either technique.
Now, we were interested in obtaining formative data on the usability of our design.
A traditional way would have been to implement the language, develop training materials,
and then do a usability study.
But implementing languages and teaching them is a very high-cost effort.
We were interested in obtaining formative data early so we could refine the design before
implementing it.
To obtain earlier-stage usability data, we adapted the wizard of oz technique from HCI.
We gave users a text editor and had an experimenter simulate error messages that a compiler might
have generated.
We also wanted to avoid teaching participants a whole new language.
Why?
Well, teaching a new language can take a long time.
And suppose that we DID teach participants a language, and then we observed that they
encountered a problem doing a task.
The problem could have been caused by any aspect of the design, rather than the specific
design decision we were trying to assess.
To address this problem, we adapted the design questions we were interested in to a language
for which we could find existing programmers.
In our case, that was Java.
Now, one key aspect of Obsidian's design is that the types can actually change as a result
of operations that occur.
So, what did we find?
The studies enabled us to identify and address a collection of usability problems.
For example, although concise syntax is traditional, we found that a more explicit, verbose syntax
helped clarify changes in types that occur to method parameters.
Also, although type annotations normally provide complete type information, we found that for
local variables, they should actually be incomplete because otherwise they could be confused with
state restrictions.
This led to adding a new language feature to allow people to assert typestate at arbitrary
points in the code.
To see whether Obsidian was better than Solidity, we ran a randomized controlled trial.
We recruited 20 Java programmers.
We assigned half to use Obsidian and half to use Solidity.
We gave them a tutorial, which took about 90 minutes to complete.
Then, we gave them three programming tasks.
The Obsidian participants completed significantly more tasks correctly than the Solidity participants:
1.5 vs. 0.5.
Now, what are some implications for other kinds of work?
First, it is important to remember that tools for experts, such as programming languages,
benefit from usability studies too; experts are people and deserve tools that take their
needs into account.
Second, the HCI methods we used required a relatively small amount of adaptation, so
this kind of work could be undertaken by most HCI researchers.
One observation was that the think-aloud method requires probing to really understand what
challenges people encounter.
Also, our technique of adapting research questions to familiar experimental settings helped us
focus our studies and made them much more feasible.
This approach may be useful in other contexts as well.
In conclusion, PLIERS is a user-centered design process for programming languages that incorporates
qualitative, quantitative, and formal methods.
PLIERS can lead to better programming languages which result in safer users.