Welcome to the presentation of our paper: On Designing Programming Error Messages for Novices -
Readability and its Constituent Factors
Let's now take a look at two students working in a typical computer lab...
...and "run". 45 errors! "The name 'i' does not exist in the current context"
What does it even mean?!? I have no idea, mine says "null reference exception". That doesn't make sense either...
Unfortunately these kinds of frustrating experiences are all too common: Novice
programmers struggling to make sense of cryptic programming error messages. Such error messages
are a type of feedback produced by software, called a compiler or interpreter,
which indicates there is some kind of error in the source code written by a programmer. The
cycle of program input and error message response represents a fundamental interaction between human
and computer. While error messages can be confusing and frustrating - even for
professional programmers - they can be paralyzing for novices who are just beginning to code.
Consider this example, which involves a compiler for the C programming language
This is a popular language used in many introductory programming courses.
The student here is attempting to write the code statement shown on top but they make a very small
error: they have forgotten to include the comma between the two identifiers, fahrenheit and celsius.
This missing comma generates an error message (shown here in red) which is not
only cryptic and mostly unhelpful... it is simply difficult to read. We propose that
improving the readability of error messages is an important first step in making them more usable.
However, we don't yet have empirically tested guidelines for making messages easier to read.
In this work, we set out to answer two research questions. The first, to identify potential factors
that might affect message readability, and the second to measure the extent to which
they do. Now i'm going to give a brief overview of the three studies we ran as well as discuss
the results of Study 1. In Study 1, we asked participants to rate (on a scale of one to ten)
the readability of 60 error messages from three popular introductory programming languages: Java, C,
and Python. In Study 2, we took a smaller number of messages and asked participants to explain in
their own words what made them readable or not, and then in Study 3 we took results from Study 1
and 2 and asked students to rate a set of error messages with respect to what we had found in
the first two studies. Results of Study 1 indicate that length and jargon were particularly important.
The shorter messages were ranked higher, generally speaking, and the longer message were ranked lower.
Also the messages with more jargon were ranked lower while messages without jargon or acronyms
were ranked higher. To elicit potential factors that influence error message readability, we asked
students to tell us what they felt made certain messages easier or harder to read. Responses
indicated themes around message length, jargon, sentence structure and vocabulary.
In a separate study, we found that all four of these factors correlated reasonably with
a measure of understandability. We present four concrete design guidelines for writing more
readable error messages and apply these to messages from our data set. Here is an example
from C. Future work includes the creation of a readability metric for programming error messages.
Finally, we hope that the guidelines presented here will aid in the creation of new programming error
messages, as well as guide efforts to revise or enhance existing messages. Our goal is to see
programming error messages become more readable, and therefore more usable, in the coming years,
easing a decades-long struggle between countless novices and their programming environments.
Thank you very much for listening to this presentation. If you're interested, you'll find more
details in the paper, and if you have any questions please don't hesitate to get in touch with us.