Hi everyone! I'm Kasra, and I'm here to talk to you about a work we recently did, combining Live Programming with Programming by Example.
But first: A bit of background!
These are Projection Boxes.
They are a live programming tool for Python programs that display the values of all available variables at each line of the program.
For example, since we are calling the function "Abbreviate" with the input "Augusta Ada King" on line 14, we can see that the "name" variable has that value on line 10.
As I modify the program, let's say by creating a variable "x" and assigning "1" to it, we can see it appear at that and subsequent lines, where it is usable.
Projection Boxes are an example of Live Programming, where the user gets immediate feedback about the program's state.
Live Programming is great for understanding the behavior of a program we already have, but it doesn't really help us with writing the next snippet of code.
Meanwhile, in another area of Computer Science called Program Synthesis, we have Programming by Example;
Techniques for using sets of input and output examples to generate computer programs.
Programming by Example is great, because (especially compared to some other specifications in Program Synthesis)
it's a really easy spec to understand and write. But, having to write all of these examples can be quite tedious.
Looking at these two, we saw a match made in heaven.
We could use the values from Live Programming to simplify generating input and output examples, which would then in turn enable Live Programming with the ability to generate the next snippet of code.
So we took Programming by Example, added it to Live Programming, and created...
SnipPy! Let's see how it works.
We're going to work through this programming problem, where we need to write a function that returns the abbreviation of the given name;
The first letter of each word, separated by dots.
As I think about this problem, one way I could approach it is to
Split it into words, get the first letter of each word and then put dots between them.
But, instead of immediately switching to my browser to search for how to split string in Python,
I instead create the variable words, and assign "??" to it.
This places my cursor inside the Projection Box, where I can now write the values I want it to have.
After pressing "Enter", SnipPy looks at my example, and the available variables, and figures out that it needs to call "split()" on the "name" variable.
I can repeat this for getting the first character of each word, but since writing out the list syntax is tedious, I will first write the name of the "words" variable before "??".
This prefills the Projection Box, and I can just modify it instead of writing from scratch.
And I can finish the task by using SnipPy one more time to create the final abbreviation.
I can provide multiple examples in SnipPy by calling a function multiple times, or in loops.
Then I can move between the rows when providing examples, and add as many or as few as I want.
Here, we will use the highlighted lines as examples, while ignoring the line that I left unchanged (and is not highlighted).
Finally, it's worth mentioning that SnipPy can do a lot by generating list and dictionary comprehensions.
In fact, for this example, we can use SnipPy to solve the problem in just a single line.
The line is essentially the same program as before, just without storing each intermediate step in a separate variable.
Okay, I don't have much time left, so let's quickly go over some of our observations from a user study that we ran on 13 participants.
There we compared SnipPy to using just Projection boxes and searching the internet.
Firstly, the response was mostly positive, with most participants saying that it was easy to use, and that they would like to have it available when programming.
When compared to searching the internet, we found that they complemented each other really well.
SnipPy was more limited in what it could do, but participants mentioned that it could find solutions more quickly,
without the cognitive burden of switching to another program,
and that it could find more compact solutions.
From these studies, we also noticed a limitation in using Program Synthesis.
We noticed that the participants' effectiveness in using SnipPy was closely related to the accuracy of their estimate of what SnipPy's could do.
There's much more discussion of this in the paper, but very quickly:
We named this gap between the synthesizer's actual capabilities and the user's mental model of the synthesizer, "The User-Synthesizer Gap".
And we argue that addressing it is a key area of future work in using Program Synthesis in tools that are meant for long-term use.
So to wrap up, we found that Live Programming and Programming by Example are perfect for each other.
So we combined them to create SnipPy.
From user studies, we discovered that users tend to like it and find it useful,
But that the error in users' understanding of what SnipPy can do was a key limitation.
We named this "The User-Synthesizer Gap", and argue that addressing it is a key future direction.