- Welcome, everybody.
My name is Jan-Peter Kramer.
I am from RWTH Aachen University,
and I want to talk about how we can use runtime
traces to improve documentation and unit test
authoring in dynamic languages.
I have worked on this in collaboration with Joel Brandt
from Adobe Research, and my advisor, Jan Borchers.
So we know from related work that documentation
for dynamic languages that includes type annotations
and unit tests are helpful to comprehend code
and to prevent introducing errors after making changes.
But we also know that these documents are
often missing from the code or are not up to date.
And we believe that part of this problem
is in the interface design of current
development tools, and as we've heard
in the first talk, in the effort required
to create these documents.
So let's, as an example, look at one single method.
It takes the parameter.
It uses Javascript parseInt function
to convert it to a number,
and then increments it by two.
And this method is actually documented.
But the documentation claims it would only
accept the number parameter, whereas
actually it also works with a string.
So it's likely that someone inserted this call to parseInt
and then forgot to update the documentation,
and that's super common, because there's no
semantic connection in the interface anywhere
between the documentation and the method.
And the same is actually true for unit tests.
So if we wanted to find the tests for this method,
we would zoom all the way out and have to know
that the unit tests are actually collected
in a separate file somewhere else.
So we set out and asked to ourselves how we can do better
and how we can design an interaction
that encourages developers
to author more documentation and unit tests.
And to answer this question, we looked at how
developers currently work.
And we have seen in plenty of previous studies
that developers switch between editing
and testing their code in very short cycles.
And for them, this has the benefit
of having a very small scope of source code
in their head to understand mentally while debugging.
But having a small scope of code
that you currently understand is
probably also pretty good to write
documentation and unit tests about it.
So our idea is to integrate this
documentation and unit test authoring
into the edit-test-edit cycle.
But of course, if we insert something else
into this cycle, we have to make sure
that it's pretty quick and easy to do
and allows for a fluid interaction.
And to achieve this, we propose to use
runtime information from the manual executions
that occurred during the test phase.
Using this runtime information, however,
can have four potential problems.
First of all, developers will typically not
exercise all the path through the code,
so the information can be incomplete.
The last manual execution might have happened a while ago.
The information becomes outdated.
Developers might execute the code
while it's in an incorrect state.
This means that the information we get
for documentation, unit tests, also incorrect.
And, as we've also heard before in this session,
the software is likely to change,
and that makes the information impermanent.
So, let me show you an overview
of the prototype we came up with.
This is the same method you have seen before.
The documentation is now contained
in a separate documentation editor
that is embedded into the source code.
And the unit tests also have their separate
editor on the right side of the screen.
And all of this is integrated in our
Vesta plugin for the Brackets editor.
Let me walk you through the interaction with this tool.
So, here's again the function you have seen
plenty of times, at the point in time
before we actually inserted this call to parseInt.
And, currently, the documentation is not yet there,
and we can just execute the code
and Vesta will collect the runtime trace
and automatically insert all the information
it found into the empty documentation to get us started.
In this case, that's the type information
for the parameter and the return type.
As a developer, I now only need to go ahead and add
custom explanations to the parameter and the function.
And this is possible using standard text editor
interactions, moving a cursor, and using
a standard documentation markup language
and Vesta switches between both representations as needed.
And this interaction actually already takes care
of the first two problems I've mentioned.
The developer probably will execute
the method he's currently working on,
so it doesn't really matter if other
methods are not covered.
And the developer's tempted to work
with the information immediately after it came in.
So it's also likely not yet outdated.
After a while, we executed our application again,
and Vesta recognized that in this execution
we actually got a call to the method that
passed the string parameter, and not a number.
In theory, this can have two reasons.
Either we introduced a bug somewhere
where we used the method.
Then we can navigate to the call site
and fix the error there.
This is how we deal with incorrect information in Vesta.
But in this case, the change is actually intentional,
because we inserted the call to parseInt
that you have seen before, so our method
does now accept both strings and numbers,
and we can use Vesta to update
the documentation accordingly.
And this is how we deal with impermanent information
in the documentation component.
So, the unit test component is separate.
You see it in the screen shot.
It works slightly different to the documentation component
in that Vesta never automatically inserts a unit test.
Instead, we provide templates for the
creation of unit tests that use runtime information.
So the first option is to actually
create a unit test that reruns a recorded method
execution and verifies that the result doesn't change.
And the second template actually allows you to create
unit tests for common edge cases,
depending on the type of parameters
that are specified in the documentation component.
The interesting question is, of course,
does Vesta now really encourage developers
to author more and better documentation and unit tests?
And to find that out, we invited
14 developers to implement a parser
that should be accessible through a
web-based API.
So, a fairly complex coding task
that should be completed in six hours.
And we assigned developers to one of two conditions:
either using our tool, or using the Brackets
editor without any extensions.
Let's first look at documentation.
In terms of accuracy and, by accuracy, we mean
is all the type information that is recorded
in the documentation actually correct,
Vesta improved the situation significantly.
If you think about it, this is actually to be expected.
Most of the type information will be
automatically generated by Vesta,
and only if developers manually edit this information,
they will have an option to make it incorrect.
More interestingly, if we look at completeness,
and completeness refers to did developers actually add
a manual description of parameters and functions,
we also see that Vesta improved the situation quite a bit.
And this is interesting, because it means
that developers were really encouraged
to put more effort into creating these documents.
When we asked the same question for unit tests,
we see that the average number of unit tests created
throughout the trial went up significantly when using Vesta.
However, once I reveal the axis, you will probably
figure out that the number of unit tests created
is pretty low in both conditions.
So we can't really call that a success.
But the question is why did the documentation
editor work, and the unit test editor
did provide these mediocre results?
And we think the real difference
is the feature you see here.
The automatic type checks that are
performed by Vesta to keep the
documentation accurate and up to date
are actually useful in a number of
scenarios beside writing documentation.
Developers used this to spot errors,
to understand how libraries they use work,
which types they return.
And this means that the documentation
component provided more near-term value
for a developer than the unit test component did.
And we believe this is what caused
the one component to work, and the other one
to provide results that were not really
overly better than before.
So to sum up, we presented Vesta.
Vesta uses runtime information from the
manual executions developers perform
in the test phase to integrate the
authoring of documentation and unit tests
into the edit-test-edit cycle.
And in analyzing this interaction
design idea, we found that to encourage
these authoring processes, the tools
need to provide additional near-term value
that make the whole process more valuable
at the time developers are actually writing code.
And with that, I want to conclude my talk,
and I'm happy to answer your questions.
(applause)
- [Audience Member] Thank you.
I think this looks really valuable.
I was just a little confused about the unit test part.
You said it was separate, and,
perhaps that contributes to why it wasn't used as well?
And if you could explain what happens if a unit test fails.
What's the process for then modifying the code?
- Yeah, that's a great question.
There's, the editors are separate,
in the way that
there is a separate editor in which you maintain
the documentation and the unit test.
There is an option for an automatic link.
We, by default, we set it up in a way
that you get one set of unit tests,
a collection of unit tests, per method.
And you can use this to structure your unit tests,
and then Vesta will automatically show you this
collection while you navigate through your source code.
So you will always see appropriate unit tests on the screen.
There's also the option to view the back links,
so after you ran the unit tests, it will, again,
do a runtime trace in the background
and see which tests are actually calling
into this message, showing you all
the tests that are testing this.
So this is regarding the back link.
There is some interaction.
Just the editors, the parts where
you edit the unit test information
and the documentation are separate.
The second question, you can execute the tests
and you will...
Well the first thing you will see is,
you will see again runtime information,
so you will see which methods the tests called
even down the call stack.
So you get this inspection, and of course,
you will see the test result,
and you can navigate to erroneous call sites.
But we really focused on the authoring
process of creating them, and
less on creating a test, a tool to run all the tests.
Because there's plenty of,
where this is appropriate.
Even going into the direction of
doing continuous testing, which I think
would be a very valuable addition
to such at tool, because it gives
you way more immediate benefit
even for the unit test.
Does that answer your question?
- [Audience Member] Thank you.
- [Audience Member] Hi,
of the University of Edinburgh.
I'm curious why you decided to use runtime.
Clearly, you just made a decision, here, but
why use runtime information, as opposed to
things like static analysis, that might
have created something more structured
and wouldn't have run into some of these
weird cases, like, for example, the string.
- That's a good question.
I think the main reason, here, is that
we focus on dynamic languaes, which are
harder to analyze statically, just because
they leave more degrees of freedom
open in the way they're written.
There is, by design, no type information
in dynamic languages,
because it's supposed to be changing,
for example, through outside APIs.
There is deliberately call site polymorphism
in dynamic languages that you might or
might not have in other languages.
There's things like eval, and so on,
that give these languages more power, in a way,
but they also make static analysis harder.
There is some pretty interesting work
on how to tackle some of these technical
challenges of applying static analysis
to dynamic languages, which I would
absolutely appreciate to integrate here.
But I think the selection mechanism
of runtime information is appropriate
for dynamic languages, and it automatically gives you,
because the developer does the execution himself,
it provides a way of filtering out of
the information what the developer's currently
interested in trying out.
- [Audience Member] Hi.
Hui Chun Won, University of Washington.
Just a quick question about have you
contrasted use of the novice programmer
versus the expert programmer?
Because sometimes documentation
hasn't been done, just mostly because
documentation practice is not just part of their workflow.
So, how does this tool help the novice
programmers to do something about that?
- Well, that's a good idea.
We made sure that we have a pretty homogenous
group of developers.
All of them are students.
We made an entry test to make sure
that they know their ways in Javascript
and have done some work with it.
I absolutely do think that this
is more valuable in a not so strict
programming context.
So once you're in professional team,
where you have coding guidelines
and code reviews that make sure that
all of this is in place,
your need for such a tool probably
diminishes from a process standpoint.
It might still, even for those developers,
make the authoring quicker, in a way,
because it gets you bootstrapped, and it keeps you,
it reminds you more constantly than probably
working towards a coding guideline would.
- [Hui] Thank you.
- [Audience Member] Hey, Sigar Rejelah,
from the University of Michigan.
So you sort of mentioned that some use cases are missed,
because it's all dependent on developers sort of thinking
about whatever they wanna test for their test cases.
Have you thought about how Vesta could
integrate the use of fuzzers, for example,
like black box fuzzing, or white box fuzzing,
in terms of getting information from the runtime,
and then sort of creating test cases automatically?
- Yeah, I've seen some pretty impressive work
on how to generate test case suggestions,
and admittedly, the analysis process we employ
to prototype our interaction design
is not out of the world complicated.
It's actually, by design, fairly easy.
I do think that there's value in to exploring
the more sophisticated algorithms for test case
generation, and to incorporate this.
The only fear I would have is that the
connection between where this information
comes from for the developer also becomes more fuzzy.
Currently, it's very easy to relate to
which information is on screen,
because it's information you, in some way,
just inputted into the system manually.
If you do a black box fuzzy test,
where you try to guess inputs, and try
to exercise different path through
the source code automatically,
this might be harder to assess
for the developer using this information.
- [Rejelah] Cool, thanks.