- And this is joint work with Otmar Hilliges
here in the room and Peter Muller at Zurich.
All right, so this talk is about the effects
of code presentations such as syntax highlighting
on the understanding of code structure.
So, syntax highlighting is something that we've used
for several decades now, but there's surprisingly
little research about the effects of syntax highlighting
in similar visualizations like that.
So this work is about the study we did
precisely in that area.
So what we did is, we compared syntax highlighting
to two richer visualizations of code,
and we wanted to see what the impact of these visualizations
on the performance of developers.
And so these visualizations are richer designed
where the medium condition is different in this highlighting
by in addition to highlighting keywords, also using
outlines and alternating background colors
to show some parts of the code.
And the high condition is sort of like this one
but in addition to it, we also have icons
instead of keywords and background colors
for some statements and expressions.
So as to further illustrate the structure of the code.
So, using these three visualizations, we had
three research questions, the first one is
how does using richer visualizations impact the speed
at which developers can answer questions
about the code structure, the second question is
what about the correctness that having richer visualizations
impede or help developers to answer questions
about code structure, and the last one is
what about the visual overload,
this having all these symbols and colors and outlines
actually make it more difficult for developers
to answer questions about code.
So here's the study we designed to test these questions.
We think the study we designed 15 yes no questions
about programming, about the structure of code,
and we asked the developers these questions in random order.
So these are questions that we find
practical kinds of representative of the low level tasks
the developers experience as they're coding.
One example is, for example, is there a nested loop
inside the method.
So, those are the kinds of questions we ask.
At the very end of the study, we also ask the developers
of their opinion of what they've seen
and we collected some demographics.
So, we did this with 32 developers.
Within each questions, we let the developers use
the three different visualizations and we randomized
the order of the visualizations as well.
And within each visualization, we showed 15 screenshots
of methods to each developer.
And so these screenshots that you see here,
they were drawn from a pool, randomly drawn from a pool
of 298 methods that we took
of the jEdit Java open source text editor.
So this is real code.
Right, so, a little bit more to the questions that we asked.
They're separated in three different categories,
each category has five questions.
So, here's an example of the question
from the first category, is subtraction used
in an expression somewhere in the method?
And the categories, the questions in this category
are questions where all three visualizations
show the construct relevant for answering this question
in a similar way.
So essentially, the richer visualizations don't provide
any particular enhancement in this case.
An example of the questions from second category
is this, is there a loop nested inside another loop.
So in this category of questions,
the medium and high condition provide some sort of
enrichment that is relevant for the construct
that the question asks about.
And the last category of question here is an example,
does the method cast an exception.
And so these are questions where only the richest
visualization provides some particular enhancement
and the others don't.
All right, so let's see some results.
Where it comes to correctness, so were developers
able to correctly answer the questions,
and these are all yes no questions.
What we observed is essentially no difference
between all the different tools.
So we asked very simple questions about code structure
and developers were mostly able to answer these
correctly many of the times.
And this is the real data here, and there's really no
significant difference between the tools.
All right, so when it comes to speed,
things are a little bit more interesting.
So here I will show you the graph
and I will explain it to you.
So, on the X axis, you see the three different categories
of questions, and on the Y axis
there's the response time in seconds.
So here is one question, and you see here
the times are rather similar.
We are also going to show you 95% confidence intervals,
and these were computed using a robust bootstrap method.
So here you see the confidence intervals
are somewhat overlapping, and there's no significant
difference for this question
between the three different visualizations.
And we see the same for all the questions
from this first category, sort of as expected
of the visualizations displayed the code
in a similar way, at least when it comes to the construct
that we asked about.
So there's no difference here.
When it comes to the second batch of questions,
there's also no difference for this first question
over here, but for all the other ones
we see that the enriched visualizations perform
significantly better compared to the standard
syntax highlighting here in grey.
And so this range is between half a second here
or it's a two seconds over here on this side
which is a significant reduction in response time.
And in the last category of question, we also observe
a significant reduction in response time
especially here for the high visualization
which provides some particular enhancement.
And this is five seconds here
for example on the last question.
All right, what about visualization overload?
So some of you might also think, okay, this is interesting
but if I stared at all these symbols and colors
and icons my head is going to start spinning.
And certainly many of the developers reported
something like that, so 23 out of the 33 developers
that we had reported at the end of the study
some form of visual confusion or overload
with these presentations.
However, it's worth to point out that this is really
a subjective perception.
And if you remember the data, there was no difference
in correctness, and when it comes to the time data,
the three visualizations either performed
sort of equally well, or the richer visualizations
performed significantly better
in the last part of these questions.
So what's interesting to point out here is
particularly this first batch of questions
where developers had to answer questions
pertaining to a construct that wasn't enriched
by any visualization.
So here there's high potential for overload
because you're looking at all these symbols and colors
and outlines that actually don't help you
to answer the question.
But even in that case, we didn't see any negative effects.
So, basically, we did not observe any visual overload
and there is a mismatch between what developers
self reported, and what we actually measured.
All right, so in conclusion, we observed that
when it comes to speed, there's up to 75%
or five seconds reduction in the response time
to answer questions about code structure.
The 15 particular questions we asked,
there was no difference in them
when it comes to correctness, the developers almost always
answered them correctly.
And we did not measure visual overload
which was a mismatch compared to
what developers self reported.
So, in conclusion, I think it's worth exploring
such enhanced visualizations a little bit more
and we think they hold high potential
for improving understanding of code structure.
So if you want to know more, you can visit
the project webpage, and you can also download
the complete data for this study
to replicate it to re-analyze it.
Thank you.
(clapping)
- [Audience Member] I really like this style
of your style of asking questions
from one of the colleagues, I'm gonna repeat it.
I'm not gonna sing though, it's kind of late
and can wake up people, but anyway.
Have you tested this kind of approach of highlighting
and some putting some visual cues in text?
And see if it speeds up the process of reading?
Or comprehension maybe?
- So we have not tested directly in text,
but I mean, you can imagine, you can see
the visualizations, you could reasonably construct
an idea, some tool that sort of renders
these lines in addition to a text.
So I believe this is close to what you're really asking.
I think a tool could do that.
We haven't done that personally.
- [Audience Member] Okay, thank you.
- [Audience Member] Hi, David Karger, MIT.
Regarding this contraction between what they were
subjectively reporting.
Do you think it might be because they had some
other tasks in mind for which they felt
that, like, you gave them specific tasks,
but did you sort of talk to them about
what they might want to be doing
using the idea, and that perhaps it was
things that you didn't test that were making them
unhappy about the visual overload that they perceived.
- Right, thank you.
So, we did talk to them about other things,
but actually, this number 23 out of 33,
that specifically pertained to problems with these tasks.
So actually, many users, including some of the pilot users
who tried it, they felt that they were slower
with the highest visualization, and they,
I mean as you said, they preferred this medium level
although they were faster with the other one,
and there was basically no disadvantage that we measured.
- [Audience Member] Thank you.
- [Audience Member] Hi, I'm from University of Washington.
I just wonder, have you guys tested this
with more, like, repeat comprehension
in terms of the functionality of the color reel
and just the structure?
- All right, thank you, so that's a great question.
So, we haven't done that.
The kinds of visualizations we do here,
they would really be more helpful
on the sort of lower level tasks.
I've seen some previous studies that
that ask students to, let's say they show them
a 10 line method and ask them, well, what does this do?
And that takes maybe 10 minutes, maybe 20 minutes
to decipher, and you see here,
we're talking about seconds.
So, this isn't really gonna help you
in such a very high level task.
- [Audience Member] I see, I see, thank you.
- [Man] So, I'm sure our speakers will be here
to answer questions, let's thank Dimitar
and everyone else again, thank you.
(clapping)