What is a software developer looking for when reading Machine Learning code?
And is it different than for regular, non-machine-learning code?
Machine learning is becoming ever more ubiquitous
and ensuring that these systems work, certainly is a good idea
and the developer seems to be a good starting point for quality control.
Now, for general software development there is plenty of prior work
to figure out how developers work.
For Machine Learning: not as much.
Ideally, we would want to just transfer all this knowledge we already have to Machine Learning.
But is that valid?
Machine Learning is a very different development paradigm.
It is not algorithms that define the behaviour but data.
That is why we thought that a comparison of those two paradigms would be worthwhile
and we started, for now, with reading behaviour.
So, what did we do?
We conducted an eye tracking study where we had six developers sit down and read some code.
Specifically, they read Python code
which was the obvious choice since it is used a lot for Machine Learning
but also for other, non-Machine-Learning things.
In total each had eight small pieces of code,
four of which were just general Python examples
and four used actual Machine Learning using e.g. the iris or mnist datasets.
To make sure the code fit onto the screen
it was relatively short
so we stuck to examples you would find in a typical Machine Learning introduction.
Now, after gathering some demographic info
every participant had to read each of those code snippets
in a little UI where we showed the code
and then asked them to summarize what it does and whether they noticed anything.
After the study, we first visualized the data in these heatmaps
and as you can see there are already some interesting patterns
if you look at the one on the left
which is representative of the people who reported little machine learning expertise
we can recognize reading patterns
which resemble Nielsen's F-patterns
where they start at the top left and the scan through the code mostly line by line.
Compare that to those that claim to have some more expertise
who really just skip most of the early parts
which usually is just a bunch of imports
and spend much more time in the details
in this case some configuration parameters.
Now, to be fair, with three more and three less experienced participants
its hardly a pattern but it certainly is worth further investigation.
We then split the code into smaller parts to see what areas are especially interesting.
So what we did is to basically label every area with what type of code it contains
for example literals, conditions, etc.
and then checked the dwell time for each area.
And this now allows us to fairly rigorously compare machine learning and non-machine learning code.
This also highlights a considerable challenge
while we tried to have our code examples relatively comparable
in terms of length and complexity
not everything from the formal grammar is equally represented.
Anyway, to at least account for this we normalized our dwell times
based on how much space each code construct takes
so - in theory - if everything were equally interesting,
it should be the same proportion.
But, as you can see, this isn't really the case
there are some things that are disproportionally more interesting
for example loops and function calls,
and some that are much less interesting like imports
This kind of applies to both types of code,
but there are also some differences
for example in traditional code the control structures, loops and conditions, appear to be more important,
while in machine learning code imports are slightly more interesting
and especially index-based operations appear to be relevant.
So, in summary
we think it is important to investigate how developers interact with the systems they write
especially with Machine Learning.
From what we can tell so far, there are some differences between Machine Learning and traditional code
regarding reading but also just general structure.
Of course, our results need to be expanded upon
by checking how more experienced developers read their code
whether the same applies for larger, more complex software
and so on.
But also, this is only a first step
which ultimately should also lead to methods that help them write better software
and better machine learning systems.