So, I want to begin by saying the last presentation is
actually very complimentary to some of what I'm doing.
But hopefully you'll see that connection
as I move forward with the next 15 minutes.
So, hi, my name's Alex.
I'm a PhD student from the University of Waterloo
and today I'm presenting Mercury,
Empowering Programmers' Mobile Work Practices
with Microproductivity.
And this was work done during my second internship
at Microsoft Research.
So when we think about programming,
and where it takes place, we normally
think about devices that look like this.
Right, the laptop, a desktop with a few monitors,
and then a desktop with many more monitors.
Anyway, programming is relatively stationary as a work
practice and it relies on personalized development
environments that are made up of a variety of different
build tools that really poorly translate
to the mobile context.
I feel like this room is probably--
I feel like I'm preaching to the choir in some sense here,
but the question that I asked, or that we asked,
in this study was how can we empower programmers
with mobility in their work?
And the way that we went about this in terms of our study
was across three different stages.
And the first stage was first conducting a contextual inquiry
in which we observed programmers in their workspace.
An online survey that was inspired by their findings
in our contextual inquiry, and the goal
here was to determine whether or not
our findings from our inquiry generalized scale.
And then finally we ended up building
the system named Mercury.
And Mercury as a mobile programming tool
that explores a microproductivity-based
solution for mobility.
So just to highlight some related work,
there are three big pillars of research
that we build on top of and extend.
The first of which is mobile programming
and mobile programming is a broad space.
Especially if you consider information
work broadly in the mobile context.
But we looked at studies of multi-device use
as well as cross device programming
in systems like TouchDevelop from MSR's earlier days
roughly a decade ago.
As well as Maria Husmann's work that has kind of
inspired things like Apple's Sidecar
if you're familiar with that.
Similarly because mobility inherently
involves pausing work, we also looked at the literature
on interruptions as well as task resumption.
Focusing specifically on interrupted programming tasks
namely the work from Chris Parnin and Thomas LaToza.
And finally, this is no surprise we also
looked into microproductivity in microtasking,
but I should point out the obvious reason we did this.
So our work is heavily integrated
into this space of interruptions and there's
been substantial work that shows that microtasking is actually
highly resilient to interruptions.
As well as it's a really convenient way to design
for limited attention scenarios which in some sense
the mobile context is all about.
So just to give an overview of our study,
there are technically three phases
and I've divided the third one into a fourth, which is
our real evaluation component.
But let's dive into the contextual inquiry first.
So our entire study was really motivated
from the get go by one simple research question
and that was how do programmers utilize their mobile device
on mobile.
So to answer this question we again went
into programmers work spaces and observed them for an hour.
And during this hour we posited two different scenarios.
One was a planned interruption that they
knew about beforehand, and then an unplanned interruption then
happened.
And during both of these contexts, or scenarios,
we looked at how they use their phone.
So this was more or less an interview process
that we recorded, and then analyzed and transcribed
or transcribed and analyzed.
Just to highlight our findings the major takeaway if you're
going to take away anything from our inquiry
is we effectively designed or found
a taxonomy of mobile programming tasks that people perform.
And the tasks themselves range from anything
related to capturing thoughts on the go,
all the way up to actually engaging in full blown
programming.
So based on the taxonomy that we saw in our online survey,
we then wanted to determine whether or not
our findings really generalized at scale.
So what we did is we took this taxonomy
and then basically asked people whether or not
they practice these tasks currently, as well
as want to practice them and are currently
incapable of doing so.
In addition, we also wanted to understand
the challenges of pausing work.
Specifically focusing on the question of whether
or not resumption itself is still a challenge.
Because this problem itself of resumption
has been studied for upward of 20 or 30 years now.
So just to highlight some findings on the right,
you'll see a plot that actually shows all of the responses that
we had for the taxonomy of tasks that people could either
currently perform or desire to perform but currently feel they
can't.
So the existing practices that we
found, which were capturing thoughts,
using email as some sort of information transfer
network between devices, and then
conducting online research.
For example, finding web resources on Stack Overflow
or MDN for example.
All of these things are generally related
to transferring and seeking out information
and they're exploratory in nature.
Now if you look at some of the things
that people wanted to be able to do,
they were generally grounded in augmenting existing information
or artifacts.
This included primarily, at least
in terms of popularity, bug triage and code review.
So when we ask people to get at resumption,
whether or not it's still a challenge,
we actually ask people to reflect on the last programming
task that they paused.
Whether it be in the last week, the last two weeks, et cetera,
et cetera.
And we ask them how long it would take to get back
into the task mentally.
And what you're looking at is a plot that effectively
shows that the resumption estimates
the longer someone's been away from something obviously
increase, and it becomes more challenging
or time consuming to get back into the task.
But the major takeaway here is even for recently paused tasks,
80% of the people said that it would take upward
of 5 to 10 minutes just to get back into something.
OK, so, having now said that, I want
to talk about Mercury as a system
and effectively how we use our findings
from our contextual inquiry and our survey
to design the system.
Mercury is a mobile programming tool
and it's built on principles of microproductivity.
And it's built on really two principles altogether
from the ground up.
And that is function-oriented microtasks and a ready anytime
task cue that you can effectively engage with as soon
as you pick up your phone and leave the workspace.
To bring our idea to life we integrated Mercury
into Visual Studio Code.
And the way it works is upon going mobile.
What Mercury will do is it will perform a real time
scan of the user's workspace to extract functions with source
code, without source code, and then
repeat the process for any other file that's in the workspace.
And from these extracted functions
we generate microtasks and you can generate
one or more microtasks.
There's not necessarily a limit for a particular function.
So just to give you an idea about what the task interface
itself looks like.
So you're looking at what people see on the phone.
And on the top left-hand corner of the interface,
you'll see the function name that the microtask
has been generated from.
You see the functions content or its context.
You'll also see the file name in the top right-hand corner.
You'll see the task panel on the center right,
and then on the bottom left will see the git branch
that the task is being pulled from.
So Mercury supports to different types of microtasks.
The first of which is grounded microtasks.
And grounded microtasks are inspired by principles
of function design and testing.
And the general premise here is we're automatically
generating unit tests for people to find use cases that
execute terribly or poorly.
So in some sense people are effectively
reviewing a recommended use case for this function call.
And the way it works is it parses the argument's presence
and then generates that unit test
for that particular function.
Now for anything that actually fails,
in terms of a unit test, what will happen
is, unfortunately it's a little bit blurry here,
you can't really see it but, on the left
you've got effectively the code before someone goes mobile
and on the after side, when someone's
completed a task while someone's been away,
Mercury will automatically generate
a source level edit in the user's code
while they've been gone.
So they effectively can see this edit
as soon as they get back to their workspace.
And just to be clear about what's actually
included in this edit, it's added to the docstring
and it includes the actual function call
in an optional note that the user can specify
about why the execution failed.
So alongside ground and microtasks,
there are also exploratory microtasks
and they're inspired by principles
of information seeking.
So in some sense this is actually
resonant of Michael's presentation just now.
The general idea here is that we take the function docstring
and actually use it as a query to Bing.
And from the results from Bing we
turn each result into a suggested recommendation.
So the premise here is people are reviewing stuff
on the go for an implementation that doesn't exist yet.
In other words, a function body that's empty.
And just to be clear about what shows up here, unfortunately
it's blurry again, but what's being injected after the fact
is the actual name of the web resource.
For example, this could be the name of the Stack Overflow
article as well as the URL from which it comes.
So that's Mercury in general and I
can talk about some more technical details
if time permits, but let's talk about how we evaluated it.
So we conducted a controlled lab study, a first use study,
inspired by best practices for evaluating both resumption
tools and software engineering tools.
And it started with a 30 minute phase
in which we invited participants into our study workspace
to begin the task.
We sent them away with a phone that had Mercury pre-installed
and then we invited them back in to the workspace
after their 30 minute period or phase two had been completed.
And then within the next 24 hours
we scheduled a 20 minute exit interview over Skype.
So just to be clear about the task
we needed something to actually give people.
Given that it's a controlled lab study.
And we resorted to a lot of what prior literature has used,
which is classic arcade games.
And we took the concept of Tetris.
Built it out ourselves and basically
made it a bit more fun and engaging.
So what we did is we added portals to Tetris.
In other words, if there's a piece falling
through the green portal on the right
it should come out on the left.
So it adds a bit of a new, unique challenge
but the premise here was we intentionally broke and left
four functions in the code base unimplemented.
And the task for our participants
was to effectively implement these functions.
So Mercury can be configured to, again, create
as many microtasks for each function as we want.
And we specifically targeted these four functions
configuring Mercury to create five tasks for each
of the functions, totaling up to 20 microtasks in total.
So just to get at what we measured during this study.
So again we had a pretty steady questionnaire
where we collected demographic information.
As well as readminister the same mobile work practices
scale that we used in our survey.
In terms of during study metrics we
had recording and log data that included the workstation
recordings that were recorded with TechSmith Camtasia
and we also had event logs from Mercury
as well as Visual Studio Code.
And finally we administered the post-study questionnaire
which included system usability scale.
A reattachment scale, that's generally
used in occupational health psychology to better
understand how well people are mentally
reattaching with their work.
And we also use the [INAUDIBLE] scale to get at whether or not
people's perceptions of themselves
were actually positive in terms of productivity, relaxation,
and engagement with their work.
So just to give a quick overview of recruitment.
We randomly sampled a company-wide email list
and the only criteria for effectively inviting people
into our study was that you have to be a programmer
and you have to have, a year's worth of JavaScript experience.
In total we recruited 20 full time programmers.
18 of which were male, 2 were female.
All were software engineers two of which were interns,
and 70% of our participants were between the ages of 25 and 44.
So there's a bunch of information
I can go through with our results.
I do want to go ahead and clarify
that because of limited time I'm going to breeze
through this a bit quick.
But in general Mercury was positively received.
85% of our 20 participants wanted
to bring Mercury into the programming workflow as is.
Only two people voiced complaints
about the device itself.
As one person said, it was nice because I
felt like Mercury was intuitively looking for things
I would have looked for anyway.
It also received good slash excellent usability scales
on average.
One of the most common threads through all
of the interviews that we had in our exit interview
was that there was a universal appreciation
for being able to continue work while on the go.
As well as continue or transfer information across devices.
Getting information between the devices is usually the problem.
Mercury kind of helps this by handling
the synchronization for you.
Now to get at the participants perspective in impact,
Mercury instilled feelings of productivity, engagement,
and relaxation.
And this was mostly universally said
across all of our participants.
There were only two or three participants
that actually said relatively negative things
about the system and we'll get to them in a minute.
But a part of how relaxed I felt came
with knowing that I no longer needed
to be in a particular place to be productive.
Now to quickly highlight the strength and shortcomings
of the microtasks.
Grounded microtasks are generally reviewed or perceived
as an opportunity to not only fix but also observe and see.
So simply being able to read the code that someone had just
written was a huge, huge thing for them.
But being able to actually take action on fixing it
was even better.
It was also noted that grounded microtasks obviously
introduce test cases that people didn't think about or consider.
Which isn't totally surprising because we
didn't tell people upfront that we
were wanting them to, say, build unit tests or write them.
In terms of exploratory microtasks
there was an opportunity gain a new perspective.
It's really how people viewed them.
It reframed the problem for me, as participant 15 said,
and there's also an element of surprise
that comes with completing them because you had
no idea what was coming next.
Now it's not all rainbows and sunshine.
There was a challenge with both microtasks, or criticisms
for both microtasks, in that they weren't necessarily
always relevant and sometimes they became repetitive.
But it should be noted that several participants had
very positive things to say about anything more accurate,
as participant 4 says, would win us the Nobel Prize.
And to really quickly highlights when Mercury would be used,
the usage scenarios were broad but generally touched
on moments of downtime.
And these range from commutes to brief breaks,
and we even had one person say that he would rather
use Mercury instead of play Temple Run while on the toilet.
So there's a really meaningful use
case that wasn't reported in the paper but you
and anyone watching online gets to hear about.
And finally, in terms of our findings,
resumption was a really big theme for several people
but not everyone.
As participant 9 said, Mercury kept things available.
In practice, Mercury would help bring down the ramp up time
when I get back to my workstation
and I can just go and code right away.
This was supported by a reattachment questionnaire
that we administered in our post-study questionnaire.
And so much of the comments about resumption
and its positivity were voiced about knowing what people would
do as soon as they get back.
So just to wrap up really quickly,
Mercury is a first step toward a new frontier
of on the go programming.
As participant 4 said, with Mercury I'm
not tethered to the desk as much as I would be
and I can still accomplish meaningful work.
And again, we see Mercury as the first step
toward a new design of systems that actually help people
be productive while away.
And similar to Michael's comments
about the notion of collaboration we also,
I won't necessarily say ignored, but we're
consciously aware that we were supporting
the individual and not something that was necessarily practical.
Now the ability to pull in reviewers,
and use canned comments, and add voice commentary,
how they code, et cetera, et cetera,
participants saw the practicality of the system
in practice and they regarded they the social aspect
of something new.
And there is a bigger story behind this,
some of this happened accidentally
in one of my earlier pilots, but maybe I'll get to that later.
And finally, as a final comment, I
do want to say that in succeeding in empowering people
in their mobility it has the potential to come at a cost.
Any work in this space in the future
should absolutely respect downtime.
Just to keep it brief, I'm out of time
and I'm happy to take questions though.
[APPLAUSE]
Hello [INAUDIBLE].
I have a question, thanks for an amazing talk.
So I think from the tool as you as users do microtask,
it looks like it is added as a comment in the codebase.
And I just wonder what the design rationale
is behind that.
So people like that comment because the clip is right there
or people that don't like is it cluttered or I just wonder.
It's a great question.
So I can tell you one of the limitations
that I didn't mention up here is, we assume that people
document their code.
That's a design decision that made sense to us
and I think there's an entire study waiting somewhere
for someone to take about understanding
how people document code.
How you can motivate people to document code
and whether or not this is really
as possibly intrusive in terms of documentation practices.
So the short answer is that probably matters.
But we didn't touch on it.
[INAUDIBLE] All right.
Thanks, awesome talk.
I think we all know that people don't talk in the code,
but I think you're getting at something really interesting.
The stuff that people want to do on the go
might be the things that people never have time for like email,
documenting code, bug fixes, review.
I'm wondering if you just vaguely ask
people, what do you want to do.
But have you examined this from a software engineering process
point of view of what actually makes sense for people
to be organizing their microtasks around?
In short, that's part of the reason why
we did our formative studies.
And admittedly the formative studies
were trying to get at what people currently do.
As opposed to what are all the opportunities
for software engineering.
So it was kind of a first step.
Admittedly there's tons of things,
documentation being one of them, the opportunities are there.
There's a follow up study dot, dot, dot somewhere
if someone wants to take that over.
Thank you again, Alex.