Trade-offs for Substituting a Human with an Agent in a Pair Programming Context:
The Good, the Bad, and the Ugly. By Sandeep Kuttal, Bali Ong, Katherine Kwasny, and Peter Robe
What is Pair Programming? Pair programming is when two people
work on the same programming problem together. One would take the role of “driver” who controls
the mouse and keyboard and inputs the actual code. The other would take the role of “navigator” who
provides directions, feedback, and suggestions to the driver. This programming technique can
be used both co-located or remotely, the latter becoming more common due to the COVID 19 pandemic.
The advantages of pair programming are easily explained by the common phrase,
two heads are better than one. Pair programming also increases productivity,
code quality, and self efficacy. It will also help to improve social skills and communication.
However, pair programming takes time to prepare and set up. There may be conflict of schedules,
technology problems, or a lack of diversity can cause issues.
So is there a way, when you can’t find a suitable partner, to simulate a pair programming
scenario while still retaining the advantages? This brings up the topic of conversational agents.
Artificial intelligence that is able to listen and respond to a human’s queries. These agents
have been becoming increasingly popular and complex with the amount of queries these
agents can respond to increasing as time passes. Our methodology consisted of simulating a remote
lab setting for both the human-human studies and the human-agent studies. For the human
agent studies we used the Wizard-of-Oz study where a human programmer would emulate AI.
We had gender balanced studies with equal pairs of MM, WW, and MW in the human-human studies, and in
the Wizard-of Oz studies we had 7 men and 7 women Our research focused on comparing human-human
interactions with human agent interactions using these three main research questions.
Research question 1: Can we continue the benefits of pair programming by replacing
a human programmer with an agent? Research question 2: What kind of
knowledge is transferred between human-human and human-agent pairs?
Research question 3: Do human programmers consider the agent as their partner?
And as we conducted the studies and analyzed the interactions, we also considered these questions:
How does gender come into play within those interactions?
What are the similarities between human-human and human-agent interactions?
What benefits does an agent have over a human? What are the negative aspects?
What is transformed?
Our first research question was, Can we continue the benefits of pair programming
by replacing a human programmer with an agent? No significant differences were found between
human-human and human-agent studies for Productivity, Code Quality, and Self-efficacy
Women performed better and had increased self-efficacy with an agent.
For the human-human pairs, their average productivity was 50.44% complete whereas
the human-agent pairs’ average productivity was 66%. Overall in our study, the productivity,
code quality, and self-efficacy scores increased from human-human to human-agent studies.
Our second research question was,
What kind of knowledge is transferred between human-human and human-agent pairs?
The agent made it easier for a participant who was stuck, to get unstuck and keep the
productivity flowing and helped foster creative solutions by prompting the participant.
However, the agent wasn’t able to explain the logic behind it’s solutions or code
and the agent wasn’t able to learn from a human as another human would be able to.
Our third research question was, Do human programmers consider the agent as their partner?
Humans were more willing to ask an agent for help, they also trusted the agent for simple tasks,
the human showed humility towards the agent like a human would show to another human.
However humans were able to ignore the agent without social cost or repercussion,
they didn’t refine the agent’s ideas, and did not trust the agent for complex tasks.
The trade-offs of an AI agent as a pair programming partner are as listed
Good: the agent acts as a non judgemental partner,
motivation for their human partner, supports active learning, creativity, and problem solving.
Bad: There is an excessive trust in the agent, and a lack of explanation
for the logic behind the agents answers. Ugly: The human won’t gain the interpersonal
and social skills as with another human partner. There is also a lack of informative
conversations on program logic where both partners are able to learn from each other.