Hi I'm Heloisa Candello from IBM Research
And today I'm presenting the work
Unveiling Practices of Customer Service Content Curators of Conversational Agents
I did this work with Claudio Pinhanez, Michael Muller and Mairieli Wessel
Conversational interfaces require two types of curation:
tech curation by data science workers
and content curation by domain experts.
Recent years have seen the possibilities for content curators
to instruct conversational machines in the customer service domain
for example, in the area of Machine Teaching.
The activities of curating specialized data
are time-consuming.
These activities have a learning curve
for the domain expert,
and they rely on collaborators
beyond the domain experts.
In this context, our two research questions are:
How do curators articulate their work
using strategies that are not previously supported
by the conversational platforms?
How do actors coordinate and integrate
their individual and collaborative activities
by means of such conversational platforms?
We present and unveil practices extracted
from eleven semi-structured interviews
and four design workshops with experts
in Banking,
Technical support,
Humans Resources,
Telecommunications,
and Automotive sectors.
First, we investigate the articulation work of the content curators
in training conversational machines.
Second, we inspect the curatorial
and collaboration strategies they use,
which are not afforded by current conversational platforms.
The studies were transcribed by two of the authors of this paper
and we applied the Thematic Analysis,
building from descriptive codes to themes.
From both studies,
we identified the main practices of curators
in a frequent case-scenario of resolving user intents
not correctly recognized by the conversational systems.
So, we have three main themes
Co-dependence work mechanisms of curators
Cooperation and Collaboration mechanisms
Management of information spaces and technologies
Answering our two research questions:
How do curators articulate their work
using strategies that are not previously supported by the conversational platforms?
We can highlight one of the findings that is
Content curators collaborate and cooperate with product owners, managers, and tech curators.
However, unlike the more immediate and direct-interaction concepts
of interactive machine teaching,
content curators were generally prevented
from directly ”teaching”
the components of the system.
Second question:
How do actors coordinate and integrate their
individual and collaborative activities
by means of such conversational platforms?
We can see:
Building and maintaining a conversational system
requires teamwork.
The strategies identified in our interviews and workshops
show that conversational platforms should work as a hub of tools
and resources,
a convergence point
to facilitate curators’ work and validations.
We draw the design implications and possibilities
to support individual and collaboration
collaboration curating practices.
Design Implication One:
We identified
a lack of direct access
for curators in conversational platforms.
Design Implication Two:
There are several tools that complement each other
to perform the curation tasks.
Synchronization of tools and actions in one place
would save time and work
on producing documents and material spreadsheets
to send updates by email, for example.
Design Implication Three:
These is a need to support multi-user collaboration,
and to provide notifications with
with a summary of updates
and tasks to be done
Transparency of update and validation actions
are time-consuming and affect productivity
to have the corpora updated for the final user faster.
Content curators would be able...
In the Design Implication Four,
Content curators would be able to work more effectively
if the support technology could unveil the content
that the chatbot knows
in a friendly and easy way.
Design Implication Five:
Conversational technologies should include features
and algorithms to track the semantics
and not only the syntactic elements of user questions.
AI systems can provide cooperation to the tasks
facilitating semantic search.
Design Implication Six:
There is a challenge to support and synchronize
similar updates in different languages.
Content curators need to overview
where specific contents or topics are available
in the chatbot corpus
in different languages
so that they can easily find where to update a topic.
Design Implication Seven:
Tools should support and synchronize the patterns
created to organize the work mechanisms.
For example,
tools should include
a pre-defined interface to
to ensure consistency
and help search for content
for content in the conversational platform.
We see this paper as
a preliminary work
to understanding the practices and the needs
of the content curators of conversational systems.
We believe that we have made the following contributions:
We unveiled nuances in the
everyday work practices of content curators,
which are often invisible to their colleagues in other team roles.
We showed how content curators
make strategic use of informal tools
and work practices,
to bring selected aspects of their articulation work
into visibility to their more technical colleagues.
We developed
seven recommendations for changes
in supporting technologies and practices,
as well as research questions for future work
Thank you so much
and please for further details check out our paper.