In this video, I'm talking about how to create an effective
chatbot for interview. My name is Ziang. I am from University
of Illinois at Urbana Champaign. I completed this work when I was
an intern at Juji. It is a joint work with Michelle, Wenxi,
Huahai, and Changyan. In the past a few years, we've all
witnessed a chatbot tsunami. Through a text-based
conversation, the chatbot has been adapted for a wide variety
of application. For example, designing an AI chatbot as a
teammate, where the chatbot can learn team characteristics and
help teaming. People design an AI chatbot as customer
representative as well, where the chatbot answers a customer's
question and provides solution. An AI chatbot has also been
created as an academic advisor to engage with students and help
them throughout the semester. Among various chatbot
applications, a promising one is information elicitation.
Interview chatbots engage users in a text based conversation to
draw-out their views and opinions. Both researchers and
practitioners have been using chatbot to engage with their
target audience and draw-out insights. Besides the benefit of
scalability, recent studies also shown that using an interview
chatbot can not only elicit rich and truthful information, but
also delivering engaging experience. However, building an
effective interview chatbot is not an easy task. We are facing
many challenges. Understanding user natural language input is
difficult, especially when open-ended questions are
involved. Users may give highly diverse, complex or even
ambiguous answers as shown on the right. Second, the chatbot
needs to handle complex conversation situations. Natural
Language conversations are nonlinear and often go back and
forth. Users might ask clarification questions in the
middle or simply do not know how to answer it. No need to mention
that some users are not so cooperative by providing
gibberish or meaningless responses. Third, it is
difficult for chatbot designers to take advantages of AI
advancement due to a lack of expertise or resources.
For example,
although deep learning methods have enabled powerful
conversational AIs and might help address the first challenge
mentioned above. Such models require a large amount of high
quality training data, which are hard to acquire, especially for
chatbot designers with
non AI backgrounds.
Given the three challenges, in this study, our goal is to
explore new ways to build more effective interview chatbots,
especially with off-the-shelf and accessible technology. As a
first step, we want to equip a chatbot with active listening
skills, which often being used by human interviewers to make
the interview participants feel being heard. Studies have shown
that active listening skills can make an interviewer more
socially attractive
create more engaging interview experience
and elicit high quality responses. For example,
paraphrasing is one of the techniques to actively listen
The chatbot tries to convey such understanding by restating
a user's input. Another technique is verbalizing emotion
where the interviewer reflects a user's emotion in words to
show empathy. Encouraging is also an active listening technique
The interviewer offers idea and suggestions to encourage conversation
Here is a demo of an interview chatbot with active listening skills.
In this study, we hypothesized that the interview
chatbots with active listening skills would be more effective
at conducting interviews and engaging interview participants.
Given the benefits of leveraging active listening skills during
an interview. In this study, we asked how to teach chatbots
active listening skills.
The second question we want
to ask is how good are chatbots with active listening skills?
We want to know
how effective can such interview chatbot be at handling complex
and diverse user inputs and affecting user experience and
interview quality.
When designing our solution, we faced many options. We use three criteria to build our prototype.
First, it is reproducibility. We selected the only publicly
available technologies so other researchers and practitioners
can easily reproduce our work. Second, we prefer technologies
that require low resource, little training or customization.
Third, adoption. We favor technologies that
non-AI experts can use to build custom interview chatbots
In the next section,
I will give an overview of our prototype.
There are two parts of our prototype
Juji's base system for conversation
and an extension for user intent prediction.
Juji is a publicly available chatbot platform that has a
rule-based system allows a chatbot designer to create a
chatbot with a GUI, and the platform provides interfaces for
conversation. The chatbot designer needs to create an
interview agenda, and the generated chatbot will follow
the agenda to conduct an interview. The Juji platform has
the ability to manage nonlinear conversations where the
interviewee can go back and forth. Also, Juji platform can
be easily extended with third party extensions or API.
To power interview chatbots with active listening skills.
The key focus of our prototype is to interpret the meaning of complex
and diverse user input. In the extension, we build task
classification models to differentiate user intents from
their input. In our extension, we have a set of components to
facilitate the construction of those text classification
models, especially for non-AI experts. The intent
classification model created by our extension is platform
agnostic, which means it can be used on any other chatbot
platform like Juji.
Our extension has a human-machine collaborative
labeling component that aimed to facilitate chatbot designers
to create high-quality dataset for the later training. The process
is progressive, where the chatbot designer can
bootstrap interview data all the time to increase the size of the
training data and make the later model more robust.
The human-machine collaborative labeling component has an intent discoverer
and a sentence ranker
Intent discoverer aims to help the
chatbot designer to identify potential user intent. We use
topic modeling algorithm, LDA, to summarize a set of user
responses with different topics. It is an unsupervised method
requires no prior training. Since LDA extracted topics have
no order, we enhance the result by ordering the topics with
their coverage to help the chatbot designer
identify intents that have been mentioned the most.
The goal for the sentence ranker is to find examples that
best-matched to the identified intents to create high-quality
training dataset. Since the extracted intents are typically
summarized by a set of keywords, it is difficult to label the
intent by the associated keywords due to missing context.
Thus, the sentence ranker will automatically rank user
responses by their semantic proximity to an intent. The
training dataset is created by including high-ranked sentences
as positive examples and low-ranked sentences
as negative examples.
Then, with the created training
dataset, labelled text will be encoded using techniques like
sentence embedding. Later, the model builder will take the
encoded sentences to build text classification models to
identify user intents.
At the last step,
chatbot designers can write different responses
corresponding to the text classification results. In our
cases, we equip a chatbot with responses
that incorporated active listening techniques.
Next, I will talk about the evaluation of our
prototype. There are two parts of the evaluation. The first
part evaluates the ability of our prototype to detect user
intents, and the second part evaluates the quality of a
chatbot that created by our prototype with active listening
skills. In the first part of the evaluation. We focus on how our
prototype can predict user intents when handling user input
The accuracy of these models directly determines how well
an interview chatbot can listen actively during an interview
Here, we present the results from average performance of
the best models per interview topic. We choose four interview
topics that are often used in our interviews.
There are a topic about self introduction , a topic about participants
hobbies, a topic about their best quality, and a topic about
the current challenges they're facing. Those topics are often
used to build rapport and used to elicit user responses. At the first
step, we focus on improving general interview topics.
We tried different text classification models for each topic and
select the best here. Overall, the logistic regression and
linear SVM performs the best. From the data, we achieved about 85%
precision in differentiating user intents. The results demonstrate
that our text classification model can effectively
dentify user intents. the ability of predicting user
intents allows our prototype to deliver correct active listening skills.
To evaluate the effectiveness of our chatbot
active listening skills, we also conducted a live chatbot evaluation
with end users. In this evaluation, we compared our
chatbot with active listening skills with a baseline chatbot.
The baseline chatbot is created by Juji's base system without
our extension. The participants are randomly assigned to be
interviewed by either of the chatbot. We recruited
total of 206 participants from Amazon Mechanical Turk.
108 participants engaged with our chatbot with
active listening skills, and 98 of them chatted with
a baseline version. There are 96 female participants and 110 male
participants. Most of the participants are between 18 to 34 years old
To measure the effectiveness of our interview chatbot,
we evaluated from two perspectives, user experience
and interview quality. the user experience includes
engagement and perception. User engagement measures
the engagement duration and response length. And the user
perception asks participants how well the agent comprehend
their input, their willingness to be interviewed by the same agent
again in the future and the overall chat experience
We believe it is a more comprehensive measure of user experience by combining both objective and subjective measures.
measures. Last, we measured response quality as
an approximation for interview quality. The measures of
response quality consider both volume and richness.
The informativeness measures the volume of information which
is calculated by Shannon's information theory. And then
we hand-coded all participants responses and calculate the
response quality index for each participant.
The Response Quality Index is adapted from prior research on chatbot survey
which considered three aspects of a user response, relevance, specificity and clarity.
More details about those measures can be found in the paper.
We analyze our data by using ANCOVA analysis. Compared to the baseline chatbot,
our chatbot with active listening skills can better engage the
participant and elicit higher quality information. People who chatted
longer with our chatbot, and provided lengthier responses.
They also explicitly mentioned that they felt the agent can
better understand their input and they are more willing
to chat with the same agent lat er. The overall user experience
is also better. The analysis on collected
responses also indicated that participants provided
higher quality responses to the chatbot with active
listening skills. All results are statistically significant.
As a result, we can answer our research question firmly.
To the first question, our prototype system demonstrated that
it's feasible to use existing and practical AI technologies
to build effective interview chatbots with active listening skills.
Second, our chatbot with active listening skills are more effective
at engaging users and eliciting quality user responses,
compared to those without such skills.
There are a few future directions we want to explore. First, it is important to interpret deeper user intents and the semantic gist of user input
Second, the chatbot in the future should be able to
generate and ask follow up questions intelligently to
deepen the conversation. Third, learning the effect of
individual active listening technique can better inform the
chatbot to exercise the most effective technique based on the
context. Our work provides three contributions. First, our work
demonstrates practical approaches to build effective
interview chatbots. The way we design our prototype can be used
on any existing chatbot building platforms. Second, our work also
provides insights for creating hybrid framework for developing
progressive chatbot platforms. The chatbot designer can
collaborate with machine and algorithm to create a more
sophisticated chatbot by bootstrapping collected user
responses. Third, equipping a chatbot with active listening
skills not only benefits interview chatbots, but also can
be adapted for a broader applications, such as emotional
support to create empathetic AIs.
That concludes my video. Thanks for watching. At the end, I would like to thank all
reviewers for their feedback and the Air Force Office of
Scientific Research for supporting this project. I'm
happy to answer any questions you have for the paper. Please
send me an email. Thank you