Hi everyone! My name is Ziang Xiao from University of Illinois at Urbana-Champaign.
I am here to present our work on using an AI-powered chatbot to conduct conversational survey.
This is joint work with Michelle, Vera, Gloria, Changyan, Wenxi, and Huahai.
As you may already be familiar with,
survey is one of the most widely used research methods in HCI and many other disciplines.
We use surveys to learn many things about a person, such as behavior, attitude, and preference.
Recently online survey becomes popular due to the low cost of gathering data,
ease of survey administration, and broadening reach across a variety of population worldwide.
However, it is not easy to collect high-quality data through surveys.
According to Survey Monkey,
on average, people spend 5 minutes to complete a 10-question survey
but only 10 minutes to finish a 30-question survey.
The average time per question decreased by 30%.
It is even harder for surveys with open-ended questions.
Participants need extra time and effort to formulate and type responses.
They could easily become bored, tired, or uninterested in the survey
and stop providing high-quality responses or stop responding.
In our study, we explored a new way of conducting surveys with an AI-powered chatbot,
where a chatbot can frame survey questions in a more personalized, conversational way.
However, no one knows if a chatbot could conduct conversational surveys successfully.
For example, a turn-by-turn chat requires participants to take extra time and effort to complete a survey.
And current chatbots are far from perfect
Their limited conversation capabilities may lead to user disappointment and frustration.
And a participant may not always be cooperative.
Some may give gibberish answers
and some may try to digress the conversation topic.
Therefore, we designed and conducted a study to answer two research questions.
First, How would user response quality differ,
especially the quality of open-ended questions in an AI-powered chatbot-driven survey vs. a traditional Qualtrics survey?
Second, How would participant engagement differ in an AI-powered chatbot-driven survey vs. a Qualtrics survey
We compared the same survey in two different formats with 600 participants,
one using an AI-powered chatbot created on Juji platform
and one form-based survey on Qualtrics.
Like a form-based survey, as you can see here, a conversational survey could be easily created via GUI and disseminate via URL online.
To measure survey response quality, we propose a set of measures based on Gricean Maxims.
We measure a piece of response from dimensions inlcuidng Quantity, Relevance, and Manner.
Our result indicates the chatbot outperformed the form-based survey on all dimensions
including response quality and user engagement.
The participants who completed a chatbot survey offered much more relevant, specific
and clear responses to open-ended questions than their counterparts did in a Qualtrics survey.
And the participants were willingly spending more time with the Juji chatbot
writing longer responses, and disclosing more information about themselves in depth and scope.
Our work suggests a promising method for conducting effective surveys with open-ended questions.
We list a few design implications for creating and employing chatbots for survey success.
For example, we can equip a chatbot with rich interview skills, such as active listening skills
We can also mix a chatbot with form-based surveys to achieve
the best results for both open-ended questions and quantitative questions
Also we should explore the possibility of using a chatbot to conduct more complex information elicitation tasks, such as semi-structured interviews
As you can see, creating a conversational survey is very simple
and most importantly, a conversational survey elicits higher quality information
and drives higher user engagement.
I would encourage you to give it a try when you are planning your next study.
At the end, we would like to thank for you attention,
our participants for their participation, and editors and reviewers for their insightful feedback.
If you have any questions, I am more than happy to take them online or through email.
Thank you!