Hi, my name is Anuschka.
I'm very glad to present to you our study on conversational agents for
information retrieval in the education domain together with my co-authors Thiemo
Wambsganss and Jan Marco Leimeister together,
we build and tested a question answering text based conversational agent called
Hermina that supports students in retrieving course relevant information and
posing course related questions at our university.
Our study shows that developing a theory motivated and user centered agent is
more effective in terms of task performance and students objective experience.
However,
deploying a conversational agent is not per se than most desirable option in
particular, when it is not explicitly designed with the user in mind,
why did we conduct this study?
So traditional information retrieval systems in educational settings
oftentimes suffer from delayed response times. Um,
also lack of information, quality, unstructured,
and excessive amounts of information,
and also enforced personal information disclosure considering yeah,
FAQ pages, but also question boxes where we have to leave our email.
For instance,
conversational agents promise to overcome these previously raised limitations by
changing how information is retrieved and understood how do they do that? Uh,
natural language processing enabled the conversational experience,
which is able to identify, understand and react to user intents. In turn,
the interaction becomes much more natural and social as conversational agents
follow a dialogue based interaction with the user and thereby oftentimes can
also incorporate other social cues. Uh, for instance, having an avatar,
a human name, following a human interaction structure,
which is especially beneficial for individuals with poor technical skills.
And lastly, this interaction is also easier,
faster and more effective in retrieving information as users can directly
acquire the correct response to their queries rather than browsing through a
collection of potential replies. However, current conversational agents,
especially in educational settings,
commonly follow a T default design and have only been partially examined
regarding their downstream consequences.
So external literature focuses mostly on the technical feasibility of these
systems, for instance, by improving the system's response qualities.
And at the same time,
what we know from theories such as social response theories by maybe focusing on
other design aspects,
such as the systems front end and by integrating social cues, these,
this can heavily foster the use and acceptance of a system as humans tend to
apply social heuristics and thus response to system in an equally social manner.
So beyond nonfunctional or focusing on nonfunctional aspects,
uh, can maybe help us explore how to best design such systems. Um,
and on the other hand,
we are interested in how users perceive and these systems and how these
affect the user affect the user in yeah. In the long term.
So also their behavior.
How did we do that? Uh, we designed,
developed and evaluated a conversational agent by following the build measure,
learn paradigm. So we first, uh, followed, uh, yeah,
an integrative design approach for creating a certain application in our case,
conversational agent by conducting both, uh, systematic literature review,
but also conducting semistructured interviews.
And later we translated these design requirements into specific design
principles,
which were incorporated in our final conversational agent term meeting.
Um, this step was followed by an evaluation, uh, with the end user.
So we conducted an online lab experiment,
which included an actual information retrieval task. And lastly,
this helps us derive, uh,
theoretical and practical implications from the evaluation,
which I will come back to later to create Hermine. Uh,
we first insulated the design principles into specific design functionalities.
So for instance, a lot of the users asked for a very proactive system.
So we included predefined question clusters and an always present feedback
button. For instance,
in the second step we developed and ma modeled the question answering intense,
and the main aim here was to cover the course content that would otherwise be
available to the students via the FAQ page.
So we modeled over 70 intents that included the introduction of the conversation
frequently asked questions and a casual dialogue.
And we thereby also added a variation of approximate utterances and user
queries for each, um, intent. And lastly,
we trained and deployed the question answering model on the classifier and using
semantic similarity matching with the classifier set at a score of 90%.
Uh,
we thereby drew on the framework space at chatbot and we ran an informal test
data collection of around hundred queries,
which we deemed as acceptable in terms of accuracy of H and responses
to evaluate our design, um, and assess the impact.
We ran an experiment which consisted of three key parts.
So we first had a pretest to collect, uh,
user dispositions and then, um,
had this information retrieval task as part of which participants had access to
the different agents and tools and answered six questions and the
post test, which measured users, subjective perception of the interaction.
And so for the mini manipulation, we had two, uh, conversational agents.
Uh, so treatment group one and two, which were built on the same backend,
I just explained, and data ever also relied on the same machine learning model,
trained on these 70 tens, allowing the two, uh,
conversational systems to provide adaptive and personalized answers to the user.
Only the ins instantiation of the design principles, uh,
in the front end were manipulated.
So these were present for Hermine treatment group one,
the traditional question answering system served as our control group, uh,
which listed the frequently answered, uh,
questions with subsequent explanations.
In a flowing text format to evaluate our hypotheses, uh,
that a learner center,
conversational agents or Hermina will improve student's ability to retrieve
learning related information and positively affect their subjective sub
subjective per um, yeah, perspec, uh,
perception of the interaction we captured both self-reported items in the post
test, but also measured their, uh, or captured their answers for the questions,
uh, to measure their task performance. What we found, uh,
was that users who interacted with Hermine had significant higher levels of
correct answers for both multiple choice and opened ended questions.
And beyond the averages for these two question types,
the shown table also demonstrates the same statistical analysis for each
question. Um,
what we also see is that for both multiple choice and open ended questions,
our results do not exhibit any significant differences between treatment group
two. So the non design driven agent and the traditional tool,
except for the first open ended question in a similar vein users
who interacted with Hermina also trusted provided information, more reported,
higher levels of enjoyment,
and which comes at no surprise rate to the system is higher in social present
and presence. And our postdoc tests confirm these results.
So our study illustrates the importance of conversational question answering for
your educational setting, namely in an information retrieval task,
on course related content, uh, the user-centered agent offers to potential,
to overcome limitations associated with traditional tools by allowing users to
receive specific information faster while simultaneously reducing the
administrative burden. We see on the educational institution side,
we also find theoretical congruence in that developing and deploying design
principles, largely focused on social cues, um,
can have an effective and positive impact,
which confirms our current understanding on social response theory. However,
the embedding of social cues actually represents a double edged sword, uh,
because inappropriate amounts of social cues, uh, might not properly reflect,
uh, an agent's competencies. Uh,
this is particular the case in a high stake setting where users are required to
challenge provided information. Um,
and this points towards a more fundamental conflict between designing for task
achievement and overarching ethical questions,
which should guide the design process. And lastly, we,
I integrated the effectiveness of our conversational agent based on the
correctness of six answers to course related questions.
So this backs the question to what extent our results hold when deploying
Hermine for conflicts questions in varying context,
and also with different users. In addition,
it is unclear what the long term effects are of using a conversational agent,
such as using it repetitively or over a longer period of time.
Thank you very much for listening and happy to discuss our study.
So please feel free to reach out.