I am Zijian Ding Jason. And I am presenting our work “TalkTive, A Conversational Agent
Using Backchannels to Engage Older Adults in Neurocognitive Disorders Screening”,
This work is done with Jiawen Kang, Tinky Ho, Ka Ho Wong, Helene Fung, Helen Meng and Xiaojuan Ma.
The rapidly ageing global population is imposing challenges to healthcare systems across nations.
Over fifty-five million of people worldwide were living with
dementia, a common type of neurocognitive disorder
(NCD) in 2020, and the global cost of NCD exceeded the threshold of US$1 trillion in 2018.
To look on the bright side, NCD has been shown to be controllable if patients can get access to
early diagnoses and timely preventive interventions. This presents a necessity
for a scalable approach in screening cognitive impairments among older adults.
Since NCDs are often manifested as communicative impairments, machine
learning al- gorithms are offering a new type of support for NCD screening.
Conversational agents (CAs) have great potential in NCD screening. Backchannels,
a kind of listeners’ response intending to hear from the speaker instead of taking a turn,
could be applied to CAs to engage older adults and elicit conversational speech for NCD screening.
As an initial step, this paper presents research in developing
the backchanneling ability in CAs in the form of a verbal response to engage the speaker.
This study includes three stages: data analysis of real-world cognitive test conversations,
development of backchanneling algorithms and user evaluation of a proof-of-concept system.
The language we studied was Cantonese, a low-resource language.
In Stage 1, we analyzed 246 conversations of Montreal Cognitive Assessment (MoCA) between
older adults and human assessors, and derived the categories and coding scheme of reactive
backchannels, RBCs, like “yeah” which showed understanding and agreements to the speaker,
and proactive backchannels, PBCs, like “anything else” which encouraged the speaker
to speak more. The inter-rater reliability of our coding scheme is Cohen’s kappa 0.803.
Data analysis on RBCs and PBCs in those conversations showed the interval before PBCs
might be longer than that of RBCs, and the number of PBCs given by expert assessors varied along 3
dimensions: within task - PBCs given throughout the progress of a task, between tasks - PBCs given
in two kinds of task, and between subjects - PBCs received by different participants.
Given the difference in functions and timing between RBCs and PBCs,
we developed two different models to generate these two kinds of backchannels separately.
For RBC prediction, a series of experiments were conducted to obtain the best-performing
configuration of features and models, and we selected the combination of ComParE features and
SVM model, and used RBCs coded in data analysis as training data. When a speaking interval was
detected, the previous speaking utterance followed by this interval would be segmented. Then,
the speaking utterance was fed into OpenSmile feature extractor to obtain selected ComParE
features, and those features were used by the trained SVM model to make RBC predictions.
Considering PBC prediction, based on insights of data analysis,
placing PBCs after a fixed-length pause was not enough.
Hence we introduced Progress Score, Pause Score and Participant Score to give adaptive PBCs.
Participant Score was calculated by passing the speech of the exact same sentences to SVM model.
In a certain task, Progress Score was updated as the task was going on, and when speaking intervals
occurred, Pause Score would be calculated as the length of pause at that time stamp. Those
three scores were independently calculated and a weighted sum of those scores was used
to trigger PBCs. The threshold of the summed score was dependent on the task.
To evaluate our backchanneling algorithms, we developed TalkTive,
a CA which can predict both timing and form of backchanneling during cognitive assessments.
It mimicked a cognitive assessment with 9 preset tasks.
Our study invited 36 older adult participants, aged from 61 to 84 with an average of 72.4.
We evaluated the backchanneling features in three conditions:
Condition 0 with no BCs, Condition 1 with only RBCs, Condition 2 with both RBCs and PBCs.
12 participants were assigned to each condition.
Here is an example of how older adult participants interacted with the system.
[Cantonese speech between participant and system, translation on screen]
We invited a trained assessor to evaluate 518 RBCs, 131 PBCs given to 24 participants in
Condition 1 and 2. Around 89% of system-generated backchannels were validated as appropriate.
Participants’ ratings on the post-study questionnaire showed that our system with
proper backchanneling features were not perceived as disturbing by older adults.
There is no significant difference among three conditions in participants’ ratings
on those negative statements: “she seemed impatient”,
“s​he interrupted me” and “she seemed cold and unfriendly”.
And the user interview indicated that PBCs were more appreciated by older adult participants
than RBCs, especially when they ran out of answers and were about to give up.
Here are two quotes of participants in Condition 2 with both RBCs & PBCs.
“(I) think it was operated by a human. For example,
just now when I had not answered, it was listening to me carefully,
and encouraging me in the meantime, just like doing Q&A with you in person. It felt like that.”
“While I was thinking, (it said) ‘uhm’, ‘keep going’, so I would really think (about the
answers), much better than if it didn’t give any response...because it made me feel like
that there was really a person there, ‘take your time’, not like facing a cold computer.”
Those quotes indicated that PBCs were appreciated by older adults
in the context of cognitive assessments.
That’s the end of my presentation. If you want to know more please check our
paper and feel free to contact me. Thank you for watching!