Hi, my name is Alexander Prange from the German Research Center for Artificial Intelligence
(DFKI). I am presenting an Explainable Automatic Evaluation
of the Trail Making Test for Dementia Screening, on behalf of my colleagues Michael Barz, Anika
Heimann-Steinert and Daniel Sonntag. If you'd like to know more about our research,
please visit our website. Our paper is focused around neuropsychological
testing and cognitive assessments that allow to diagnose several neurological conditions,
including dementia. The problem is that neuropsychological testing
is mostly done paper-pencil-based, which includes a time-consuming manual scoring period.
In addition, there is no direct digitization and the results are open to subjective interpretation.
In this paper we present an Interactive Congnitive Assessment Tool, that allows for the real-time
digitization of tests using a digital pen. Our system provides automatic and explainable
scorings with direct export to hospital information systems and includes visual feedback for clinicians.
Our system is potentially more objective and compared to other proposed methods does not
change the clinically validated testing methods. We focus on the Trail Making Test, which is
a commonly used, standardized and validated cognitive assessment.
It is a paper-pencil test on an A4 sheet of paper, where the task is to connect numbered
nodes, similar to a child’s connect-the-dots puzzle.
There are two versions, TMT-A which has only numbers, and TMT-B which has alternating numbers
and letters. The main scoring metric is the execution time
which is then compared to age-specific normative values.
To collect our data we used the Neo N2 smartpen, which is a ballpoint pen with an integrated
camera that can stream its data in real-time via Bluetooth to a nearby tablet.
In the lower left you see our setup in the geriatrics daycare clinic. In the ride side
of the picture you can see the participant performing the TMT using a digital pen.
The paper is imprinted with a special micro-dot pattern that is recognized by the camera of
the smartpen and then translated into a real-time stream of x/y coordinates.
In our hospital use case we deploy a distributed architecture consisting of multiple components.
After the patient has performed the test, the raw pen data is streamed to both, the
document processing and indexing server as well as the pen data processing server.
Both work together to analyze the assessment in its entirety.
The resulting automated scoring is then transferred to the hospital information system where it
can be accessed through the clinician interface to be evaluated by a clinician afterwards.
The entire process takes place in real-time and is also deployable in a general practicioner’s
office in a lighter version. Here you can see a real-world example of our
automated scoring output. On the left you can see the unaltered sample
for comparison. Whereas in the middle you can see the analyzed
version after being processed through our automatic recognition system.
The colored segments visualize the output of the algorithm, thereby providing transparent
feedback to the clinician. As part of the recognition process the algorithm
also produces a set of explanations for the result.
These explanations are depicted on the right and include correctly and incorrectly connected
paths as well as missing nodes and errors. All this information is automatically summarized
and exported into a structured PDF report. We also provide an unaltered real-time playback
of the input. Our evaluation is focused on showing that
the automated scoring system performs at least as good as human raters.
The digital pen also allows us to analyze additional parameters that could not be considered
otherwise. Our hypotheses are as follows:
H1 The scoring results produced by the automated system are as accurate as the results produced
by clinicians. H2 The transparent explanations generated
by our system are equal to the assessments made by clinicians.
H3 The additional, automatically measured parameters correlate significantly with the
TMT’s main performance measure, which is execution time.
The additional parameters include pause duration, velocity, pressure, pen lifts and others.
For the experiment we recruited 40 participants from a geratrics daycare clinic.
Their minimum age is 65, half are female and 95% are right-handed with mostly high and
mid-level educational backgrounds. During the experiment participants sit in
a distraction-free room at a table opposite of the clinician.
Our experiment has two conditions, paper-pencil and digital pen. The order is balanced.
The experiment task is to perform first the TMT-A and then the TMT-B.
Overall, we include 2 clinicians to diminish personal instruction bias.
Our results look promising. Testing H1, the comparison of manual vs. automatic
time measurements shows high Pearson correlation values and intraclass correlation coefficients.
However, there is a small but significant difference in the time measurements of around
a second, which we attribute to slow human reaction times when triggering the stopwatch.
Regarding H2 (transparent explanations), we find that our automated results are as good
as human ratings. Several of the additional measured parameters
in H3 show high correlation with the test results.
These include average pause duration, average velocity, velocity inside nodes, number of
pauses and pen lifts. In summary, we have presented an automatic
scoring system for the trail making test. Thank you for listening.