Many people in New Zealand face mental health issues, some of which are serious enough to
require hospitalisation and highly specialised care to support people through some of the
most difficult periods in their lives.
How do the people who provide mental health services
measure how much of an improvement they make in the lives of their patients?
To find out, we undertook a performance audit focused on people who have spent at least
one night in an inpatient unit in a mental health hospital. We wanted to understand how
well these people are supported in their transition back into community-based services to give
them the best chance to live with as little disruption as possible in the presence of
their mental illness.
To help us do that, we used millions of anonymised patient records to build our own data model
which placed people at the centre. We did this because we were determined that the focus
of our work should be the people, rather than the systems which serve them.
Our data model allowed us to analyse how well the numbers reported by the sector reflect
any real improvements in the service. The two indicators which we studied in detail
were re-admission and follow-up rates. We discovered that these measures are heavily
impacted by a small number of patients, a factor which can skew the overall picture
and make trend analysis difficult.
However, we also consider that there is great potential to do more with the available data
to create better measures that more accurately represent service outcomes beyond aggregated
averages and tallies.
By making our thinking transparent and by incorporating feedback from clinicians and
analysts in the sector, we constantly refined the quality of our analysis to develop the
best independent perspective we could.
In keeping with our aim to focus on outcomes for people, we built upon some innovative
thinking being done by a team of analysts at one of our DHBs.
By plotting the intensity of a patient’s interactions with the service, they could show
the extent to which peoples’ lives were being disrupted by their mental health conditions,
as well as indirect evidence of whether things looked like they were getting better or worse.
They called this kind of picture a “patient journey”.
When they combined the journeys of many people in a single picture using a heat-map to show
the more intense spots, they discovered that there were many variations in patient experiences
and that studying these patterns could help clinicians better understand the needs of
their patients. They could also help service designers think of ways to more accurately
measure outcomes.
Our audit team adapted the patient journeys concept and began to develop interactive visual
maps of the lives of the people we were studying.
Starting with an empty timeline stretching out
over the four years for which we had data, we began to colour in the various types of
interactions between patients and the different service teams. We added markers for some of
the existing indicators and also included interactions with non-mental-health services
(such as visits to Accident & Emergency wards, or treatments for chronic conditions such
as diabetes).
When we drew these pictures for all 20,000 people in our study, some very interesting
and informative patterns emerged. Most people spent short periods in hospital, but a small
number seemed to stay for months or even years at a time. Many people received regular care
both before and after a hospital stay, but there were others who seemed to arrive in
hospital abruptly and then seemingly disappear just as suddenly. These pictures can’t capture
a person’s complete story, but they did help us to ask better questions.
Multiple patient journeys viewed together resemble a kind of “fingerprint”:
each individual story unique but also a thread in a much wider narrative weaving thousands
of peoples’ lives together through their interactions with mental health services.
By comparing the fingerprints of different DHBs we could see how, despite some significant
differences, many larger patterns remained similar. Each DHB, regardless of size,
has a small proportion of patients with complex needs whose interactions look very different
to those of most patients.
In fact, these people with highly complex needs make up a small and identifiable group
of fewer than 100 people nationwide. These are highly impactful patients with unusually
long hospital stays covering months or even years at a time, yet they are not well represented
by the existing performance indicators which don’t differentiate between groups of people
with distinct needs.
Some health services elsewhere in the world have adopted a data science approach toward
using patient data to cluster similar patients so that they can better classify risks in
an attempt to predict critical events such as relapses before they happen.
Our summary view is that the mental health sector collects a wealth of valuable data
about the people it services. There are many energetic and innovative people working in
the sector who are trying to find ways to use this data to improve upon how services
are measured, designed, and ultimately delivered for the benefit of the people they serve.
We hope that our work has shed a helpful light on a few of these innovations and that the
sector finds ways to improve its support for the people who developed them. Doing so will
move the sector closer to realising the untapped potential of its extensive data collections.
Mental health services are highly complex and demanding, and that’s unlikely to change.
We believe that the collaborative work on this performance audit is a showcase for how
agencies can combine their efforts, share their data, pool their expertise, and act
collectively to improve services and build better lives for the people of New Zealand.