- One of the most misquoted sayings in business is:
"If you can't measure it, you can't manage it."
This statement is meant to say that, to improve something
we need a precise metric that captures it
and that should be tracked in order to understand
if our efforts to improve it or actually effective.
It's interesting
that this quote is actually the complete opposite
of what W Edwards Demings originally said.
He said, "it's wrong to suppose
that if you can't measure it, you can't manage it.
A costly myth."
This difference between the original
and the commonly used version highlights why
it's dangerous to rely on a single metric to
assess how well a business is performing
that one metric can be manipulated
in ways unrelated to what it's supposed to measure.
This is the phenomenon described by Campbell's law.
Campbell's law states that the more important a metric is
in decision making, the more likely it is to be manipulated.
In other words
when a single metric is used to determine success
or failure, human beings are likely to try to
optimize their behavior to improve that metric.
Sometimes with ridiculous or dangerous consequences
people manage the metric rather
than using the metric to help manage the underlying issue.
The following story was shared by a product designer working
for a TV streaming subscription service.
This company was highly concerned with customer retention.
The longer subscribers stayed with it, the more money made.
As a result, the company's leadership wanted to
reduce subscription cancellations.
The company defined saving an account
as occurring if someone entered the cancellation flow
but then abandoned it
and didn't actually cancel their subscription.
While this metric was intended to capture the ability to
change consumer's minds and make them want to stay
with the company, it had an unintended consequence.
It incentivized the company's designers to
make the cancellation process intentionally difficult
and in user testing.
They observed that customers would become overwhelmed
while trying to cancel their accounts online
and they'd give up, in the company's analytics metrics.
This type of behavior was counted as saving an account
but the account wasn't actually saved.
The customer simply called the customer service
department to cancel instead.
As a result, the metric looked really good
but the outcome of this farce was angrier
and more annoyed customers who were not likely to return
or recommend the company, higher costs
because the company had to pay
for the customer service calls instead
of letting it happen online.
And then accounts that were still eventually
canceled anyway.
So how can UX professionals avoid the trap
of tracking metrics that are easily abused?
First, recognize that all metrics are limited.
A metric is just a signal that reflects the goal
or outcome you're seeking, but is not the full picture.
The popular North Star framework where
a single metric is chosen as the overall health indicator
for a company can be really dangerous
because it amplifies the inherent limitations
of that individual metric.
Second
combine metrics together to obtain a fuller understanding
of the phenomenon that you're trying to study.
Satisfaction ratings can be a useful metric
but they can be a lot more powerful when combined
with behavior metrics such as success rate.
It's a lot harder to game several metrics
than to gain any one of them.
Third, never rely on quantitative measures alone.
Triangulation with qualitative data, such
as the insights that come
from user interviews, usability tests
field studies and diary studies help you to
understand the nuanced consequences
of design choices that might otherwise be missed entirely
if you just rely on passively collected analytics data.
- Thanks for watching.
If you want to see more of our UX videos
take a look at these over here and consider subscribing
to our channel, on our website, nngroup.com.
You can access our free library of over 2000 articles.
You can also register for one
of our UX courses that offer live hands-on training.