It’s my pleasure today to talk about the psychometric properties of a standardized questionnaire
to measure user experience – the UEQ.
Whenever someone interacts with a product, system, or service he or she experiences some feelings and emotions.
What exactly? That depends on the users’ characteristics, their previous experiences , and their current needs
but also on the product properties.
As the designer’s goal is to provide a pleasant experience, we need to understand the underlying processes.
One theoretical model that links the designer’s intentions with the users’ experiences
has been proposed by Hassenzahl in 2001.
He divided the different properties of a product into two classes:
the goal- or task directed ergonomic quality on the one side and the non-task oriented hedonic quality on the other side
When someone uses a product, he or she appraises ergonomic and hedonic quality,
which both together result in a judgment about the appeal of the product.
These judgments influence the user’s behavior and emotions.
While user behavior and emotions are observable, they do not provide a direct link to product properties.
However, for product improvement, we need this direct link.
UX, on the other side, is not observable directly.
But we may ask the user, for example, by a standardized questionnaire.
One questionnaire that has been widely used for more than 10 years and has been translated into different languages
is the UEQ by Laugwitz and colleagues.
It consists of 26 items, which are bipolar.
With these items, 6 different dimensions of UX are measured that match to the theoretical model by Hassenzahl.
If we use a standardized questionnaire, we expect to measure a construct objectively, reliably, and validly.
As the UEQ has a manual that describes how to instruct the participants, how to analyze the data, and how to interpret
the results, an objective measurement is possible.
So far, there are only a few studies with a limited number of products and a small number of participants that
evaluate the reliability and validity of the UEQ.
The goals of our research were,
first, to evaluate the reliability of the questionnaire by a wide variety of people and products.
Second, we wanted to know whether the items used indeed measure the intended constructs.
And third, we wanted to show whether the questionnaire is sensitive
to experimental manipulations of the product's properties.
In a first study, we combined the results of 23 previous user studies with more than 1 thousand participants overall.
The reliability of the UEQ ranged from acceptable for the scale dependability to good for the scale attractiveness.
To evaluate the construct validity, we run a factor analysis.
So, we expected that items that measure the same dimension, for example efficiency, load on the same factor.
Indeed, most of the items loaded on the correct factor.
However, there were also some items, highlighted in red, which had some minor loadings on a second factor.
We had even some items which loaded on the wrong factor
So far, it is unclear why this is the case. Maybe their semantics is ambiguous.
We also observed high intercorrelations between some of the scales.
Efficiency, dependability, and perspicuity which are all supposed to measure ergonomic quality
were highly correlated, as well as stimulation and novelty.
Thus, we run a second factor analysis - this time with a two-factor solution.
In this case, the items loaded as expected on pragmatic and hedonic quality.
In a second study when wanted to test whether the scales of the UEQ were sensitive to experimental manipulations
of product properties. Here, we focused on the two dimensions proposed in the model by Hassenzahl.
We started with designing a reference app.
This app was very simple with only one button to record ambient noise.
We assume that is quite simple for a user to reach the goal to record a sound,
That is, the ergonomic quality of this app should be very high.
However, the whole user interface was designed very simple.
Thus, the hedonic quality is probably very low.
For a second app, we added a nicely designed tutorial.
This tutorial describes to the user how he should measure ambient noise,
As the app is still very easy to use, the tutorial should not affect the ergonomic quality.
However, we hypothesized that the tutorial motivates the participants and
increases their interest in the task at hand.
Thus, the hedonic quality should be higher in comparison to the reference app.
For a third app, we added some technical features which guided the user to a correct measurement.
For example, recording a sound was only possible with a GMS signal.
We hypothesized that these features increased the complexity of the app.
However, without any explanations, it is probably more difficult to reach the goal to record ambient noise.
So, the ergonomic quality should be lower compared to the reference app.
In a fourth app, we added a tutorial that explained the new features.
So, reaching the goal should be as easy as with the reference app – the ergonomic quality should be similar.
Compared to the reference app, this app looks more innovative, which should be reflected
by an increase in hedonic quality.
We tested these hypotheses in a user study with about 500 participants.
Each participant was asked to use one app and to record ambient noise – as often and as long as they wanted to.
Afterward, they filled in the UEQ and also the System Usability Scale (SUS).
Indeed, the scales were sensitive to our experimental manipulations in the expected way.
In summary, the results of our two studies show that the scales of the UEQ correspond nicely
to the underlying theoretical model.
The reliability ranged from acceptable to good, which is sufficient for practical use.
With the UEQ, the perceived ergonomic and hedonic quality aspects of a product
as well as the overall judgment of appealingness can be measured validly.
However, the original six scales are highly correlated,
in particular the scales which measure the same quality aspect.
This makes the interpretation, in particular in research projects, difficult,
but it may still allow educated guesses about the areas where improvements will have the highest impact
in industrial projects.
Further research is necessary to include specific and more detailed aspects of UX in theoretical models
and to measure them with a standardized questionnaire.