Hi everyone, my name is Marit Bentvelzen, and in this video I will present our work on the
Technology-Supported Reflection Inventory. Reflection is an often addressed design goal
in Human-Computer Interaction (HCI) research. And an increasing number
of artefacts for reflection have been developed in recent years.
These design efforts span a wide spectrum of applications, including technology that aims
to encourage users to reflect on their mental health, physical activity or their daily life.
Reflection theories highlight that reflection often does not occur automatically,
but that it needs to be encouraged. Technology could offer such encouragement.
An example comes from Trackly, a mobile app that helps patients with MS to manually track
their symptoms and to reflect on the collected data through visualisations.
Another example is LOOP, an ambient system that aims to encourage reflection on physical activity.
These artefacts can potentially encourage reflection; yet it remains challenging to
evaluate if and how an interactive technology helps users to reflect.
In a systematic review from 2014, Baumer et al address this lack of consensus regarding
the evaluation of reflection support systems. Studies that used quantitative
assessments rarely measured reflection, but rather focused on user experience.
Qualitative evaluation approaches encounter similar difficulties, such as the lack of a
clear definition of reflection and not asking users directly about reflection.
This lack of proper evaluation methods makes it difficult to
compare different design alternatives, which complicates future design efforts.
To address this gap, we aimed to create a ’artefact-centric’ instrument that enables the
evaluation of technology-supported reflection and measures the qualities of an interactive artefact
that lead to reflection. To this end, we developed the Technology-Supported Reflection Inventory
(TSRI), a scale that evaluates how an interactive system fosters personal, data-driven reflection.
We began the scale formation by generating initial items for the scale. Each researcher first created
items, based on related work, on their own. We then arranged a discussion session in which all
the generated items were merged and discussed. The initial list of items was then subjected
to two rounds of expert reviews, resulting in a final list of 58 items.
We then designed an online survey consisting of these 58 items, and we recruited participants
through Mechanical Turk. To ensure that participants used some form of an interactive
technology that can facilitate reflection, we indicated on MTurk that participants needed to
use tracking technology such as a smartwatch or a tracking app to partake in the survey.
Exploratory factor analysis, item reduction and several tests to evaluate the scale
validity were performed and this resulted in the Technology-Supported Reflection Inventory (TSRI),
which is a 9-item scale consisting of 3 dimensions: insight, exploration and comparison.
The insight dimension describes to what extent the interactive technology offers insight to users.
It evaluates if the system provides users with information that is insightful to them, and
whether it inspires them to look at situations from a new perspective
or makes changes to their lives. The second dimension of the TSRI is exploration.
This dimension evaluates the ease and enjoyment of exploring personal data in the system.
In order for users to reflect on their personal data, it should be easy for users to access
and review their data. This dimension can therefore be seen as a necessary condition
for reflection on personal data And finally, the third
dimension of the TSRI measures the social dimension of a reflection-supporting interactive
system, so the extent to which users can compare themselves to others. It is often argued that
current interactive technologies rarely support reflection as a social activity, but instead
focus on individual cognitive processes. Yet, reflection can be encouraged by conversations
with others and can provide reflectors with an additional or alternative perspective.
The 9 items are scored on a seven-point Likert scale and the sum of the scores
indicates the extent to which an interactive system facilitates reflection.
Higher scores indicate that the artefact offers more support for reflection.
The TSRI can be used in the early stages of designing interactive systems for reflection.
Because the scale is relatively short, it can be used for rapid feedback.
Using the scale enables the evaluation and comparison of different designs for reflection.
And we hope the scale will contribute to the field by
enabling designers to rapidly compare prototypes or assess novel artefacts.