Thank you.
Hi, I'm [INAUDIBLE] from Northeastern University
in Boston.
My presentation is on the potential applications
of video games to crowdsource raw sensory data annotation.
So our research group is primarily
interested in designing activity recognition algorithms.
And one of the big challenges we face in this process
is gathering high quality accelerometer data annotations.
These annotations require human input.
And so we're interested in using video games to crowdsource
this data annotation task.
So in this paper, we are presenting
two human computation games to crowdsource raw data
annotation.
We demonstrate that casual game players
may be able to label movement patterns
in the raw sensory data.
We showed that puzzle-based games
may be more suitable for this kind of labeling task
than a more rhythm or action-based games.
And finally, we discussed some opportunities
for algorithms, players, and experts
to come together and collaborate on labeling
large-scale accelerometer data sets for health surveillance
studies.
Accelerometer-based activity recognition
is basically detecting activities
using machine learning algorithms.
Most of the time, they use body worn miniature accelerometers
like the one on my wrist.
And they process raw accelerometer data
from these devices to infer activities.
So these devices can be worn on wrist, ankle, hip, or tie
on any other part of the body.
And they measure acceleration, which
is the rate of change of velocity
along x, y, and z-axis.
And that's why different movements generate
different visual patterns of acceleration along these three
axes.
For example, if a user is running,
it generates a more rhythmic pattern along these three axes.
If a user is sitting and typing, it generates some [INAUDIBLE]
pattern of acceleration.
Most of the research today follows a very similar protocol
where domain experts or researchers
gather raw accelerometer data from a set of participants.
These participants view this one or more sensors on their body
and perform several activities.
Researchers then label this data either retrospectively
or in real-time when the participants are
doing these activities.
And finally, they use the label data
set to train their activity-recognition algorithms
using a variety of machine learning approaches.
But we know that most of these approaches
require large data sets and they depend on very high quality
of annotations.
But labeling accelerometer data, especially the raw data,
is not trivial.
Many small research groups don't have sufficient human resources
to label these data sets.
And honestly, labeling large accelerometer data
sets is very time-consuming and a very boring task.
So we are interested in a scenario where
in the perfect world, domain experts can
take their large data sets, send it
to casual game players, who just by playing the game
will label the data.
And then these label data set can
be used to train activity recognition algorithms.
So in this paper, we designed two human computation games
to crowdsource this accelerometer data annotation
task.
The first game we designed is called
Mobots, which is a rhythm-based labeling game inspired
by Guitar Hero, which most of us are familiar with.
So here, players label moving accelerometer data
fragments one at a time.
So these data fragments contain one or more
different activities that the participants have done,
and players have to identify and label them.
The second game we designed is called
Signaligner, which is a puzzle-based game
to label accelerometer data.
The game is inspired by a genome sequencing game called Phylo,
where players label or match accelerometer data patterns
by cutting and aligning data fragments.
So unlike Mobots where they have to identify
an activity in the data and then label it,
Signaligner is just a pattern-matching task.
So they have to check if a given fragment visually
matches with a template that they have been provided.
The data that we used in the pilot game prototypes
was collected from some participants
who wore accelerometer on their non-dominant list.
And our research group then labeled
this data set into several categories,
including walking and running activities,
sedentary activities, and sleep patterns.
We then divided this data set--
this labeled data set-- into two parts.
The first part that was used in the game
is called validation data, which is used as the ground
truth in the games to provide in-game feedback to the players
and to provide training during the tutorial levels
when they're teaching them how to label these activities.
The second part of this labeled data set
is called unknown data, which is used to get the annotations on.
So this is the data that researchers
should be actually using to get annotations on their own data
sets.
But games do not know the ground truth on this data set.
But because our research team already has all the labels,
we used this to evaluate players' performance
after they had played the game.
So we evaluated our game prototypes in a pilot study
with 100 players for each game using Amazon Mechanical Turk.
So the games were play tested stayed
on first the interplayer agreement.
That means how much did the players agree
on a label for a given sample of data.
Labeling accuracy-- how correct were
players labels compared to the ground truth data
we already had.
And finally, for exploratory purposes,
we also compared players' labels with a state-of-the-art
activity recognition algorithm that we had developed in a lab.
So we can take a look at the results in details
for each of the games.
So for Mobots we have, let's say,
a 4 minute sample of raw accelerometer data.
And then we have the labels that our research group
had assigned to this data set.
So you can see the first part of the data is walking activities.
Second part is sitting activities.
And then we have labels from casual game players
who played Mobots.
These are the players who have no background in machine
learning or exercise science.
So Mobots players had a labeling accuracy of 89%.
But what's interesting is that they
did a good job of identifying all the walking and running
activities in the data fragments.
We think one of the big reasons for this
could be that walking and running activities have a very
distinct rhythmic patterns.
So this pattern is easily distinguishable,
even in a game like Mobots when the signals are
moving at a very high speed.
If you look at the results from the second game
we design, which is a Signaligner,
let's say we have the 20 minute sample of the raw data
that players played on, followed by the ground
truth labels from our research team.
And once again, from a set of casual game players who played
this particular data set.
Signaligner players actually had much higher
labeling accuracy of 99% and a higher interplayer agreement.
Signaligner players actually agreed more often
than the Mobots players when labeling data.
We think there are two reasons for this.
First, Signaligner is just a pattern-matching task.
They don't really have to identify an activity
and learn how to spot it.
They just had to check if it matches visually
with the template that they have been provided.
So it's a much easier task to do.
Second, in Signaligner, players could spend their own time
to explore the signal, find out the right transition points,
and cut out the fragments, even leave whenever they want to.
So that's why we think perhaps a puzzle-based game approach may
be more appropriate for this kind of a complex and rather
boring data annotation task.
And we also compared players' labels with an activity
recognition algorithm.
In the interest of time, I'm not going
into the details of what the algorithm does,
but we have provided that detail in the paper,
along with many other interesting results.
So let's say we have two minute sample of raw data
from Mobots players.
These are the labels from the players with their interplayer
agreement for the same data.
And these are the labels from the algorithm
for a 30 second time window with the algorithm's confidence.
So we see some really interesting trends here.
First, we can notice that there are instances where
the algorithm is clearly wrong.
It thinks the person is sleeping but actually
it's not asleep data.
But players got it correct.
These are the instances where players
with no background in machine learning
can provide feedback to an algorithm
just by playing the game.
But there are also instances where the algorithm is correct
but players are wrong.
And here, algorithm can be used to provide feedback
to the players when they're labeling unknown data.
Between these two edge cases, there
are opportunities for algorithms, players, and domain
experts to collaborate and label large scale accelerometer data
sets.
So for instance, the way we envision
this is the domain experts can take the raw data,
pass it through the algorithms that they have,
and get the first set of labels.
These labels can then be sent to casual game players who,
by playing the game, would refine the annotations further.
The refined annotations can either
be used to retrain the algorithms,
or they can be sent back to the experts
so that experts can annotate over.
This way we would not need a lot of players
to actually play the game at the same time.
We could save the human computation time
or the amount of time players spend labeling this data.
So in summary, we presented two human computation games
to label raw accelerometer data.
We showed that casual game players may
be able to label movement patterns in raw data,
but in this study we have barely scratched the surface.
And clearly, more research is needed.
We showed that puzzle-based games may be
more appropriate for this task.
And we discussed some opportunities
for algorithms, players, and experts
to collaborate together and label large scale accelerometer
data sets that can be used in a lot of public health studies.
The work was funded by National Institute of Health.
And we are sincerely thankful to all the players
on Amazon Mechanical Turk who spent their time playing
the game and contributed valuable data for our pilot
study.
Thank you.
[APPLAUSE]
I was fascinated.
What are your questions?
Yes.
And then in the back.
Hi.
Thank you for the talk.
Sarah Wiseman, Goldsmiths, University of London.
I had a question.
Do you ever ask any of the Mechanical Turk players
whether they considered this a game?
That's a good question.
Yeah.
It looks a lot like a task.
So I'd just like to know.
Yeah.
No, that's one of the challenges we also had.
So when they play the game, they have an option
to provide feedback.
So some of them did provide--
so some of them did find it like a game.
And the reasons they stated they wanted to play
was because they thought it is contributing to science.
But we don't have a lot of data to back that up.
So only a few players actually providing that feedback.
Hi.
[INAUDIBLE], Leuven.
Yeah.
Actually, I think my question is resonating
the same underlying thought that, of course,
you want to trigger intrinsic motivation,
or to a certain extent you want to ensure
that these participants actually enjoy doing this
and would want to keep on doing this.
But I understand that you currently don't
have a measure for that or--
So our primary purpose was to check if we can even
gamify this task.
Yeah.
Sure.
Of course.
So if we can get the quality labels
that we can use in a research.
Future work.
Absolutely.
Thank you for the interesting talk.
Katja Hoffman, Microsoft Research, England.
My question is in the same direction as the previous two
questions.
I was curious if you provided any additional incentives that
would gamify this.
So for example, was there some kind of score
or some match or something else that the player
could obtain when they did the task correctly?
So in Mobots-- if I can just quickly go back to the slide.
So in case of Mobots, the player's goal
was to completely fill up this power bar.
And once they fill up the power bar,
they move to the next level.
And the goal is to reach as many-- or clear
as many levels as possible.
In case of Signaligner, they had like five tutorial levels
with the ground truth, and then they
have one challenge level with unknown and ground truth
data mixed in.
And their goal is to completely solve the puzzle at that level.
And so that's the game element in both the games.
That was like some very efficient question asking.
We have time for more questions.
That's so great.
Yeah.
Over there.
Wait for the microphone, please.
Active, sedentary, or are you actually trying
to obtain other labels?
Can you repeat through the microphone please?
Are you just looking for active or sedentary,
or are you trying to obtain other labeling?
So we're interested in-- ideally,
we're interested in very specific labels,
like, for example, brisk walking,
which is a more specific version of walking.
So we are interested in labels like walking, running,
sedentary, sleep, including sensor non-wear, which
is basically a device kept on the table
and still collecting data.
So there are a wide range of labels that we
are ideally interested in.
The pilot study was done only with these high level
categories.
I'm sorry.
Thank you for your question.
I'm just insisting on the microphone
because I've been sitting in the back during the first session,
and I could not hear it.
You might think you project but you only project to the front.