Hi everyone. In this paper, we introduce the CrowdAct system,
which allows crowdsourcing applications for smartphone sensor systems
to achieve highly accurate training datasets in activity recognition
by addressing three main remaining challenges.
First, the collected labels lack in accuracy
as workers are typically not experts in the field
and may not be specifically trained for labeling.
Second, workers often lack motivation and sustained engagement.
Although they exhibit early enthusiasm,
workers may lose interest and drop out over time
or might not be highly invested in producing high-quality labels.
And finally, the lack of sustained engagement and motivation
as a reason for cheating behavior in crowdsourcing scenarios,
which affects the task output
So, this situation leads to low-quality data collection and biased data.
We assume this as inaccurate data and propose a method to detect these situations.
Based on the remaining challenges of crowdsourced data labeling,
we set three research goals to achieve them.
First, we employ an active learning algorithm to address the lack of accurate labels.
Second, we apply the gamification concept to overcome the lack of motivation and sustained engagement.
And finally, we propose an inaccuracy detection algorithm
using supervised machine learning
to minimize inaccurate data in crowdsourcing applications.
Here we summarize a high-level overview of the proposed system called “CrowdAct.”
where we propose two novel independent ideas directed to the same goal on the right side
in order to get high-quality training datasets for activity recognition.
First, we employ active learning to improve recognition accuracy and user engagement
We first ask the worker to perform the labeling task
using our provided Android smartphone application
And then, we build an active learning model, a special case of supervised machine learning.
to train the data from crowded labeled data
we then build the system to evaluate and provided a confidence score for each label
that is "Gamification Points"
Then we provide those points to users through a smartphone notification
So here workers can easily understand how well they completed the labeling task by looking at the score
and then they can optimize data for their data labeling
and on the right side,
we then expect that we can get an accurate training dataset that is ready for use
for our activity recognition system
And for the second method
we also ask the worker to perform the labeling task using the same Android smartphone application
and here, we build an inaccuracy detection model
to train the data from crowd labeled data
and then we build the system to detect inaccurate labels from crowd labeled data
in this state, if our system detects inaccurate labels from crowd labeled data
we'll provide these results immediately to users through a smartphone notification
so the worker can get a notification that the system has detected inaccurate data from their data labeling tasks
and then they can avoid making inaccurate data for data labeling
so that's why on the right side, we expect that we then get
an accurate training dataset same as the previous method
To evaluate the differences between the three conditions,
we designed a large-scale activity data gathering system.
including, task creation, task mobile application, and task manager
we collected variables that may impact the data labeling performance
including age, gender, digital literacy, smartphone usage, smartphone gaming, and working hours
We collected over 6,549 activity labels from 19 activity classes.
And overall, participants in the two proposed conditions submitted more labels
and one more class on average compared to the baseline condition.
We also applied user-dependent training for evaluations
to show accuracy improvements for each participant in each condition
without considering side effects such as different sensor positions.
We also applied user-independent training to show overall accuracy improvements
in each condition that can be applied to new users.
And also, we assessed the validity of the crowdsourced data
to determine whether the accuracy levels are sufficiently high for application to real-world data or not.
we then measured the user engagement by using two popular approaches;
first, the user engagement scale short form (USE),
and second, we recorded the user events, including button press on the smartphone application
and Clickthrough rate (CTR) of push notifications
We further explored the relationships between the number of button presses,
CTR, and activity recognition accuracy.
We showed the correlation between variables
by using Regression analysis and the network plot.
We also showed an average percentage of inaccuracies in the collected data
In summary, we proved that all our research questions were supported.
And while this research enabled us to effectively collect crowdsourced activity data,
there are limitations to be addressed for future research.
So if you are interested in this research, please read this paper in detail.
And also, if you require any further information, please feel free to contact me by email.