Hello everyone, this is Haotian Li from HKUST
I am glad to share our paper "A Visual Analytics Approach to Facilitate the Proctoring of Online Exams"
Online exams have become widely used in recent years, especially during the pandemic of COVID-19
However, proctoring online exams is challenging
First, according to a previous study, online exams are easy to cheat and about 29% of the students have cheated in online exams
Furthermore, lack of face-to-face interactions leads to the low efficiency of online proctoring
and makes it harder to catch cheating than conventional exams
Here are some comments collected in our interviews with teachers in a university
They all agreed that it is challenging to proctor online exams
Thus, how to provide teachers with convenient, efficient and reliable proctoring for online exams is worth our exploration
Existing online proctoring methods can be roughly classified into three categories
First, it is very common to employ human proctors to monitor the exam remotely
However, this method is labor-intensive and time-consuming
To improve efficiency, some fully automated methods have been proposed to apply machine learning techniques to detect cheating behaviors
However, these methods suffer from the concern on the "black box" nature of machine learning models
To address the concern, semi-automated proctoring methods have also been explored to involve humans in making decisions
A common drawback of them is that they do not have a convenient method to assist teachers in exploring original data like videos
In our paper, we propose a novel visual analytics approach to facilitate online proctoring by leveraging both machine and human intelligence
We first apply machine learning techniques to help identify abnormal behaviors
Then these suspected cases are presented to teachers by visualizations for their verification
Before we design our approach, we have conducted in-depth interviews with teachers and conclude several design requirements
The first three requirements ask us to present suspected cheating behavior in different levels of details,
including student level, question level and suspected-case level
They believed that this can help them locate cheating behaviors fast
Furthermore, teachers want to achieve convenient comparison across students and questions to get rid of the effect of the question design or students' habits
Finally, exploration of raw data is also thought to be important for reliable decisions
To fulfill these requirements, we first design the suspected case detection module
In our approach, we collect both students' webcam video data and mouse movement data
Then we extract abnormal behaviors of head and mouse movements from collected data
Abnormal head movements identified by machine learning models mainly reflect that the student does not look at the screen
Abnormal mouse movements can show that the student uses unauthorized materials on his computer
After detecting suspected cases, we present them to teachers with different levels of details
Student List View provides a quick overview of students at a high risk of cheating during the whole online exam
Question List View facilitates the selection of high-risk periods of each student
Behavior View presents the students’ detailed behaviors of head and mouse movements
Playback View makes teachers able to conduct a final confirmation on students’ videos and mouse movements
This is a screenshot of our entire user interface
To evaluate our method, we first explored several usage scenarios, including how we can find and verify a cheating case conveniently
and how cheating cases can be found through in-detailed observation on visualized students' behaviors
Also, we conducted a user study with 16 participants who have been student teaching assistants
The user study shows our approach has better accuracy and less time cost than the current proctoring method used in the university
Participants also rated our system highly on the usability and visual design
Furthermore, we presented our approach to several university teachers and they confirmed that our approach is useful and intuitive
Here are some final take-home messages
First, as an initial exploration to apply visual analytics to online proctoring,
our approach utilizes the power of both machine learning and human efforts
to achieve good effectiveness, efficiency and reliability
Second, the usage of mouse movement is proved to be quite helpful in proctoring online exams
Finally, since real-time online teaching is more popular nowadays,
more visual analytics methods can be explored to help teachers and students while preserving their privacy.
That's the end of my presentation
For more details, please visit our project page
Thanks for listening