Hello everyone, I am Pragma Kar and today I will be presenting our paper Bifurcating cognitive
attention from visual concentration utilizing Cooperative audio visual sensing for demarcating
inattentive online meeting participants. To begin with we must firstly define the
concepts of visual attention, cognition and multitasking in context with online meetings.
In this work visual attention is referred to the visual engagement of the subject to
the device's screen with or without them being aware or conscious about the meeting context.
However, cognition is related to the awareness of the person and involves the working memory
and information processing. In Psychology we also come across terms like inattentional or change
blindness that shows how visual engagement cannot ensure awareness. Moreover in online meetings it
is not practical to assume that a participant will continuously Gaze on the screen thus making
gaze-based attention estimation unsuitable. Even if a person continuously focuses on screen it is
difficult to detect if the person is looking at the meeting or has opened a different Tab
and is reading a different article. This can be a common scenario as multitasking
is often unavoidable in online meetings and can have both positive and negative impacts.
The major contribution of this work comes broadly from theoretical and practical aspects.
First we present the theoretical analysis of cognitive attention and conduct a human study
to validate those assumptions. We explore the contribution of expressions in attention,
the impact and the types of multitasking and their involvement in online meetings. We have also
developed a system called EmotiConf that works on these findings and performs significantly well in
estimating attention levels and detecting and classifying multitasking instances.
To better understand the correlation between the attention, multitasking, and online meetings we
circulated a survey with 18 questions and received about 600 responses from all over the world.
The first point to understand was whether inattentiveness impacts the meeting's quality.
We asked the participants what they would do if they noticed that others are inattentive during
their presentation. To this 24.5% said that they will be demotivated and 15% would summarize and
wrap up the content. This will have an adverse effect on the meeting's quality. Next, we ask if
multitasking is common during an online meeting to which 83.2% replied positively. Further for young
adults of age 18 to 30 only 26.5% said that they would read relevant articles during the meeting
next we ask the participants what they would do if they were not being able to pay attention to
the meeting. 35.3% answered that they would read relevant articles on different tabs. This type
of visual multitasking can be categorized as positive. However, if the participant chooses
to read irrelevant articles, then the activity will be classified as negative multitasking. The
concept of positive and negative multitasking has also been supported by the observation that 61.1%
of the participants mentioned that multitasking is something which is necessary in online meetings.
In the second human study we aimed at understanding the correlation of facial expression
and attention. Here we asked 9 annotators to label the perceived facial expression of individual
meeting participants from five pre-recorded videos as either happiness, anger, surprise,
disgust, contempt, neutral, sadness or fear. The average inter annotator agreement on labels was
found to be 0.813. We then answered the question whether the facial expressions of participants
get changed during an online meeting. This graph shown here reveals that for 22 out of 24
participants the annotators agreed upon the fact that their expressions changed over the meeting.
Next we explore whether facial expressions of most of the attentive participants matched
at a given instance. The figure here reveals that expressions like happiness
and neutral which might look similar for some participants were present in individual frames.
This led us to the observation that a drastic change in expression was caused inattentiveness.
In view of the findings from the human study and the existing literature we developed EmotiConf,
a system that works in synchronous or real-time mode for estimating the participants' attention
in online meeting. It also detects and classifies the participants visual multitasking instances
in an asynchronous mode. In the synchronous mode the
model uses a pixel based feature to detect the speakers. For non-active participants
the model uses a quorum function to match the detected facial expression
with that of the other participants and accordingly adjusts the attention scores.
For detecting the multitasking instances EmotiConf relies on the change of ambient
light reflected from the participants face, caused by visual multitasking like opening a new tab.
Pixel-based features are analyzed by hierarchical temporal memory model for the detection of
multitasking instances. For their classification, EmotiConf considers vocal intent, vocal emotion
and head movement of the participants. Finally a rule-based mapping based on human behavior
has been employed to match the facial expression and head gesticulation of the non-speakers with a
vocal emotion and intent of speech of the speakers at that instance. This classifies the detected
instance of multitasking as either positive or negative. We have tested the performance
of the system using different experimental setups. In the pilot study we have considered
30 online meetings with 3 to 12 participants 18 of them were General discussions 5 were group
discussions with formal presentations and the rest were classroom teachings. Further we assess
the usability of the system with 96 participants through the system usability scale questionnaire.
We present the performance of the attention estimation model through Precision, recall,
and F1 score for all 30 videos as well as the subcategories of videos.
The figure here indicates that overall EmotiConf achieves an F1 score of 0.91 with Precision of
0.89 and recall of 0.94. It has a threshold of 47 and works consistently for different
types of meetings. We have employed an optimal threshold based Naïve Baseline for detecting the
multitasking instances and compared it with the hierarchical temporal memory model. These plots
show here that for all cases the performance of HTM model was better than the Naïve Baseline.
Finally we performed an Ablation Study and showed that all the features used in this paper were
required to get the best classification accuracy of the multitasking instances.
The system usability study generated an average score of 80.5. The figure to the
left here shows that the positive statements received higher agreement whereas the negative
statements received lower agreement. The figure to the right shows that the majority of the
participants gave a higher usability score to the system. Some of future Scopes and insights
are discussed next. Firstly even though it is a client-side application EmotiConf requires access
to the camera and audio. We also need to test our model by using deep learning models for multitask
classification instead of rule-based methods. Even though we have empirically shown that the overall
change in color contrast is not too significant within an individual meeting whether a particular
user can cheat the platform by using different blurred backgrounds will be explored in future.
Lastly, even though the performance of EmotiConf is significantly well, any system like EmotiConf
is rarely free from false positives and can lead to impaired assessments in some scenarios.
To conclude, we present an attention and visual multitask detection system EmotiConf that aims
at promoting the quality of online meetings and in doing so, the system relies on facial expressions,
vocal emotions, speech intent, and lip movement of the participants.
Overall we observe a satisfactory performance with a feedback of high usability.
thank you