Hello everyone, this is Shengzhong Liu from the University of Illinois at Urbana-Champaign.
I will present our paper titled
"GlobalFusion: A Global Attentional Deep Learning Framework for Multi-Sensor Information Fusion."
Our problem is motivated by the proliferation of various smart devices.
For example, the smartphones, smart shoes, smart glasses, and smart watches.
Several devices may co-exist in a typical body network.
And each of them is equipped with multiple sensors like the
accelerometer, microphone, and the light sensor.
They can be collaboratively utilized to capture the user context, perform environmental measurements,
and recognize body movements.
umm, meanwhile, in many sensing applications deep learning algorithms have shown non-trivial accuracy improvements
compared to conventional feature engineering based machine learning methods.
The feature extraction and the classification are automatically conducted by the neural network.
How to perform sensor fusion within the neural networks becomes a new challenge.
The key question in this paper is:
how to effectively and efficiently perform multi-sensor information fusion within the neural networks.
Our answer is an end-to-end learning framework called GlobalFusion.
Well, let's look at the current literature.
Information fusion has been done in traditional neural networks,
but the wiring of the global context is fixed as a function of the data of local feature maps.
And attention mechanisms can allow for flexible data-dependent fusion.
Different weighting of the same local feature can be presented in different global contexts.
And in previous works,
mean query has been used for sensor fusion.
They address the sensor information correlation, but they don't address the information heterogeneity.
Our insight here is that
aggregated information from higher layers of the neural network
is better at recalibrate and filter the local sensor features,
so that the information heterogeneity is also captured.
Specifically we target at the multi-device human activity recognition application.
Multiple inertial modules can be deployed simultaneously on several body positions,
while each IMU module consists of an accelerometer, a gyroscope, and a magnetometer.
For each (sensor, position) pair, the data is in the format of time frequency spectrogram.
This is the end-to-end overview of the GlobalFusion framework for multi-point multi-sensor classification.
It's generally divided into four stages:
individual sensor feature extraction,
spatial information fusion,
sensing modality information fusion,
and the classification module.
The first three modules are all convolutional modules, while,
the classification is by an RNN-based classification network.
The convolutional modules do not address the heterogeneity among the sources,
but we ultimately want to see the features from informative sensors and positions to be amplified,
while the unrelated noise is being suppressed.
So we cut the direct connection between the original feature fusion out and the input to the next layer,
and add one global attention module between them to enhance the fusion output with complementary local features.
The concrete design of the global attention module is shown in this page.
The preliminary global feature extraction module refers to a three layer convolutional network.
We extract the global query from the merged global features,
and extract the local keys and values from local feature maps
The attention weight of each sensor type or position is evaluated as the similarity between the global query and the local keys.
By doing so, the class specific information in global features becomes the standard to evaluate to the informativeness of the local features.
Next, the weighted sum of the local features is denoted as the merged local features,
which is then merged back to the global query through a residual connection.
Next is the evaluation.
We evaluate the performance of sensor fusion models on four public human activity recognition datasets.
And five attention-based sensor fusion baselines are compared.
Their design details are skipped here.
We first show the quantitative model performance by their accuracy and F1 score.
We can see that the GlobalFusion improves the performance by a clear margin, compared to the backbone DeepSense model,
while the two versions of the GlobalFusion show that the spatial attention module is more effective than the modality attention module.
The confusion matrix indicates that the error mainly comes from the ambiguity between sitting and standing,
which are two static gestures.
We also evaluate the model efficiency of each model on Nexus 5 with TensorFlow Lite.
It turns out that the GlobalFusion has higher time and energy efficiencies than SADeepSense.
And only negligible overhead is caused by the global attention modules,
compared to the backbone DeepSense model.
So let's briefly conclude the paper.
We have proposed a global attention mechanism for multi-sensor information fusion,
and integrated into an end-to-end learning framework, named GlobalFusion.
And we also demonstrated the effectiveness and efficiency of GlobalFusion on four public HAR datasets.
Thank you for your time, and please feel free to contact me via emails if you have any questions or feedbacks.
Thank you!