Hi everyone!
This video is to present our paper modeling the trade-off of privacy preservation and
activity recognition on low-resolution images.
I am Zirui Cheng, an undergraduate student from Tsinghua University.
This work is a collaborative effort between Tsinghua University and University of Washington.
Nowadays, low-cost cameras are increasingly ubiquitous.
With the help of artificial intelligence, these cameras are capable of automatic recognition,
providing smart services to users.
However, a key concern with the ubiquitous cemaras in smart environments is visual privacy
exposure.
We expect a vision-based system that can provide intelligent applications while preserving
visual privacy.
To achieve this purpose, a fundamental solution towards the construction of a privacy-preserving
vision-based system is to lower the image sensor’s resolution from the hardware level.
However, there is a trade-off regarding the effect of the image resolution on the main
recognition task and visual privacy preservation.
We believe that modeling such a trade-off will provide guidance for the future design
of privacy-preserving systems with low-resolution sensors.
In this paper, we focus on a smart home scenario where low-resolution image sensors automatically
recognize activities of daily living.
Based on prior works, we regard such a trade-off problem as mathematically optimizing the objective
function S(r) shown here, to reveal the trade-off between privacy preservation and machine recognition
with both machine and human factors taken into consideration.
Therefore, we proposed a framework to implement the modeling method.
We conducted a user study to obtain humans’ perceived importance of various privacy features.
To evaluate our model in realistic environments, we contructed an ADLs dataset with visual
privacy features.
After this, we also conducted a user study and several experiments to model the recognition
abilities of both humans and machines under different resolutions.
In the end, we described the calculating procedure of our objective function to model the trade-off
between privacy preservation and activity recognition.
In the user study to quantify the importance of privacy features, we recruited 125 participants
and received 115 valid responses in total.
Participants were required to rate the importance of each privacy feature with or without the
low-resolution conditions using a 100-point slider presented in a web-based user interface.
Based on these results, nudity, identifiable face, valuable property, and relationship
are chosen as the most important privacy features.
To construct the ADLs dataset, we selected qualified videos from the publicly-available
PA-HMDB51 dataset, which were then divided into five basic ADL types including functional
mobility, feeding, intimacy, entertainment, and personal hygiene.
After identifying what visual privacy features users value the most in the previous study,
we provided both frame-level and clip-level labels of the privacy features.
Some examples of the annotated frames in the video dataset are shown here.
After this, we developed a web-based user interface to model the effect of image resolution
on human’s performance in recognizing ADLs and visual privacy features.
We utilized a randomization strategy on the back-end server so that each participant could
view 4 different videos from our dataset.
They were required to answer a set of questions under seven different resolutions.
Our results indicate that low-resolution images are effective in preserving visual privacy
but the effects are highly dependent on privacy features.
Also, lowering the image resolution has a significant negative impact on human recognition
performance on ADLs.
Then, we adopted cutting-edge deep learning methods to explore the effect of image resolution
on machine’s recognition performance.
We compared the performance of ResNet50, Efficient Net, and Vision Transformer on ADLs recognition.
We found that these models can achieve high recognition accuracy even on low-resolution
images like 20 times 20.
For the task of privacy features recognition, we tried four types of deep learning methods
to examine their capabilities of facial identification, nudity recognition, property and object detection,
and relationship classification, respectively.
So far, we have modeled both human's and machine's recognition performance under different resolutions.
Here we provided additional analysis to justify whether super-resolution techniques may influence
our results through a user study.
We adopted one of the cutting-edge image super-resolution methods SwinIR to upscale the videos in our
dataset.
Statistical analysis prove that participants did perform better on super-resolution videos
than on raw videos.
However, the improvement in recognition performance brought about by super-resolution technology
is still less than that brought about by increasing the resolution itself.
Finally, based on the results we have discussed above, we can calculate the objective function
S(r) shown here for each resolution we have sampled.
As is demonstrated in this figure, the value of the objective function S(r) shows a trend
of first increasing and then decreasing with the increase of resolution.
For the case where lambda is 1, the objective function takes its maximum value at a resolution
between 20×20 and 30×30, which indicates a proper resolution for balancing privacy
preservation and activity recognition.
However, there do exist several limitations of our work and we describe them here.
First, prior works have proved the effectiveness of leveraging multimodal data in activity
recognition.
With multimodal information, we can alleviate existing algorithms’ depends on images,
thus allowing for a lower resolution of image sensors.
Second, some researchers have proved that low resolution alone may not provide enough
privacy preservation.
We expect future research can explore the effect of more privacy-preserving methods
on this trade-off problem.
Third, we think that the perception of privacy may still be affected by many other factors.
The estimation of the perceived importance of privacy features obtained through our user
studies is possibly not applicable to populations in different cultural contexts across the
world.
Therefore, we expect there will be more independent works to explore the influence of other factors
on humans’ perception of privacy.
To conclude, the contributions of our paper are two-fold.
On the one hand, using the at-home ADLs recognition as a scenario, we proposed a pipeline to investigate
the effect of the image resolution on both human's and machine's performance on the main
activity recognition task and visual privacy awareness.
On the other hand, we presented a model for calculating the trade-off of visual privacy
preserving activity recognition using low-resolution images.
Based on the proposed model, we can calculate an optimal resolution range for privacy-preserving
activity recognition applications.
Please read our paper for more details.
Thank you!