- Good day everyone, My name is Jaybie de Guzman from the University of the Philippines Diliman
Today, I'm going to talk about our work on Unravelling Spatial Privacy risks of mobile mixed reality data
This is in collaboration with Dr Kanchana Thilakarathna of University of Sydney
And also Prof Aruna Seneviratne of University of New South Wales; also in Sydney, Australia.
Following the popularity of the Augmented Reality (AR) game Pokemon Go, mobile platform have soon launched their own
augmented reality SDKs and APIs. For example, Apple has launched ARKit, Google has launched ARCore
for the android. And here are examples of how they envisioned multiple users can interact and collaborate
virtually but sharing the same physical space. Aside from these SDKs and APIs, mobile platforms have
also improved their sensing capabilities by improving their visual sensors. And, aside from these hand-held
form factors. We also have head-worn form factors such as the Microsoft Hololens which also pushed
the developments in the holographic display technology.
Now, given these capabilities as well as the ubiquity of these devices capturing our spaces,
we are actually exposing ourselves to potential adversaries Or attackers that have access to these 3D capturing APIs/SDKs.
In an earlier work, we have actually demonstrated this risk as a location inference attack, where the location can
be inferred from the spatial data, without access to location data. Now, in this work,
We try to leverage on these same attackers, and, then, We present a potential spatial measure that can be used
as an indicator of spatial privacy risk. That is, we look at whether the inherent structural complexity
Of the space indicates its spatial privacy risk.
So, now, here are examples of these mixed reality data captures. (a) Here's the actual scene,
(b-c) these are captures of the Hololens, (d) the generalized versions of Hololens, and, then,
(e-f) here are also some of the captures from ARCore.
Now, using these 3D data, what an attacker can do is to use those historical captures as its reference set.
And, then, create model out of that. And, then, use that model to infer subsequent spaces.
This is our reference attacker. So we used two attackers: NN-matcher and pointnetVLAD. The NN-matcher tries to leverage
The local structural information, and the pointnetvlad uses neural network.
On testing, we change the size of the revealed space. And, obviously, as we increase the space, we also increase
the attacker success.
Now, let's go to the actual spatial privacy risk indicator. Here, we leverage on these self-similarity shape functions
as our potential [pre-]indicators of risk. So we play around these three potential measures: we have a [D2] distance-based measure,
[V2] normal vector similarity-based measure, and, then, a neighborhood normal vector similarity-based measure.
The first two [D2 and V2] are random-heuristic, While [xi] is more exact as it has to compute its nearest neighbors
And it's relatively slower than the previous two [D2 & V2].
We have here the distributions of the shape similarity measures for some example spaces in our dataset.
For a given space, there are portions that are less similar, Hence, lower values; and it also has portions that are
More similar, hence are higher valued.
Given these distributions, we got the mean median value Of these measures as our threshold, and, then,
Use the fraction of these low-valued similarity as our [Chi] complexity measure.
Which, then, we use as our indicator of risk. That is, A higher [chi] means a higher spatial privacy risk.
To determine the viability of these metrics as an indicator Of risk, we get their correlation with the attack success.
Here, what we can see that the normal vector similarity-based Complexity metrics [chi_v2 and chi_xi] have high correlation
With the attack success. While the distance-based metric [chi-d2] has a weak and mostly inverse correlation;
Hence, it's not a good indicator of risk. However, we see as well that for a neural-network
Based attacker pointnetvlad, it seems that its attack success has no [consistent] correlation with these complexity measures.
So, we can takeaway that deep-learning based attackers are less affected by complexity, which poses stronger risks.
But, despite these deep-learning based attackers not affected by complexity, we can say that these
Normal vector-based complexity measures, both the v2 and xi, are still good spatial inference risk indicators.
Especially if the attackers relies on these local [structural] information just like the NN-matcher.
To conclude...
Thank you!