Hi, I am Haojian.
I am going to present our project Lean Privacy Review: Collecting Users’ Privacy Concerns
of Data Practices at a Low Cost.
This project is collaborated with Hong, Mayank, Swarun and Jason.
The key idea behind Lean Privacy Review is to have a low-cost approach like heuristic
evaluation to identify users’ privacy concerns.
So the input here is not user interfaces, but data practices.
And the output is users’ privacy concerns.
Imagine you are a data scientist at Target.
You find that you can predict whether a user is pregnant using their purchase history?
How would you use that data?
In 2012, Target sent a diaper coupon to a teenager because they believed she was pregnant.
This diaper coupon helped the father found that his daughter was actually pregnant.
This has become one of the most famous privacy stories and caused many user complaints.
This is not only about target.
Today, practitioners are facing a variety of similar dilemmas.
They have a hard time to understand how to collect and use data in a manner that users
feel comfortable.
Large companies like Google rely on formal privacy reviews in navigating through these
dilemmas.
This process requires experts in different domains, such as privacy engineering, policy,
product management, software engineering.
As you can imagine, many teams would be involved in the process.
It is a quite expensive and slow process.
We cannot apply this to every privacy-related design decision.
Instead, practitioners often go with something simpler called the front-page test.
Imagine that your decisions were reported on the front page of the local newspaper?
Your parents and friends will see you in the report.
Would you still make that decision?
It is an ad-hoc process with little actual feedback from users.
The actual user feedback are often only available after the deployment.
If the intuition is wrong and the design is bad, the company has already wasted lots of
labors and caused some negative user experiences.
Towards this problem, we introduce Lean Privacy Review, a low-cost method to collect direct
feedback from users through the proxy of crowd workers in the early stages of design.
Lean Privacy Review can be a stand-alone method.
But we also developed a system to make it easier to use that method.
Here we show an example demonstration to illustrate the workflow.
First, the practitioner needs to create a privacy storyboard to communicate their data
practice.
The basic units in a privacy storyboard are four types of data actions: data collection,
data sharing, data processing, and data usage.
Users can create the text description for each data action and assemble these data actions
in a tree topology.
That’s the input to Lean Privacy Review.
It usually takes 10-20 minutes to generate a storyboard.
Once finished, the practitioner can generate a Qualtrics survey file using the website.
Importing that file to Qualtrics will generate a survey automatically.
The survey contains a set of template questions for each data action.
Each participant first needs to evaluate the data action quantitatively and then elaborate
on the reasons in free text.
We then leverage another group of crowd workers to annotate the privacy concern in each free-text
response.
Here is an example task.
We ask crowd workers to check all the privacy concerns mentioned in the text.
When the annotation tasks are finished, the web interface will aggregate the data and
compute the range and magnitude of privacy concerns.
This plot is a summary view.
Each row corresponds to one type of privacy concerns.
For example, the first row is invasive monitoring, the second row is violation of privacy expectations.
each column corresponds to one evaluator.
The redness in each block indicates the level of uncomfortableness.
When a practitioner hovers her mouse on a colored block, a tooltip containing the raw
textual feedback will pop up.
Through these visualizations, practitioners can have both quantitative and qualitative
view of potential privacy concerns.
Lean Privacy Review is fast, cheap, easy to access, and can identify practitioners’
blind spots.
First, the inspection process costs around $80 and requires 5.5 hours wait time.
So a product manager may have some idea in an afternoon, create a storyboard in like
15 minutes, get the results overnight, and present them in the standup meeting the second
day.
In contrast, it often takes weeks to receive feedback from the formal privacy review.
Second, Crowd workers are easy-to-access through many platforms.
Lean Privacy Review can be a powerful tool for small teams and independent developers.
Imagine a small startup working in the garage; they would not be able to afford privacy experts.
By making the process more accessible, practitioners can evaluate their design directly without
going to dedicated privacy experts.
It reduces the communication overhead, and avoids unnecessary miscommunication.
Third, although individual crowd workers may have limited privacy expertise and only find
a few privacy problems, these concerns are often complementary to each other.
Our experiments show these privacy concerns saturate once we have 14 crowd worker participants.
And aggregating their feedbacks can identify 139% additional privacy concerns that 6 data
practitioners are not aware of.
What makes me most exciting is the opportunity to iterate and explore the design space through
fast privacy feedback.
For example, from left to right, these are the output from Lean Privacy Review for three
different designs.
We can easily see the one in the middle is the best.
The privacy storyboard is modular.
So practitioners can slightly modify the storyboard to test different variations, do it repeatedly,
and find the ideal design that balances their interests with users’ privacy concerns.
For more details, please check our paper.
In the paper, we explain more information about how we address the three key challenges
to design lean privacy review: First, how to communicate the data practice
design to crowd workers, Second, how to scaffold the privacy inspection
for crowd workers, Third, how to present the output from crowd
workers to practitioners.