Hi, welcome and thank you all for joining my virtual talk today.
I’m Eugene, the lead author of this project focused on the topic of privacy with smart
speakers. In particular, we studied features tied to
privacy in data management, and how they can affect user trust toward, and the usability
of, smart speaker products. So without further ado, let me share what
this study is all about. Before digging deeper into what our research
questions were, some might ask why? Why the topic of privacy and trust with smart speakers?
Well these days, it wouldn’t be a surprise if all of you own at least one smart speaker
at home, not to mention that virtual assistants are embedded in our mobile phones we carry
24-7. And, as the popularity of smart speakers grew,
the biggest concern surfaced above was the idea of, for instance, Alexa listening to
our private conversations all the time, which naturally led to the question of “Can I
trust Alexa with my data?” As a response, Amazon started to implement
some features to alleviate those concerns. And, the most straightforward way to deal
with that is to delete all the voice recordings, period.
However, for this to happen, users have to be really proactive and motivated to log into
their Alexa app and go through the cumbersome process of adjusting their settings, and then
deleting all the voice recordings, manually. That’s why, Amazon added a couple of more
intuitive and automatic measures, including the option to enable deletion by voice, and
setting up automatic voice deletion periodically, from every 3 months to 1 and a half years.
The question here would be, would these measures actually work?
At the end, regardless of how many times users delete, Alexa needs to continue recording
voice to work properly anyways. So, we started with a very basic research
question to answer if adjusting privacy settings, in this context, meaning getting a chance
to delete audio recordings, actually made users trust Alexa more, and the content delivered
by her as well, more. To answer that question, we developed an Alexa
app that allowed users to delete audio history. I think it’ll be easier for me to show you
how the app worked through a video clip.
But, before I play the video, I want to clarify that users interacted with an Echo device
not a mobile phone. Just for the purpose of demonstration, I actually opened up my own iPhone to
show you the flow of the app. So let’s play the video.
[Alexa] Would you like to delete your most recent voice recordings? Remember, deleting
your voice recordings may degrade your experience using voice features. Would you still like
to delete your most recent voice recordings? Please answer delete or save.
[User] Save. [Alexa] Okay, your most recent voice recordings
are saved. Would you like me to automatically delete voice recordings on a monthly basis
from now on? Please answer yes or no. [User] Yes.
[Alexa] Okay your voice recordings will be automatically deleted once a month.
Alright, so this was the Alexa privacy feature we developed for the purpose of this study.
Pretty neat, huh. Again, users talked to an Echo device not a
mobile phone, and they were also free to choose whether they wanted to save or delete the
voice recordings. Then, how did we test if the use of this type
of privacy feature worked? We added another group of users who did not
adjust privacy, and compared the self-reported trust level between two groups.
I know, some might have questions. Then what did the control group do?
Regardless of if users used the privacy feature or not, we needed to build some type of interaction
between users and Alexa, so that it makes sense for any of them to even consider adjusting
privacy settings, right?
So, we created another Alexa skill, designed to offer health answers when users asked 3
specific health-related questions as presented on the right-hand side, which were of topics
of general interests, but with statements that were controversial in nature.
That way, if participants trusted Alexa more after adjusting privacy, even with controversial
content, it means the privacy features are actually pretty powerful!
That wasn’t the end of our study, though. We had another question.
What if users adjust, or in professional terms, customize something else than privacy?
In fact, many studies point to the positive effects of content customization on user empowerment.
If that’s the case, maybe if users customize content, they may feel empowered and the trust
toward Alexa wouldn’t be compromised, even when privacy issues are not directly taken
care of. So, we added a content customization step
to the Alexa skill designed to offer health information that I just shared, which focused
on customizing three things: first, speech speed, second, content length, and third,
information source. I will show you how it goes, but again remember,
participants interacted with an Echo device not a mobile phone, so they did not have any
screen nor text information. [User] Alexa, open health answers.
[Alexa] Welcome to Health Answers! Before we begin, would you like me to adjust the
speed of my speech? Please choose one option from slow, fast, or regular.
[User] Regular. [Alexa] Okay, were request was saved. Would
you like to adjust the length of my health answers? Please choose one option from normal
or abbreviated. [User] Normal.
[Alexa] Would you also choose your source of preferred primary source of health information?
Please choose one source from WebMD, Mayo Clinic, or GenieMD.
[User] WebMD [Alexa] Okay, all set! What would you like
to ask me today? You can see users were allowed to adjust content
speed, length, and source, which they were free to choose whichever options they preferred,
and afterwards they asked questions to, and received answers from, Alexa.
And likewise, we compared the effects with a control group which did not customize content
at all. Now that we have two customization options,
one privacy settings and the other content, we had to test if they interacted with each
other. Hence, we conducted a 2 X 2 experiment with
participants randomly assigned to every possible combination of four conditions.
However, in reality, regardless of their customization choices, all users received identical health
information from Alexa. This was for the purpose of experimental control, to make sure the
mere act of customization, and not the content difference, had effects on user perceptions.
It doesn’t stop there. Considering that the effects might not be identical to all
users, we added two particular media dispositions related to privacy perceptions and information
control behaviors, which were the individuals’ level of privacy concerns, and power usage
over general technology, a.k.a tech-savviness, to see if the customization effects varied
by individual differences. Wow, there were quite some questions, but
don’t worry, this is the last. Just to give you a quick re-cap of our research
questions. First, we examined if the privacy deletion
feature made users trust Alexa more. Second, we also explored if content customization,
not related to privacy, can affect trust. Third, we tested if the two customization
effects interacted with each other, And at last, we examined if individual differences
altered the above effects. Now, are you ready to check out the results?
Here, I’m about to show you the results of our analysis from 90 participant responses,
ready? Tada! I know, it’s a lot.
So just fyi, the actual study has more variables, research questions, and findings involved,
so please check out our original paper for more information.
But for the sake of time, I want to focus on two major significant effects relevant
to the two dependent variables related to trust, which are trust toward Alexa and content
credibility, a) b) up top, just for points I made throughout the previous slides.
First, privacy customization did affect trust toward Alexa, but the effects were different
based on the level of power usage. Second, privacy and content customization worked
together to affect content credibility, but the effects varied by individuals’ level
of privacy concerns. I’ll explain further.
I would like to start with the graph for the first effect I talked about, that is, about
the effects of privacy customization and power usage on trust.
To make it easier, let’s start with the left-hand side of the x-axis, which represents
non-power users. You can see the privacy customization condition
with the red line is higher on trust compared to the control condition with the blue line,
which means adjusting privacy setting actually made non-power users trust Alexa more.
However, that wasn’t the case for power users.
If you shift to the right-hand side, it’s pretty much a reverse mirror image.
For individuals, especially tech-savvy individuals, trust went down as they adjusted privacy settings,
which seems to reflect some sort of negative privacy priming effects.
The effects of customization on content credibility is a little bit more complicated.
Let’s start with the left-hand side graph, when users did not customize content at all.
The pattern is similar to what you just saw from the previous slide, but the effects vary
by the level of privacy concerns. That is, privacy customization worked for
non-privacy-sensitive individuals, but held back those who reported high privacy concerns.
Here is a silver lining, though. When you move to the right-hand side graph, do you
see the minimized negative effects among privacy concerned individuals? When they customized
content including speech speed, length, and source, the negative effect of privacy customization
dissipated. That is something.
So what did we learn from the findings? There are a couple of interesting ones.
First, despite its intentions, offering customization options for privacy settings exerted negative
effects, especially, by lowering trust toward Alexa among tech-savvy power users, and decreasing
content credibility among privacy-conscious individuals.
This is somewhat counter-intuitive compared to the positive customization effects documented
by many studies, which suggests that we should be careful in implementing privacy features
in smart speakers. On the other hand, content customization showed
potential to alleviate some negative privacy priming effects on content credibility.
Inherently, smart speakers do live by audio data, so the tension between privacy and convenience
will be something hard to dissolve, but at least the findings from this study give some
insights on how to approach to help users keep that balance by letting them actively
set the tone of their content. Then, how do the findings inform us practically?
First, from the pricey cost of privacy customization, we learned that we should try to integrate
the customization processes into the smart speaker experience as much as seamless, intuitive,
and frictionless and possible
Second, we learned content customization is still powerful in subtle ways, so we should
consider trying to marry compatible customization or personalization techniques together to
bolster content credibility Third, personalities really matter, so we
should consider developing adaptive systems to cater to individual factors and preferences.
And, that was what I wanted to share today. I really appreciate all of you viewing this
video for the project I put a lot of heart into, and I also wanted to share some special
thanks, which first and for most, includes you all attending my virtual talk, and of
course my co-authors, and don’t forget the CHI reviewers for their
constructive feedback, and at last, the Madonna of this study, and now my significant work
from home companion, Alexa. Just to add, I am also in the job market to
become a user researcher, and I wish I could’ve met you all in Hawaii, but I don’t mind
a connecting through LinkedIn or Email, so if you want to have nerdy talks with me about
research, please reach out. Thank you all, and stay safe.