Hello my name is Jacob Abbott and today I will be presenting work that I did with
Dr. Sameer Patil from Indiana University regarding
“How Mandatory Second Factor Affects the Authentication User Experience.”
Two factor authentication is becoming a commonly used method to improve security for organizations
so access is not limited to being protected by a single password.
Many times two factor authentication is implemented in a way similar to this following example.
Users are asked for their username and password as with most standard websites
Once those are verified a second prompt is presented
requesting how someone would like to authenticate with a second factor.
For this example. we’ll choose to have a push notification sent to our phone.
Finally we can choose to approve the authentication attempt
and continue on to the system we were accessing.
When Indiana University announced they would be implementing two factor authentication
in a phased rollout over three periods we thought it was an opportune time to investigate
how two factor authentication might affect the user experience.
Looking at the university’s planned phases the process was broken up as follows:
First two factor authentication would be mandatory only for select highly sensitive systems
for staff, faculty, and student employees,
such as systems accessing health records, financial records, or student information
The second phase would be mandating two factor authentication for all systems for employees.
Finally the third phase would require two factor authentication for all systems
for employees and students, so everyone associated with the university.
This phased approach allowed us to gather information over an extended period of time
to explore the following research questions.
What are the differences in the user experience
in different modes of two factor authentication implementation?
What are user preferences for 2FA use?
And what implementation options for two factor authentication could balance
security benefits and user experience challenges?
We collected three types of data regarding the use of two factor authentication
and attitudes surrounding it.
To do this we performed a survey during each stage of the two factor authentication roll-out.
Additionally we worked with the university’s IT office to gather chat logs with people
discussing two factor authentication with the university’s online helpdesk.
And finally we gained access to authentication log data from the IT office to investigate
how the system was used in practice beyond the self reported behavior of the survey respondents.
Next I’ll proceed in presenting findings pulled from each of these data before synthesizing
our recommendations and implications as a whole.
In the survey we asked questions focused on participant’s practices and preferences
of using 2FA, what security measures they used on their own mobile devices,
the task load required when using two factor authentication which was based from NASA’s task load index,
some technical expertise questions to gauge their familiarity with computer security,
and finally, demographics to better understand their relationship with the university.
Responses were broken up into three groups. 44 responses were from the Single Factor group
for those who were not required to use two factor authentication,
74 in Select Systems for those who used two factor authentication
only for the highly sensitive systems,
and 447 in the All Systems group
for those who had to use two factor authentication
for accessing any university system.
The all systems group had the response rate
as the university’s policies changed over time and made use of two factor
authentication for all systems mandatory for everyone.
For the remainder of the talk the Select Systems group will be referred to as 2FA-Select
and the all systems group will be called 2FA-All.
So in our findings we looked at the survey responses,
we found no significant differences between the Single Factor and
2FA-Select groups’ responses
for the task load of authenticating.
However, compared to the Single Factor group those in the 2FA-All group
reported statistically significantly higher levels of frustration when authenticating,
with a p equals 0.037 value.
They also reported statistically significantly lower levels of overall satisfaction with
the authentication user experience with a p value of 0.016.
While these were statistically significant, the small effect size suggests that differences
in single interactions may not be drastic, but could create a larger impact over time.
In regards to what method people most often used as their second factor,
the majority of respondents reported using push notifications as their main choice
for use as a second factor,
followed by phone calls, physical tokens, SMS messages,
and lastly one time passwords generated by an app.
Looking at early responses to some open ended questions we asked, we found that responses
were often expressing negative views regarding the use of 2FA.
To investigate a bit further
we added some questions regarding what systems
individuals might find useful to have protected by two factor authentication.
We asked for a range of different systems from those requiring data protection by law
down to classifieds or even the suggestion of no two factor authentication for any system.
We found that the majority of respondents were happy to have 2FA protect highly sensitive
systems such as those protecting financial, health, and student records.
While over 1 in 4 said they would rather have two factor authentication for all systems.
Meanwhile only 8% said they did not want two factor authentication at all for any system.
Sixteen hundred chat Logs were qualitatively coded by myself and two other students
and we achieved a high level of intercoder agreement with a Cohen’s Kappa of 0.901.
Themes were pulled into two categories, emotions and topics of conversation.
We found that the most observed emotional responses in the chat logs was confusion
at 73% with people being confused
as to varying aspects of two factor authentication.
Meanwhile we only saw frustration in 4% of the conversations.
We found that 81% of conversations were specifically in regards
to questions about setting up two factor authentication for an account.
28% asked about adding a device in the setup process or adding a new device afterwards.
16% asked questions about what to do and how they could authenticate when their device
was missing, such as if they had left their phone at home or had bought a new phone
and the old one was no longer in their possession.
15% of discussions asked about the use and
availability of physical tokens for access.
While 6% asked about the steps required to
authenticate with remote access through a VPN.
Authentication logs were cleared of identifying data before analysis and spanned from
December 2016 to April 2019 as shown in this figure. You can see the number of daily authentications
started below 50,000 authentications per day and hit a peak of just under 450,000 authentications in a day.
The orange arrow pointing to February of 2017 indicates the start of the second phase
that required all employees to use two factor authentication for all systems, resulting in a large growth
in the number of daily authentication requests.
The beginning of two factor authentication being required for everyone for all systems
is highlighted by the green arrow pointing to November of 2017.
This period had a sharp growth before a drop in activity over the winter break.
There is a sharp rise in February of 2018 for a few days
but that is an anomaly due to external system issues.
In the remaining time period it can be observed that activity
drops during the summer months and the time frame around Christmas and New Years,
while the highest spikes of activity coincide
with the beginning of each semester in August and January.
Please note that the graph shows the total number of authentication attempts each day
with color coding purple for successful authentications and orange for failed authentication attempts.
There additionally is a small gap in March of 2017
where a coding error did not collect logs for a period of two weeks.
When we take a look at the rate of successes and failures over time, as shown in the current graph,
we can see that the average rate of failed authentication attempts is roughly 1 in 10.
This is a fairly high percentage and so we looked into what reasons might be given for these
unsuccessful attempts.
In the authentication logs whenever a failed attempt was recorded, the system also logged
an automated reason for the failure. Here we show the top five reasons that were applied
to the failed authentication attempts logged.
Of all observed failures 61% were marked as "User Cancelled"
and still flagged as a failed authentication attempt.
Therefore the majority of failures may have been due to users choosing not to authenticate
rather than failed attempts due to errors.
The second highest reported reason for failure was “No Response"
which was paired with push notifications and phone calls, implying that the user likely
did not interact with their phone before the time frame ended.
11% of failures were attributed
to invalid passcodes being entered from physical tokens,
SMS messages, and one time passwords generated from applications.
2% of failures were attributed to no keys pressed which was
only applied to phone calls, it may be that these failures were attributed to the system
counting voicemail triggering as a response to the phone call received but no keys pressed to continue on.
Finally at 1%,
accounts being Locked Out for too many failed attempts made up the fifth largest reason
for failed authentication attempts.
The most important goal behind two factor authentication implementation is
protecting systems with sensitive information.
Limiting two factor authentication to only select highly sensitive systems that handle sensitive information
should offer the protections organizations seek without creating an undue burden on users.
Responses of our participants suggest that limiting the extra security requirements of
two factor authentication to critical systems
and letting users opt-in to use two factor authentication for other systems might
provide elevated security and reduce user burden at the same time.
In fact, our findings show that users would likely welcome the added protection
despite the small increase in authentication burden
Take this quote from a participant as an example.
Limiting two factor authenticatio requirements to highly sensitive
systems that both organizations and general users deem worth the additional effort may
be a potential compromise between additional security and lowered user experience.
Quantifying the organizational value of 2FA is difficult as it would require precise quantifications
of security benefits that are often difficult or even potentialy impossible to determine.
Future work could further incorporate economic analyses that analyze the collective productivity
losses and emotional impact of mandatory two factor authentication
in relation to the realized organizational security gains.
Thank you very much for your attention.
Further details are available in the paper and supplementary materials.