Hello, and welcome to this case study overview on Understanding the AI Feature User Journey
in Enterprise.
My name is Sarah Hanses, I am user researcher at Microsoft and I developed this work with
my co-author, Jennifer Wang.
From sales to retail to financial services, AI is impacting businesses in fundamental
ways.
User experiences in the enterprise space may include configuring or setting up an AI model,
understanding how an AI model performed, or consuming AI insights to make business decisions.
According to a McKinsey & Company global survey on the state of AI in 2020, 50% of respondents
reported having adopted AI in at least one business function.
Use of AI in the workplace is a trend that is expected to continue.
These survey results also demonstrated that while businesses are increasingly focused
on leveraging the power of AI, only a minority of these companies have recognized the risks
associated with AI use.
While AI technologies have impacted both consumer and enterprise applications, less research
has focused on the unique considerations for AI enterprise experiences.
To enable responsible design, it is important to understand these systems as a whole and
within a broader human computer interaction framework that includes the user, the system,
the tasks and context.
There is a growing consensus that designing for AI is complex, and that a framework for
understanding human-AI interactions can enable more effective envisioning and refinement
of AI use.
This research sought to understand how enterprise users manage and interact with intelligent
features within a situated ecosystem and across their entire lifecycle.
The intended outcome was to create a framework for understanding human-AI interactions, inclusive
of risks, so that these risks may be mitigated when designing AI moving forward.
In this study, 57 participants took part in interviews that were conducted between August
2020 and January 2021.
For this study, business profiles were aggregated into four personas representing our primary
users across various verticals, including IT Pros or Admins, Makers, Analysts and End
Users.
In semi-structured interviews, participants described their end-to-end process and overall
experience working with AI features, and responded to high-level questions, with sample topics
including perceptions of AI, accountability in using AI, and how AI is used to inform
business decisions.
After synthesizing findings across all personas, the following phases of the user journey were
identified: Onboard, Operationalize, Launch, Use, Maintain, and Evaluate.
While these phases are presented as linear, much of this journey is iterative, while some
phases and subphases are experienced in parallel.
Using this map, we created a framework for identifying commonalities and differences
between enterprise personas.
In the Onboard phase, enterprise users identify a need or an opportunity to implement AI and
decide to use a particular AI feature.
In this phase, users draw on their existing perceptions and understanding of AI, as they
evaluate potential technical solutions and select one.
In the Operationalize phase, users plan and gather system requirements, collect and prepare
data that will be necessary for the AI feature, create and build AI models in various tools,
and then test and tune those models until a desired level of performance is achieved.
In the Launch phase, the AI feature is deployed and integrated into business workflows, as
users begin to adopt and acquire the necessary skills and knowledge to use the AI feature.
In the Use phase, users configure an AI feature by defining variables and data to be used
based on business questions.
They work to understand an AI s output and validate it before using it to inform a specific
business decision.
In the Maintain phase, the AI feature is monitored to ensure that it continues to work as expected.
It may be updated based on newly available data or evolving business requirements, and
any technical issues will be inspected and resolved.
Lastly, in the Evaluate phase, enterprise users measure the effectiveness of an AI feature
to see if it enabled the anticipated efficiencies.
They may also reflect on how it impacts their work.
For more detail on how each persona s experience varies across the different phases, please
refer to the full case study paper.
The primary asset resulting from this research was a high resolution Figma file that outlines
the AI feature user journey in enterprise.
This map overviews the process that an enterprise user engages in and the steps they may take
in pursuit of a specific goal when using AI in the workplace.
It synthesizes user actions, experiences, and feelings, in a narrative that is visually
represented.
In our user journey map, the top is focused on the user actions by personas throughout
all phases and subphases.
The bottom is focused on synthesized pain points, opportunities, and quotes to contextualize
these experiences.
Red bars indicate the severity levels of every pain point and we also included a key to help
guide viewers through this map.
This research uncovered multiple opportunities related to using AI features in the workplace.
The primary pain points and opportunities expressed by users include the following:
1) data quality and quantity, 2) security and privacy, 3) lack of adoption and AI comprehension,
and 4) maintenance and evaluation.
Regarding data quality and quantity, there is an opportunity to create consistent data
quality standards in the industry and enable easier connection to data sources, in addition
to enabling easier formatting and transformation of data needed to train an AI model.
Regarding security and privacy, there is an opportunity to enable users to monitor what
an AI feature is doing or has done.
Regarding lack of adoption, and lack of understanding, opportunities include leveraging in-context
guidance and documentation to make AI features and their capabilities comprehensible and
transparent.
It also encompasses addressing real-world concerns and limitations related to potential
impacts of using AI.
Regarding maintenance and evaluation, there is an opportunity to better define accountability
for AI feature management.
We recommend that enterprise users adopt clear sets of ownership and accountability around
AI features, including but not limited to who owns training of the model, validation
of its outputs, feedback on those outputs, and ensuring that data sets are representative.
To improve evaluation, we recommend enabling more systematic processes to submit and collect
feedback within the AI system itself.
The results of this research were shared out through a series of presentations to internal
product teams that develop AI features.
These research findings have been applied by product development teams in their planning
processes, as teams have begun using this framework and these learnings when proposing,
planning and prioritizing improvements to AI features and capabilities.
In summary, the use of AI features is cyclical in nature, and they require ongoing oversight
and collaboration across many business roles.
There are multiple opportunities to improve the experience of using AI features in the
workplace, and it is critical that these opportunities be addressed, by both individuals and teams
using AI features, and by those building them, to enable appropriate use of AI in business
applications.
This is the first version of this user journey.
Future considerations to expand upon this work include:
Exploring the experiences of enterprise users who do not interact with AI, to understand
their perspectives and potential barriers to entry, as well as understanding the experiences
of additional enterprise personas Future research may also inspect the differences
across AI features that are out-of-box vs. custom built, as well as segmenting this journey
further based on the specific type of AI technology used.
It will also be important to utilize quantitative methods in order to better assess pain point
severity.
In closing, we expect that adoption, awareness and knowledge of AI features in enterprise
will continue to evolve over time, making this a dynamic journey that will need to be
continually updated.
Thank you.