Hi! My name is Maximiliane Windl and I am very excited to present our paper
"It is not always Discovery Time ": Four Pragmatic Approaches in Designing AI Systems.
Systems that use artificial intelligence are increasingly becoming part of our everyday lives.
Therefore, we need to understand how AI changes design processes.
AI has specific properties that make the design process complex. These include
Understanding the functionality and capabilities of AI,
prototyping & testing,
creating user value,
and collaborating with developers and data scientists.
Since designers need to control these various factors, we need to study design processes with AI
to identify how they can be improved and how we can train future interaction designers.
Current research investigating the influence of AI on the design process is limited to a relatively small
and specific sample. The focus was on designers who had extensive experience designing AI-enhanced products,
who worked at very large companies, and designed products with a vast user base. In this work,
we wanted to portray the diverse landscape of designers.
Therefore, we conducted 20 semi-structured interviews with designers who all had different experience levels
designing AI enhanced products ranging from under 1 to 10 years.
Most had a design background and 7 had different backgrounds, including computer science, law and engineering.
The participants also worked at companies of various sizes, from fewer than 100 employees to large corporations.
In total, we recorded and transcribed about 22 hours of audio data and used thematic analysis to make sense of our data.
This process led to four themes: Models and Data, Process, Team, and Translation.
Models and Data describes the challenges around data collection and preparation.
Several designers reported that they were confronted with tasks and issues around data collection for the first time.
Some designers were involved directly with in-situ data collection or creating the data
as part of the data collection process.
Process describes how designers experience the changes in their work routines when using AI.
We found that the time at which designers got involved impacted their activities.
In projects where they built on existing AI, their work focused on interaction design.
In projects where AI was to be added, designers participated in data collection and preparation activities.
In the context of Team our participants stressed that team diversity was critical for successful AI projects.
For example, designers reported being asked by the developers for input and visualizations
for the creation of new models and features and designers stressed that a fundamental understating of AI
and its abilities and limitations were very important for this collaboration to work.
Translation describes the designers' tasks and responsibilities around communication
and mediation of different perspectives. For example, designers reported having to mediate
mediate between AI experts and non-AI experts in the team, explaining AI's constraints,
and dealing with the buzz around AI. This means they had to turn unrealistic expectations
into feasible features that add user value.
Based on findings from the interviews, we contribute four pragmatic approaches to how designers work with AI:
a priori,
post-hoc,
model-centric,
and competence-centric.
The first two are more temporal-focused and leverage a clear separation between the UI and the data and model.
The last two are process-orientated and leverage a close collaboration between the designers and the technical team.
The a-priori approach was adopted by teams that did not engage with AI whatsoever and treated
the AI functionality as a priori correct.
An advantage of this approach was that the design team was independent of the AI specialists
and could use their usual, unchanged design process.
However, adopting this approach also introduced additional vulnerabilities as most models
are imperfect and change over time. Another risk is that the finalized product lacks real user value
and thus real-world applicability because the designers were involved too late.
The post-hoc approach is a mirror image of the a priori approach. This approach is primarily adopted by teams
who desire a clear separation between model and interface development.
The ultimate goal of the design team is to provide the data team with a detailed specification
of what the AI in the system should do. This approach gives the design team a high degree of independence.
Yet, it ignores possible design constraints posed by the technical implementation.
Teams that adopted the model-centric approach used the data model to drive the design process forward.
In contrast to the previous two approaches, the designers were now directly involved in technical aspects.
This approach allows developing the model and the interface in parallel.
It does, however, come at a cost which is that teams had to spend extra effort and resources
on coordination and translation. This approach is also highly dependent on the team members' skills
in sharing knowledge.
Teams with diverse expertises used the competence-centric approach.
Here, designers and the technical team decide together when to train and build models
or when to use simulated input. This resulted in a divergent design process where multiple artifacts
were developed at the same time. This approach fosters the individual skills and creativity of the team members.
However, the divergent nature of this process might make it difficult to control the temporal aspect of the process.
To contextualize our findings, we relate our four approaches to the five human interventions
in data science practice by Mueller et al. In the a priori and post-hoc approaches, discovery, capture and curation
are delegated to a different team and therefore, not in the responsibility of the designers.
However, we observed intensive translation work between curation and design in the model-centric approach.
Here, design teams used artifacts that helped them communicate what the model could do
and how it could be improved. They sometimes even developed fake AI or partially functional models.
These intermediate artifacts allowed the entire team to be part of all five interventions.
Finally, in the competence-centric approach, the teams developed numerous divergent artifacts.
This way, different members of the design teams could participate in different interventions at different times.
In contrast to previous work that found a clear separation of technology and design,
our findings suggest diverging levels of designer involvement in the data collection and model construction.
Our Four approaches can serve as lenses to analyze processes in retrospect.
and our four approaches can also serve as a guidleline to decide for a certain process to reach desired outcomes
or to react to constraints of a project.
We hope our work inspires further research on AI design.
If you have any questions, please get in touch. Thank you so much for your attention.