Hi, I am John Joon Young Chung.
I’m excited to present our work on Creativity Support Tools.
We conducted a large scale literature review on
how technologies, roles, interactions, and users
intersect to form a design space
of art-making creativity support tools.
This work was done in collaboration with Shiqing He,
and my advisor, Eytan Adar.
Creativity support tools, or CSTs,
have become an integral part of artistic creation.
They are powered by computing technologies
and these technologies have expanded which roles CSTs can serve.
For instance, machine learning-powered generative algorithms
now can generate content without specific user input,
like playing drums along the user
who improvise on other instruments.
Advances in technology has also changed
how these tools interact with the user.
With our drum example, users no longer play drums by themselves.
Hence, their interaction with the drum track would change.
Technological advances in CSTs are also changing
the range of users who can create.
With our drum tool we can open the doors
to those without drumming expertise
So, we can see that there are many aspects of CSTs.
While some of these have been investigated by previous work,
our focus is how all these factors intersect
to form the design space of art-making CSTs.
We believed that investigating art-making CSTs with this perspective
would allow us to identify existing design patterns
and under-explored areas of art-making CSTs.
With this motivation in mind,
we analyzed technologies, roles, interactions, and users
of art-making CSTs,
by studying 111 publications on art-making CSTs.
Here, we defined art-making CSTs as
those that support the creation of artifacts
with aesthetic qualities.
We focused on research systems as
they include tools that are novel but not yet commercialized.
Moreover, research systems accompany papers
with detailed descriptions on aspects of our interest.
In our analysis we also limit the scope of tools
to those that support a single user.
With our criteria, we sampled papers from those
that are cited by previous literature review work,
while sampling newer papers with the sampling approach
used by the previous work.
We also searched for work that is cited or citing the work
that we already found, or those from recent HCI conferences.
We coded 111 sampled papers
with a combination of inductive and deductive approaches.
For a detailed description of the sampling and coding approach,
please refer to the paper.
With the coding, we identified codes for
taxonomies of role, interactions, technologies, and users.
Due to time, in this presentation, we focus on few codes.
For more details, please take a look at our paper.
For CST roles, we identified a taxonomy of
resource roles and process roles.
Resource roles signal which type of benefits the tool offers,
whether they are about ideas or skills.
In contrast, process roles categorize
which part of the creation process the CST supports.
Here, we explain a bit more details about process roles.
One process role is execution assistance.
These tools usually augment user’s actions to implement the artifact.
For example, in the case of digital brush,
as it augments the user’s action to create brush strokes,
it would serve the role of execution assistance.
Another process role is producing,
which generates artifacts on behalf of the user.
The aforementioned drum playing CST would fall in this category.
The last process role we want to mention is critique,
which gives feedback to the user so that they can improve the artifact.
In interaction approaches, we found three dimensions.
They are directness, predictability,
and whether output is implementing or influencing.
In the interest of time, let me focus on two specific dimensions.
First is directness of the input.
Direct input is close to the artifact,
like being the brush strokes on the canvas,
or a part of an artifact,
like vocal recordings in dubbed animations.
Indirect input, on the other hand,
is distant from the final artifact.
Examples of indirect input include
natural language queries or parameters given to the tool.
Another dimension of interaction approach is
whether the tool is supporting by implementing or influencing.
One example of an Implementing tool includes
aforementioned tools that generate content.
Influencing tools, on the other hand,
give information that can change the user’s behaviors,
like feedback, scaffolds, critique, or analysis.
For technologies, we identified a broad spectrum
but I’ll focus on one type:
learning algorithms.
Learning algorithms are usually machine-learning algorithms
that are trained on data.
They can be used to recognize patterns in artifacts
and recent ones can generate partial or whole content.
For users, we identified taxonomies according to expertise
and whether they are specific populations
that require special accommodations,
like for children or those with disabilities.
With our identified taxonomies, we found many interesting patterns.
For some cases, patterns are inherent and natural.
For example, CSTs coded as “producing tools”
tend to implement artifacts, as implied by the code.
For some cases, patterns happen because
the community focus on a certain types of CSTs.
For instance, many CSTs for users with disabilities
focused on building tools that supports with
skills rather than ideations,
while such ideation tools can be designed.
For other cases, patterns are due to technical viability,
and they are expected to change with technological advances.
For example, critique tools tend to receive direct inputs,
like artifacts themselves,
to give feedback on how they can be improved.
Indirect inputs, like commands and parameters,
on the other hand, were relatively less complex,
and have not been subject to critiques.
However, advanced machine learning generative algorithms
can receive rich and complex natural language descriptions
to generate content,
which is indirect input,
and they can also be a subject to critique.
To summarize, we could identify design patterns of art-making CSTs
and how they can be expanded by looking into under-explored,
and not yet technically viable patterns.
With our analysis, we also could identify those CSTs
that were specifically powered by generative AI algorithms.
We call these AI-CSTs.
While autonomous generation is technically doable to some,
we identified that these CSTs are
still designed to provide a high degree of control to users.
Hence, in process roles,
more generative AI-CSTs supported “execution assistance”
rather than “producing.”
At the same time,
we could find that when AI-CSTs support “producing” functions
they somewhat mimic human-human interactions.
For example, a tool that serves the role of the actor in a comedy skit
or interactive narrative
assumes that these tools are used to replace human actors.
Similarly, the drum accompanying tool also
mimics interaction with human drum players,
interacting through non-verbal cues.
These tools are also common in that
they mainly interact through generation,
like by taking an action as a character or an actor,
or by continuously playing drum along with the users.
We would be able to expect that these types of more autonomous AI-CSTs
would be more feasible with more technological advances.
There are other interesting patterns between
technologies, roles, interactions, and users
of art-making CSTs in our paper.
For example, we found critique tools tend to receive direct inputs,
while being unpredictable and influencing.
For more details, we again refer you to our paper.
In this work, we explored the intersection of
roles, interactions, technologies, and users of art-making CSTs
to learn the design space and under-explored area of research.
Thank you for listening!