Hello everyone, we are a team of researchers from UC Berkeley.
In this video, we'd like to share with you our study:
Whither Auto-ML? Understanding the role of automation in machine learning workflows.
Automated machine learning or Auto-ML, is a new field targeted at increasing
automation during machine learning. It has witnessed a rise in popularity, driving an
explosion in self-proclaimed Auto-ML tools.
researchers in HCI have been grappling with: how can Human and Automation Best integrate?
We answer this question in the specific context of machine learning, by studying
what automated machine learning is; and what roles automation currently plays in the ML workflow.
In order for Auto-ML tools to effectively meet user needs, we must first understand what those
needs are and how users incorporate Auto-ML tools in their existing machine learning workflows.
Towards this goal, we took a bottom-up approach by studying practitioners
with experience in using Auto-ML tools for real-world use cases.
Our approach allows us to understand real users’ needs and wants that are difficult to
derive without taking their perspectives into account. And it allows us to arrive at design
recommendations that are relevant and usable for users and sustainable for tool designers.
We conducted semi-structured interviews with a total of 16 participants who had
prior experience applying Auto-ML in professional capacities. For study analysis, we engaged in an
iterative and collaborative process of inductive coding to extract common themes in our data.
One of the research questions was: what is Auto-ML?
We categorized the existing Auto-ML offerings into three groups:
open-source software, hosted commercial solutions offered by cloud providers,
and enterprise solutions offered by companies dedicated to developing Auto-ML platforms.
A given participant’s ML skill-set and use cases can potentially influence how they use
and perceive Auto-ML tools. To facilitate the discussion on how contextual information serves
to explain user behavior and sentiment, we categorize the participants as follows:
novices, ML innovators and ML engineers. And we categorize their use cases into three groups:
research, production applications and prototype.
perceived by the participants. These are the existing capabilities of automated machine
learning that tool designers should preserve. Overall, ease of use, effectiveness,
and efficiency are the highest-rated qualities of Auto- ML tools by the interview participants.
Here we list specific benefits reported by the participants that corroborate the ratings.
Findings from our study also reveal perceived deficiencies of auto M L.
These deficiencies of Auto-ML tools have near-term solutions
using existing techniques. We provide many such suggestions in the discussion section at the end.
However, there are other limitations of Auto-ML tools that stem from the complex
social and psychological implications of human-machine collaboration in ML workflows.
These limitations are overcome by practitioners and point to the indispensable human functions
in a machine learning workflow, which we list on this slide.
Auto-M L tool designers must respect and preserve these roles of the human users
because to make Auto-ML tools more effective for users, the goal should not be to completely
remove the user from the process but instead to build human-compatible tools by working
to achieve trust, understanding, and a sense of agency in the user.
We found that complete automation is neither a requirement nor a desired outcome for users
of Auto-ML. Current efforts in Auto-ML that strive for an eventual fully automated system
through more tool capabilities and more accurate models fail to consider
the ways the technology is actually used in practice and what users really need.
We discuss the directions for the further development of Auto-M L tools based on our
findings such as balancing between human control and automation and system and UI improvements
that can remedy technical deficiencies of Auto-ML. For details, please refer to our paper.
Thank you for watching and let us know if you have any questions or comments.