Hello everyone my name is Dakuo Wang I'm from ibm research. Today I'll talk
about the CHI 2021 paper on AutoDS towards human-centered automation of data science.
First of all, what is automation of data science? And what is data science?
In the past few years, our research team have done various products to explore and understand how
human data science teams work in various contexts. The core of data science project is the model
building task. Simply put data scientists prepare various type of data reads them in and go through
the data pre-process, feature engineering, and the model training and the testing steps. Then finally
it can generate a combination of different model pipelines and their hyper parameters. But there
are certainly more steps than that. In this paper we are primarily focusing on these three tasks.
The use of automation techniques to automate these tasks is called automated data science.
If we straight the three small steps that we saw from the circle, to the top, it's the
data science workflow. And we can use various algorithms to automate those tasks. And together we can generate various version of the model.
The target user are data scientists. Here's what we build. The user interface actually has two simple steps: the first step asks user to upload
the data set and then after user upload data set AutoDS can automatically decide what type of
prediction task this is. Now for example in this UI, for this data set is a binary classification task.
If user satisfied with this configuration they click on run experiment. After a few minutes or
a couple hours then AutoDS can automatically generate model results. Here to the top
of this resulting page, you can see a visualization and to the bottom is the pipeline leaderboard view.
They are showing the same information to the top as we can see there are p1 p2 p3 p4 to p8
eight different model results being generated and they use two different algorithms:
xgb and gradient boost classifier. To the bottom we can see those different models being presented
in a leaderboard and using ROCAUC score to rank them. Then we can also see there's a particular
useful function called save model. If a user clicks on save model they can see automatically generated
notebook. After we build the system we designed a between subject user study to evaluate the AutoDS
systems performance. In this task we gave the same dataset to 30 different data scientists,
half of them use AutoDS, the other half of them use python in notebook
and then we give them the same 30 minutes to build a model. Some very high-level findings here:
First AutoDS can generate 16 models in 3 minutes versus data scientists can only generate
the first model using 15 minutes on average. Also this automation system doesn't make any
errors, but 8 out of 15 data scientists in the programming group made some human error.
Our interview result also confirmed the findings. Some unexpected findings
it's about the trust or the confidence in the final models. In the AutoDS condition,
the participants on average rated the final model as 2.4 out of 5 when we asked them how confident
they are to deploy that actual model. In comparison uh the participants in the notable condition
felt the model are much more trustworthy. So this is suggesting that the model
with a higher performance does not transform into a higher confidence score.
In summary, we build an AutoDS solution and the tested it in between subject user evaluation. We
found the performance and productivity indeed increase with the use of automation solution
for the model building task. We suggest in the future the human in the loop automated
data science solutions should focus more on building up users trust.
So that we can really fulfill this future of human AI collaboration. With that, thanks for listening