Hi, my name is Soya park. Today i'm going to talk about our work done at IBM Research we
my wonderful collaborators April Wang, Ban Kawas, Vera Liao, David Piorkowski and Marina Danilevsky.
when building an ml model in industry there are many two groups
domain experts and data scientists involved in a data science team so domain experts
are the people who are knowledgeable of the domains. for example they're medical
doctors or lawyers and they have this desire to integrate machine learning into their workflow
and but they do not know machine learning so they are asking help to data scientists or
machine learning engineers however they are new to the domain but they are the people who are
able to build this desired model because they are the experts in machine learning
however due to this different background previous research found that building a machine learning
model in such a multidisciplinary team involves lots of frustration and requires lengthy time
to understand what is current practice and challenges in gaining domain knowledge between
domain experts and data scientists we interviewed data scientists and asked them how they've been
building model with domain expert's help as a result that we found that data scientists actually
struggled to understand the domain and it took several months to get familiar with the domain
and they want structured feedback and want to learn
right phrase to ask questions to domain experts to gain information they want one difficulty of
building models in complicated domains is that you don't know what part of a domain is relevant
so you have to you have no choice but have to learn everything one of our interviews
remarked that when they build a model in regard to legal domain they end up becoming a semi lawyer
and last thing is domain experts are actually busy so data scientists cannot getting unlimited
lists for resources from them and they have very limited time to get their domain experts help
we are envisioning a tool to help the situation with our tool domain experts can provide enough
domain knowledge efficiently and data scientists can use to generate data in a way that helpful
for them to understand the domain and build the model so we present a new system Ziva with zero
knowledge how do i develop a machinery model zip delivers two different outputs to data scientists
domain knowledge extraction and attention elicitation method in domain knowledge extraction
Ziva automatically selects representative inputs because domain experts are busy we want to make
best use of their time and don't want to ask so many questions then a domain expert can
extract the topic from each input and build the information architecture of the domain
let me give you example in restaurant domain many people are familiar with restaurants so as
a preliminary study we took a restaurant data sets to volunteers in this case they are domain experts
of the restaurant domain we told them let's say you're teaching people who do not have knowledge
in restaurant can you highlight the importance concept of the domain and teach them as a result
even with five instances they're able to come up with fairly rich domain concept and taxonomy
so for example in restaurant there's food and food could be tasty warm or cheap whereas service
could be incomplete overwhelmed slow and staff could be friendly or not friendly to facilitate
domain concept extraction we came up with the card story interface where each instance of a
domain data set is a card domain experts can group cards together that share a topic if each instance
has more than one topic in it users can make duplicates of a card to assign multiple topics to
instance secondly data scientist wants to receive structured rational from domain experts
that they can use it for modeling so when they don't have domain knowledge
a chunk of data is hard for them to understand domain experts can help
this by highlighting and letting focus on the development parts of the inputs
it allows to focus on the specific aspects of a complex input we call this attention elicitation
to design the interface that actually extract domain knowledge we conducted paper prototyping
study and let users use the prototype and see how they interact with our prototypes
so we invite volunteers and make them annotate on splashes to various prompts had they been using it
we can see if they successfully accomplish the task or not as we iterate the design
we ask these questions to ourselves first can domain experts understand understand the prompt
second we are getting consistent answers from these different domain experts for example we
are giving 10 inputs and the first one is that's right the red velvet cake oh this stuff is so good
and as you can see participant 1 and participant 2 they are returning pretty much similar answers
after we got the result from domain experts we bring this
and show this to data scientists with their help we enhance the prompt
and do the pilot study again we repeat this process a few times until we have reliable designs
we came up with five different registration methods and each method is informed and inspired
by interviews and literature review i will present the interface of each elicitation method
first one is backup words in this method we asked domain experts to pick keywords that indicated of
a label of instance so for example took an hour to get our food only four tables in restaurant my
pasta was lukewarm our server was running around like he was totally overwhelmed the cure would be
took an hour look warm and overwhelmed and the next one is label perturbation in this case we
are asking domain experts to change the input in a way that is no longer the previous label so one
domain expert can change it to took very little time to get our food many tables in restaurant my
pasta was hot our server was running around like he was totally having a blast next one is also
sentence sentence simplification we also ask them to change the input so it could be simply
simplified to took an hour to get our food my pasta was lukewarm our server was overwhelmed
and the next two method we are actually using the domain concept
that is extracted by the domain experts in the previous stage of the interface
so concept back of words is we're asking to take a cure of the extracted concept in this case we
are asking the keywords of services slow and for this network so the keyword would be tuken hour
and lukewarm next one is concept annotation and in this case we are asking to get keywords of each
block of the concepts so we are asking the keyword for service slow food and not hard
there are the summary of our five potential education methods
we designed this method to help data science teams to share domain knowledge to evaluate our system
we first take our interface to domain experts and if they like our interface so after conducting the
task using our interface participants were asked to fill out the nasa tlx form as a result backup
where users think they are very successful at doing the tasks then the users with other
interface at the same time backup voice users are more stressed than most of the other conditions
next we take attention station data from our domain expert study to data scientists and
ask them to rank and share use cases of our education elicitation data we get a number
of inspiring usage and how they will use the illustration data in the data science workflow
for example one of them is to build a rule-based model using elicitation data
using domain taxonomy from zipa data scientists can build a concept dictionary after this how
can you tell if each element is better good to build a sentiment dictionary for each element we
can refer to attention station hence positive food is about tasty worm and cheap and so on
combining the concept dictionary and the attributes we can write semantic rules
to label data sets so we actually build the rule-based model according to this
idea as a result our potential station models for reliable and robust across multiple classes
stand backwards you can find the details of implementation and results in our paper
so this was about our new system to facilitate domain knowledge learning in multi-disciplinary
team ziva ziva facilitates the learning process by guiding domain experts to write
useful information for data scientists thanks for listening to my talk and i'd be happy to
take questions. Thank you