hello my name is Allard Oelen and I'm a PhD student from Hannover Germany today I will present
crowdsourcing scholarly discourse annotations let's get started scholarly communication is
largely document based for example when reading articles we are often faced with pdf files
while document-based communication generally works fine for humans it's created for human consumption
after all machines cannot easily parse the contents or in other words the articles are not
machine actionable and one of the consequences is that finding relevant articles becomes more
cumbersome as more articles are getting published in this example more than a million results are
displayed in google scholar when looking for user interfaces based on machine learning
when knowledge graphs are used it is possible to create paper descriptions that can be understood
by machines this makes it possible for machines to understand the semantics or the meanings of
the article and for example provide ability to have better support for finding articles
in this example a paper is displayed in gray you see the metadata of the paper and in red
some of the contents in this case an annotated sentence with a respective discourse class
having structured paper data has many benefits however creating this data
is not that straightforward broadly speaking there are two approaches
firstly is to automatically generate these descriptions for example
using natural language processing techniques but this is not very accurate currently and
often requires domain-specific models and they are not always available secondly there is the manual
creation of structured data for example using crowdsourcing compared to automated techniques
the accuracy will be higher but since human labor is involved scalability is an
issue also it requires skilled users to make decisions regarding the modeling of the data
in this research we suggest a hybrid approach to generate these structured paper descriptions
using a paper annotation system this uses both machine intelligence and human intelligence
and it tackles the weaknesses of the techniques when they are employed in isolation
it even provides a synergy in the form of an intelligent user interface
to create such a hybrid system we will answer the following research questions how to design
an intelligent user interface to populate a scholarly knowledge graph using crowdsourcing
and a second question how to employ a machine in the loop approach to assist users in this process
let's now discuss the system design this is an overview of the system design
the interface is displayed in the middle and the use cases
for generating structured data on the left and for using the structured data on the right
the workflow is as follows a user or a researcher uploads a paper in pdf format to the system
afterwards the user annotates key sentences within the paper
each sentence is annotated with a discourse class describing what the sentence is about based on the
annotations a knowledge graph is created let's now have a look at the individual components
first let's look at the data entry use cases and specifically the paper submission use case
this describes a scenario where researchers can annotate their papers during the paper
submission process this could for example be during the camera ready submission
in this case the author is of course familiar with the paper's content and can therefore make good
decisions on which sentences are most important and therefore which sentences should be annotated
now let's have a look at the user interface in the interface you'll see the article is
displayed on the right side and the annotations with their discourse classes on the left now we'll
discuss the three machine assistance components starting with the automatic sentence highlighting
the first smart component automatically highlights potentially important sentences in the paper for
that the paper is first automatically summarized using an extra extractive summarization tool
this created summary is split by sentences and each of those sentences
are highlighted in the pdf file here you can see the highlighted sentences within the paper
and here you can see a button to disable the highlights in case they are not helpful
highlights only serve as suggestions and it's therefore not mandatory to use them
next the maximum sentences per annotation this component counts the amount of selected
sentences for a single annotation a warning is displayed if more than two sentences are selected
this encourages users to only select key sentences and not to annotate whole paragraphs also
this warning is only a suggestion it's still possible to annotate more than two sentences
next the automatic class suggestions once a sentence is selected a set of potentially relevant
classes are displayed these classes are determined using a zero shot classifier and are based on the
contents of the selected sentence also here in case the suggestions are not helpful according
to the user they can be ignored now let's have a look at the human role in this process
a user of course has to select the sentences within the article and then choose the relevant
discourse class they can choose from 25 classes which are coming from the discourse elements
ontology or deo finally there are multiple use cases related to how to use the generated
knowledge this includes an improved search system or automatic generation of paper abstracts
now we will look at the demonstration of the interface
this is the annotation interface the first thing we do is to upload a paper in pdf format
then we can start annotating a sentence by making a selection
we can either select a type from the list of all classes
or we can choose one of the suggested types in this case the class problem statement seems
to be right after adding the annotation it is displayed in the left sidebar the
completion bar indicates if there are at least two annotations for the five most important classes
we can enable the smart sentence detection which will highlight potentially interesting sentences
throughout the article a highlighted sentence can
be annotated by clicking on it and then selecting an appropriate class
in case more than two sentences are selected for a single annotation a warning will be displayed
this warning informs the user that it's better to annotate a maximum of two
sentences finally the result can be saved now I will discuss the evaluation of this research
the main goal of the evaluation was to determine the task feasibility
and see whether researchers are willing to annotate their papers with the system
for example during paper submission in total 23 researchers were part of the
evaluation they were asked to entertain the paper they authored themselves
or to annotate a recently read article afterwards they were asked to fill out a questionnaire
the questionnaire consisted of three parts first to assess the attitudes towards the task itself
and the smart system components secondly a system usability evaluation was conducted
to evaluate usability and finally the task load was evaluated using the nasa task load index
now the results the system usability scale results show a score of
76 which is considered good the nasa task load index was 35 which is low compared to
a meta-analysis related to this index and in this case the lower the better
finally here are some results of the participants attitudes towards the interface for example most
participants are willing to annotate a paper during the camera ready submission interestingly
they were mostly positive about the integration of smart technologies in the interface
even though they indicated that both the smart type suggestions and the sentence highlighting
tools were not always providing helpful suggestions they still appreciated the integration
finally the conclusion we conclude that machine assisted paper annotation is a feasible task
researchers are willing to annotate their papers also they appreciate the smart elements the other
evaluation results show the usability of the system is good and the workload is acceptable
the limitations are mainly related to the evaluation for this more participants should
be included from more diverse backgrounds this is also something we are addressing in future work
additionally we will use other nlp tools possibly supported by humans
to further process the annotated sentences this was the presentation
thank you for your attention and please let me know if there are any questions