Integrating Machine Learning Data with Symbolic Knowledge from Collaboration Practices of Curators to Improve Conversational Systems
Traditionally in computer science, the source code of a program is composed
Computers ignore the documentation, but developers read and work with both documentation and code.
This paper explores a scenario where a computer is able to read the documentation of the program,
and use it to improve its performance; and to support better the work of the developers.
The context of this study is the development of chatbots by content curators
The curators list examples of questions associated to each answer.
The examples constitute the code, and the nameIds are the documentation.
By mining and combining all the nameIds, a computer can create graphs such as the one shown in the right,
which we call an intent proto-taxonomy.
This paper shows that it is possible to mine those taxonomies from the nameIds of the chatbot,
and use them together with the examples to improve the performance of the system.
The first is about the nature of the curators' work practice of using taxonomies.
The second demonstrates that a computer can use the documentation to improve the performance of the chatbot.
The third argues that a computer can use the data from the documentation to improve the development process.
Is using intent proto-taxonomies a common practice of curators of conversational systems?
We performed a qualitative study with curators of a professional development platform, called here ChatWorks.
This study provided evidence that using taxonomic structures in nameIds is a deliberate and well-defined practice,
aimed to facilitate the collaboration among all people involved in the development of the chatbot.
We also did a quantitative study with almost 7 thousand chatbots.
Then, using this algorithm, we showed that using taxonomies is a very common practice, particularly in more developed chatbots.
We then investigated whether those taxonomies can be used by a conversational system
to improve its computational performance.
The basic idea is to use a modified machine-learning classification algorithm,
Our first tests were using data from a large chatbot in Portuguese language,
We saw decreases in Equal Error Rate with the algorithm USE+S;
and in the False Acceptance Rate with USE+T and USE+C.
Due to time constraints we will not present in the video the study regarding the 3rd research question.
We believe the studies presented in the paper support positive answers for the first and second research questions.
We could also answer positively part of the 3rd question.
We were able to improve the accuracy of the machine learning algorithms and support better some of the curators? tasks.
We see this work as a demonstration of the feasibility of the idea of having computers
We plan to explore similar ideas in traditional programs and in data notebooks like Jupyter.