Hello everyone! I am Kazunori Komatani from Osaka University, Japan.
The title of this talk is knowledge graph completion-based question selection for acquiring
domain knowledge through dialogues.
This is a joint work with Yuma Fujioka, Keisuke Nakashima, Katsuhiko Hayashi, and Mikio Nakano.
Knowledge is crucial for various systems.
For example, restaurant database is useful for restaurant search and movie database is
useful for movie recommendation.
Knowledge graphs are widely used as such knowledge, and it consists of a massive set of triplets.
An example triplet is shown here.
A triplet consists of subject, relation, and object, and it is represented as a directed graph.
The two nodes are called entities and the link between them is called relation.
This example means tom yum goong tastes spicy.
The knowledge graph is always imperfect, because it is very difficult to exhaustively describe
possible relations between all entities.
So, it needs to be constructed by human annotators.
We believe that dialogue interface is promising for such a task.
As you know, smart speakers like Amazon Echo are used in many places.
It would be helpful if we can perform such manual annotation during talking with such interfaces.
Let me show an example dialogue here.
A human user and a system are chatting about unusual food, and the user said I had Momiji-don last week.
This Momiji-don is an unfamiliar word for the system and the system does not know its
cuisine type and ingredients.
Here if the system can predict its cuisine type may be Japanese, the system can make
a question like, "I've never heard of it, but it sounds like Japanese food, right?"
Then, if the human user affirms it, this triplet is confirmed as correct and acquired as new knowledge.
Furthermore, if its ingredient is predicted as rice, the system can continue the dialogue,
for example, "I guess it is a dish made with rice".
We believe that a dialogue system can be used as such a promising user interface.
Knowledge graph completion can be used for realizing such a dialogue.
The system needs to obtain plausible question candidates to ask.
We use its scores as confidence.
The knowledge graph completion is a technique applied to knowledge graph and predicts a
missing triplet, based on tensor factorization after formalizing knowledge graph as adjacent tensors.
Various models have been proposed, such as ComplEx, which we used in this paper.
We believe the combination of knowledge graph completion and dialogue system is new and
promising, which have started recently.
However, predicted triplets are often incorrect.
A human user will stop using the system if it repeats incorrect and stupid questions.
For example, if the system asks, "Is flour used to make fish?", the user will think the
system must be stupid.
Last year, we showed that incorrect questions lower user impression via a user study.
So, our proposal is to select questions with the scores about predicted triplets that are
missing in the original knowledge graph.
We assume that more correct questions are better, like this example: "Is white sauce
used to make chicken doria?", because it will not lower user impressions.
Our contributions in this paper are listed here.
First, we proposed a novel framework of using knowledge graph completion scores to select
questions to acquire knowledge.
I have already talked this point.
Second, we applied two modifications for improving knowledge graph completion.
This point will be largely skipped because of limited time.
Third, we conducted a user study with truly unknown predicted triplets having higher knowledge
graph completion scores.
This slide shows a summary for our first contribution.
From a knowledge graph, the system predicts unknown triplets by the knowledge graph completion.
For example, it predicts the cuisine type of Momiji-don, and several candidates are
obtained, like these, with their scores.
The system can select a triplet likely to be correct and ask the user to verify it.
For example, "Is Momiji-don Japanese?"
Then, if the user affirms it, the system can obtain that the triplet is correct.
We applied two modifications to knowledge graph completion.
First, we added new links using substrings of entity name, denoted as SUB.
Second, we limited range of negative sampling during training, denoted as NF.
But, let me skip more details. Those can be found in our paper.
This slide shows the experimental results.
But let me also skip this.
The performance improvement was shown with cross validation.
This slide shows score distributions.
The x-axis is the knowledge graph completion scores, and the y-axis is precision rate of the prediction.
The first point is that the red line, denoting our two modifications, was better than the
black baseline.
The second point is that the higher scores, in the right side in each graph, indicate
more correct results.
These results support that our score-based question selection is reasonable.
Let's proceed to the user study.
We trained the knowledge graph completion with the whole knowledge graph to test for
truly unknown triplets.
Previous experiments were conducted by cross validation, but here, we tested as an open test.
Because there were no reference labels, we verified the correctness of questions by asking
them to crowdworkers.
The questions were made from predicted triplets having higher scores by using templates.
For example, "Does mango mousse tart taste sweet?" was made from this triplet.
We recruited 200 crowdworkers and asked mainly with the five-point scales, from "obviously
correct" to "obviously incorrect".
This is the result on how correct the questions were.
First, our two modifications leaded to more correct questions even with this open test.
Second, question selection based on the knowledge graph completion scores worked well.
The correctness became better by selecting only triplets having higher scores.
This slide shows example questions.
You can check them in our paper.
After the experiment, we found that obviously correct questions, like "Is curry rice served
at a curry shop?", may be also felt as stupid and should be avoided.
This point will be considered as future work.
Let me summarize today's talk.
We proposed question selection based on knowledge graph completion.
We also checked the score distributions and conducted a user study with truly unknown
predicted triplets.
Future work includes to implement and to deploy the framework to a chatbot.
That's it. Thank you for your kind attention.