Hello everyone, my name is Thanet Markchom.
I’m from the Department of Computer Science, University of Reading United Kingdom.
My topic is Augmenting Visual Information in Knowledge Graphs for Recommendations
Recommender systems are designed to suggest items relative to users’ personal interests.
One popular approach is called collaborative filtering or CF.
This approach uses user-item interactions for predicting recommendations. However,
the CF effectiveness decreases if user-item interactions are sparse. Therefore, to overcome
the problem, some side information has been used including knowledge graphs (KGs) and images.
knowledge-graph-based recommender systems can be divided into three major types
The first one is Embedding-based methods that use node embeddings
for recommendation such as Metapath2vec.
The second type is
Path-based methods using paths connecting between user and item for a recommendation.
The last type is Hybrid methods that combine both of previous types. One
of the the-state-of-the-art models is Knowledge Graph Attention Network.
However, most previous models usually consider semantic factors
such as categories or genres of items and ignore visual factors.
Therefore, in this work, we would like to incorporate visual information into a
knowledge graph for recommendations. The proposed approach begins with extracting visual factors
from item images. These visual factors are generated from pre-computed image features.
In this work, five feature types are considered which are SIFT, SURF, ORB, color histogram
obtained from HSV color space and CNN-based feature obtained from the Caffe reference model.
After that, we include these visual factors in a knowledge graph creating a visually-augmented
knowledge graph which contains both semantic factors and visual factors.
To use this augmented knowledge graph, we propose a visually-annotated meta-path
to generate the hybrid context containing both semantic and visual information.
Then we adopt a method called Skip-gram to learn node embedding based on the hybrid context
and use them to build a user-based CF recommender system.
We adopted two real-world datasets for conducting experiments. The first
one is Amazon dataset. We selected only the Clothing, Shoes and Jewelry subset
of the whole dataset. This dataset includes users, items and item categories as semantic
factors. The second dataset is MovieLens dataset which is a HetRec2011-MovieLens-2K version.
It includes users, items, genres and tags for semantic factors.
In the first experiment, we compared our approach named as VR and the baseline named as
MR which uses metapath2vec based on meta-paths with only semantic factors. We applied several
meta-paths selected from the literature for comparison. According to the results,
our approach outperformed MR models in terms of both precision and recall.
We also compared our approach with different types of visual factors.
From the results, our approach with SIFT performed best for Amazon dataset
while color histogram and CNN-based features performed best for MovieLens dataset.
Lastly, we applied the proposed visually-augmented knowledge graphs in Knowledge Graph Attention
Networks named as GA approach. For Amazon dataset, there is no significant improvement.
However, there is an improvement in the MovieLens dataset results,
when we applied the GA approach with color histogram visual factors.
In conclusion, this work proposes a knowledge graph called visually-augmented knowledge graph
which incorporates visual information into a regular KG.
To leverage the proposed graph, we also introduce visually-annotated meta-paths
to form the hybrid context containing both visual and semantic factors.
Then we propose an approach to use this hybrid context
to build a user-based CF recommender system.
According to the experiments, the proposed approach outperformed the
baselines that consider only semantic factors.
We also show that the proposed visually-augmented knowledge graph can be applied in the
state-of-the-art KG-based recommender system to improve the performance as well.
Here are the references.
Thank you very much.