Hello everyone, this is Yali Bian. I work with Dr. Chris North at Virginia Tech.
Our paper is "DeepSI: interactive deep learning for semantic interaction".
First, what's semantic interaction. Semantic interaction is a commonly
used interaction method in visual analytics systems for sensemaking.
With semantic interactions, users can naturally express their preference about the data
through updating the data projection, instead of manipulating the model parameters.
For example, the user can make four personalized clusters about different risk factors on the
data projection to express how he thinks the covid-19 research papers should be categorized.
The underlying model needs to infer the intents behind users' interactions
and provide visual feedback. The performance of semantic
interaction is highly dependent on the quality of the data representations.
For example, the SI system can not capture users' semantic level interactions of
smoking-related concepts, if the documents using bag-of-words as input features.
Deep learning is the powerful representation learning method, which can extract high-quality
representations from the raw data. This offers a new approach to improve the
performance of SI systems with deep learning. We name the new SI system as DeepSI
One of the basic DeepSI methods is using the DL model as the fixed
feature extractor, which we call DeepSI-vanilla. DeepSI-vanilla using the frozen DL model as the
preprocess before the interactive training loop. Even this basic method
has shown state-of-the-art performance. DeepSI-vanilla could capture user-defined
two clusters and separate them into two directions in the projection.
However, this method still not fully use of deep learning model.
This paper aims to fully use of deep learning model by gaining
contextualized representations from it so that even more complex intents can be captured.
As shown in the Figure, providing accurate feedback for four different
risk-factor paper clusters related to covid-19. For this goal, we need to solve
two research questions: 1. How to use semantic
interactions to adapt the deep learning model to gain user- and task-specific representations?
2. How to make the adaptation process efficient, so that the user just needs
to perform a small number of interactions to express their complex intents?
We propose DeepSI-finetune, which uses semantic interactions to fine-tune
deep learning and gaining contextualized representations, which is then used to define
users' intents, behind their interactions. Our solution consists of two design rationales,
as shown in the figure. first, integrating the deep
learning model into the interactive sensemaking pipeline, to make sure users' interactions are
used to update the model parameters. Second, introducing minimal parameters
to the pipeline to effectively train the DL model without any information loss.
The detailed pipeline can be described in two directions.
In the forward model prediction direction, the raw text d will be mapped to high-dimensional
representations x through the bert model with updated parameters.
The new x is projected to the 2D scatterplot through
the interactive DR component, specifically, MDS. The analyst can perceive the updated projection
and gain insight into the data. In the backward model updating direction,
the analyst will first update the projection layout by moving some data points through y.
The interactive-DR component, inversed-MDS, calculate the pair-wise distances in the
projection layout y and use it as ground truth to train the
high-dimensional representations of the data. Consequently, updating the internal parameters
of the BERT model from the last layer to the first layer through backpropagation.
Our proposed pipeline is implemented as a web-based prototype in two parts.
The backend part is implemented in Python with Flask.
We use pytorch lib to set up the DL framework. The pretrained BERT model is from the huggingface
transformers. The MDS is from the Scikit-learn.
The frontend visual interface is developed using D3.js.
The visual interface has two views: the scatterplot view
displays the data projection, which can be updated by both the model and the analyst.
The sidebar view shows the selected document content.
Here is the demo for the prototype. As shown previously,
the user drags covid-19 related papers to different clusters based on risk factors.
It is worth noting that the ground truth color information is only for user evaluation and
interaction. Only user interactions are passed to the system to finetune the deep learning model.
Results shows our method can accurately capture users intents and provide feedback.
In the experiments, we measure the performance of our method in two respects:
1. accuracy, how accurately can the method capture the user's intent?
2. efficiency, how many interactions does the method need to capture the intent?
We compare our method with the baseline method, DeepSI-vanilla, which is the
best-performing model from previous studies. We performed two complementary studies:
the case study, which is a human-centered qualitative analysis.
the simulation-based evaluation, which is a algorithm-centered quantitative analysis.
The case study uses the covid-19 open research dataset, as shown in the demo.
The ground truth is the four risk factors. During the case study process, the same
interactions are performed in both systems. Results show that our method can accurately
capture the intents and provide projection with four clear clusters.
While the baseline model can't capture the intent.
Further study shows that vanilla model only support a relatively simple two cluster concept.
The simulation-based evaluation replace the analyst with a simulation component,
which consists of two parts: 1. the interaction simulator
to generate a training projection 2. the evaluation simulator to evaluate
the accuracy of the predicted projection. In this study, we use three tasks with
different number of clusters to evaluate the performace in different difficulty levels.
Results shows that our method accurately and efficiently
learn concepts in all these three tasks. While the vanilla method can only partially
solve the simple SST task, with two clusters. This is consistent to the case study results.
Besides performance, we discuss the method in the following three perspectives.
First, our method is general enough to apply other deep learning models into the semantic interaction
pipeline, for other visual analytics tasks. In addition, for scalability, the MDS is still
the main time-consuming component, which is a hard big O(N2) time complexity.
Finally, from the perspective of deep learning. Our method is a semi-supervised
interactive deep metric learning model, with a specially designed loss function with inversed MDS.
For the next step, we plan to augment our method in two perspectives.
First, enable explanationable AI, which is known as XAI.
the current method only supports the projection scatterplot as an explanation,
which is case-based reasoning. Further interpretation is need for
the deep learning method to facilitate analysts in making sense of complex dataset and understanding
the complex deep learning model. Second, replace the traditional DR
component with projection networks. As discussed in the last slide,
the traditional dimension reduction method MDS is both time-consuming and inconsistent.
We plan to use deep metric learning method and projection networks to solve these drawbacks.
In conclusion, contributions of this paper includes two parts:
1. the deepsi-finetune framework to improve semantic interaction inference
with deep learning. 2. two complementary studies
to measure the performance of our method and reveal improvements.
Thank you for attending. Please feel free to ask questions after this video.