Hello, my name is Sara Evensen. My colleague Jonathan Engel and I will be
presenting our work on Teddy, our interactive text analytics tool. This research was conducted
at Megagon Labs together with Xiong Zhang of University of Rochester, and our colleagues
Cagatay Demiralp, Yuliang Li, and Wang-Chiew Tan.
Reviews are integral to e-commerce services
and products. They are a powerful distributed representation of a service or product, and
can provide reliable signals about how to improve them.
But working with thousands of reviews, which
are typically noisy incomplete text, is a daunting task. We conducted a user study to
better understand the current methods and limitations of this type of work.
In interviews with fifteen data scientists
who work with review text, we found that many could benefit from interactive systems for
review analysis tasks.
In response we introduce Teddy, an interactive system that enables data scientists to quickly
obtain insights from reviews and improve their extraction and modeling pipelines.
In this presentation we will first outline
the findings from our user study, and then provide a demonstration of the tool we created
to address the bottlenecks we found.
We interviewed 15 participants (12 male and 3 female), all of whom were employed in technology
companies with the task of analyzing review data.
Although the sample size is small, because
we focused on a specific domain with low workflow variation, we believe that these findings
are valuable. Here is what we learned.
Most of the time, data scientists are satisfied using existing language models such as BERT.
They are less concerned about developing new models, and instead they are focused on improving
the quality of their training data.
Many interviewees voiced that the quality of the training data was as least as important,
if not more, than the type of model used.
Similar to findings from our earlier work, we observe a dichotomy between senior and
junior data scientists. E.g, pipeline management and provenance is an important concern motivated
by the cost of context-switching among senior data scientists.
On the other hand, junior data scientists are more concerned about data labeling and
crowdsourcing as bottlenecks. Division of work b/w senior and junior data scientists
is perhaps only one aspect of the dichotomy here.
Unsurprisingly, data scientists spend most of their time on data preparation, lending
further support for the currently accepted but not necessarily acted upon wisdom.
We find that the most time consuming or challenging
data preparation steps in review mining are, however, labeling, designing and specifying
crowdsourcing tasks, and interactive exploration, not necessarily cleaning
Finally, and perhaps most surprisingly , participants
didn't report scalability to be a significant bottleneck, even those who regularly work
with very large review collections. They often resorted to sampling and were content with
it.
Informed by these insights and our own experience along with a prior data exploration paradigm,
we derived 5 design considerations for the Teddy prototyping system.
1. Users should be able to explore, inspect, and compare clusters of review text based
on the semantic similarity as determined by the opinions and topics within the reviews.
2. Users should be able to explore domain-specific aspects of
reviews as defined by the domain that they are working in
3. They should be able to programmatically control the data
displayed. In particular, the prototype should support sorting
filtering, and pattern matching on raw text and extracted features.
4. They should be able to get an overview
of the reviews from all the entities or from a single entity.
and 5. They should be able to zoom in and out the data visually and semantically in
a multi-scale fashion, accessing details for entities and reviews on demand.
Now my colleague Jonathan Engel will show us the tool in action
Hello, my name is Jonathan Engel and I'll be walking through the demo portion of this
presentation.
Let's say we're a data scientist working
at a company that owns several hotel properties, assigned to identify insights about the performance
of various properties by assessing reviews of those hotels on tripadvisor.
We've already developed an algorithm that can identify opinions in text data, and we've
applied that to our data set, which we've now loaded into our web app.
First things first let's look at our summary statistics for the data set as a whole. We've
got about 5000 reviews, and we can see here various average length measurements for these
reviews in either characters, words or sentences
Looking further down, we can also see the
top words and top bigrams for the data set.
Somewhat unsurprisingly for a data set about
hotels, hotel and room are very common, and we also see a lot of bigrams about San Francisco,
which makes sense because that's where a lot of these hotels actually are.
Now, scrolling down further, we can also see the average attributes that were extracted
with that algorithm that I mentioned earlier
and we can see that in general people seem
to have a pretty positive opinion of these hotels and in particular seem to have positive
opinions about the staff, in addition to the location, food, and facilities of these hotels.
Let's draw our attention to the actual clustering view of this app, and we can see here on the
right that we have a cluster that is both larger than the other clusters and has a different
shade
verging on gray, which would indicate that
its sentiment score overall is closer to neutral. If it were red, that would indicate a negative
overall sentiment
So let's click on it and see a bit more of
what we can find here
and we see that our general sentiment has
taken a pretty significant hit over the prior dataset and in particular our evaluation of
staff seems to have dropped
Meanwhile, location food and facility evaluations
seem to be roughly the same as earlier, and if we look up at the bigrams and top words,
those are also roughly the same
Now you can also see now that we've clicked
on the cluster that a bunch of text has appeared on the right. Each of these text blurbs corresponds
to a review, and you can see that we've highlighted extracted opinions in blue or red depending
on the associated sentiment that was detected
Now if we go ahead and click on these reviews,
we can expand them and see the full text.
However, we're interested in seeing what's
going on with these negative staff reviews so let's actually filter this list to only
show us reviews that have the staff attribute extracted
Now when we first run this is only going to be over the reviews that were visible, however
if we want to run it over the full backend we can click this button and see that we have
a lot of reviews, and as predicted many have a negative review of staff
Now if we go back to the other clusters and compare them side by side we can actually
see how their attributes line up and we see that they have a higher staff score as predicted
as well as a significantly higher overall general sentiment
However, we're interested in drilling down further to see what's going on with these
negative reviews so let's double click on this large cluster and see the subclusters
that are associated with it
Now we can see that much like the previous setup that here again we have two smaller,
more positive clusters and one larger, more negative cluster
So let's drill down again on this negative cluster and we can see here that the scale
has significantly zoomed in
so even though staff reaches this first line
this is actuall .0001 positive sentiment value
Meanwhile, facility seems disproportionately
high so let's zoom down a bit further to see what else we can find
again we have most of our negative reviews in this larger cluster and here we see that
the negative staff reviews have been placed into this cluster by our algorithm
From here we can also see that there seems to be some correspondence with the negative
facility score as well
if we drill down further we see that we actually
can't further subcluster so we're pretty much done with this vertical
However let's say that instead of being asked to survey the dataset as a whole we were actually
instructed to assess just one hotel
Say, the first hotel that we have up here, which is the Hotel Drisco
Now if we load this cluster we have a very similar interface but we're only looking at
reviews corresponding to this hotel
and we see that there's generally positive
sentiments overall and if we are to look again at the most negative, largest cluster and
drill down on it
we see very quickly that there's a large neutral
cluster that again has low staff sentiment which seems to be correlating with low facility
and low overall general sentiments here
So there's maybe a trend that we're seeing
here that negative opinions rarely come alone they come together with other attributes
But let's try to find some good things about this, like what were people thinking about
the restaurants in this subset
we can use a grep tool that's built into this
and search for restaurant regardless of capitalization
and we see a few results seem to be generally
positive ones, let's see what else we can find here let's try looking up breakfast and
see what shows up
and we can see here that if we click on them
we'll actually see matches highlighted and we have some very happy customers when it
comes to the quality of breakfast at these hotels
This concludes the demo portion of this presentation.
We believe that in developing the Teddy review
explorer, we have created a tool that can help data scientists more quickly explore
and understand new datasets of review text, addressing key bottlenecks identified in our
user study.
For future work, we intend to continue developing
Teddy by adding even more metrics and visualizations for analysis, built-in classification algorithms,
and expanding to cover other types of text data in addition to reviews.