Hi everyone, I’m Sherry, fourth year PhD student at University of Washington.
Today I’m going to talk about “Tempura: Query Analysis with Structural Templates.”
This is the work I did during my internship at Apple.
We expect modern search engines and intelligent agents to answer questions.
Many of these tools are pretty good now,
but these systems still fail at times.
In a recent paper, when a user asks the same question to an agent within a day
the agent gives him different answers.
to improve performance, developers still need to analyze the query data to understand
what type of queries users ask,
where the models work well, and where they can be improved.
We want to perform these analyses on groups of similar queries
so they are more generalizable than individual queries.
But the grouping is hard.
If you have structured data, you can say, let’s do a groupby, and call it a day.
But for text data, you usually don’t have a good specific feature or column to slice on,
and have to rely on more fuzzy methods like clustering or topic modeling.
These methods are not so interpretable or actionable.
Consider this three queries,
How can I learn English? How can I learn Python? How can I install Python?
they will all be clustered together because the algorithm cannot distinguish that they are similar in different ways.
t’s difficult for the developers to predict what queries are included and what are not,
and it’s even more difficult to figure out ways to fix the clusters.
But if you look closer, compared to some other form of text data like an article,
query text are shorter, more compact,
and usually have some interesting structures that we can use for grouping.
Like here, these three questions all have the same question word,
have some VERBs,
and are about some language or programming language.
We take advantage of these structures,
and propose to group query data with structural templates.
Structural templates abstract queries by replacing certain tokens with their linguistic features,
so the queries with shared templates can be grouped and examined together from different angles.
For example, you can replace “PYTHON” with its named entity “PL”
so we can find the programming languages the users care about
when asking “how can I learn” questions.
Or, we can also gather queries on “PYTHON”
if we abstract "learn" to its part-of-speech tag, VERB.
Structural templates also help us examine queries at different granularities.
“How can I learn python” here could just cover this one exact query,
“How can I learn PL” can only cover programming languages,
and “how can I learn NOUN” will also cover natural language,
and even more general, non-language queries like “how can I learn physics.”
These templates are useful for various analysis tasks.
For example, we could represent our dataset with the templates,
and recognize that maybe “how can I learn LANGUAGE” is a very common question pattern.
Or, we can also analyze our model with the templates, and answer questions like
my model is wrong on “how can I learn python.”
Is it because it cannot answer questions about how to learn programming language
or because it does not recognize Python?
We integrate all the nice properties of structural templates
in one interactive user interface called Tempura.
Tempura summarizes a dataset distribution with overview templates,
provides access to raw queries,
and supports in-depth navigation among related templates.
Now, let’s walk through Tempura with a specific case.
The case is from our user study with several NLP developers at Apple.
We want to use Tempura to better understand the performance of a query classifier.
Our task here is to predict if a query is in one of the four categories:
food, health,
procedural explanations on how to perform some tasks,
or none of the above.
As a starting point,
we provide a set of the templates to summarize the query distribution.
Each template has some attributes associated with it,
so users can sort or filter on them to answer various data or model related questions.
The attribute starts with *coverage*, or the number of queries covered by the template.
Then, there are several attributes relevant to the modeling task.
For our classification task,
we have the error rate, or the ratio of incorrectly predicted queries;
The distribution of the groundtruth label,
and the prediction distribution, so we can see if the model would overfit to a syntax structure.
The templates can be cross filtered with the raw queries,
so that we can either get an overview from template statistics
and then find corresponding queries,
or start with a particularly interesting query
and generalize to a larger template group.
For example, here, to find template that’s difficult for the model,
let's sort by error rate in the template overview.
The template with the highest error rate is “how can PRON VERB NOUN”,
with 26 covered queries.
46 percent of them are incorrectly predicted.
Inspecting the covered queries, we find some noisy labels.
Here, similar queries like “how can I cook salmon” and “how can I boil eggs”
are labeled differently as procedural or food.
This makes sense, as queries on cooking procedures
can make these two label pairs non-exclusive.
With this supporting evidence, we might switch to a multi-label classifier.
Now, because all the query tokens can be changed to different linguistic features,
the number of templates for a dataset is usually large.
to effectively summarize query distribution,
we only select a subset as the overview.
The details of the algorithm is in the paper,
but intuitively, we consider two aspects:
To construct representative groups of queries,
we prioritize templates with high coverage.
At the same time, to effectively initiate analysts’ exploration,
we want to include templates that are more preferred than any of its neighbors.
What do we mean by “preferred”?
Like we mentioned before, templates have different granularities,
and you can move or traverse between them.
Let’s again use “how can -pron- verb noun” as an example.
We can traverse down and make the template finer-grained
if we split the NOUN TOKEN to the actual NOUNs.
This will give us many child templates,
each asking how to deal with some random stuff:
how can I or you VERB emails, chess, eggs
On the other hand, we can also traverse up.
Here, by replacing “how” with its abstract part-of-speech tag “adverb”,
we get to a sparser parent template,
and it would express more general questions by merging other question words,
“where can I do something” or “when can I do something.”
Through traversal, some of the resulting templates
are more informative to explore than others.
We prefer templates that all of its traversal are not “informative”,
or less likely to yield more interesting exploration.
This way, templates that are interesting can be in the overview from the beginning,
without the need for additional interactions.
If we plot the query coverage distribution of each template group,
we will see that splitting “NOUN” might not be interesting,
because there is a large number of sparse and unrepresentative stuff,
and the resulting children groups will be too small.
We can color the traversal arrow to indicate how informative it is.
merging how to ADVERB is also not informative:
the sibling groups where and when only have a small number of queries.
As a result, the merged group will still mostly contain questions starting with “how”,
but with a meaninglessly vague question word “adverb.”
But traversing up at “can” might be relatively more interesting.
With both “do” and “can” having many queries,
abstracting “can" to “VERB” gives us a significant gain of queries to view together.
Still, because “should” and “would” are long tails,
our selection would recommend
analyzing do and can as two separate groups initially.
For other templates, analysts can access them through interactive traversal,
so these templates can provide additional views of the data on demand.
Back to our case, we traverse up the relatively more informative “can”,
to “how verb PRON VERB NOUN.”
We then see three other sibling templates here triggered in the neighboring template table,
with verb tokens “do”, “should”, and “would.”
We would imagine questions asked in the form of
“how can” and “how do”, “how should” etc to be quite similar,
but in fact, when we preview the covered queries,
“do” seems to cover more food-related queries.
Let’s verify this with label distributions:
Indeed, in “how can PRON VERB NOUN”,
queries are labeled as health, none, and procedural,
whereas those under “how do PRON VERB NOUN” are mostly food related.
This is an unexpected label distribution difference,
but we can easily fix it with data augmentation.
For example, we can create another batch of food questions
by replacing their “do” with “can.”
Despite the distributional gap,
these templates still seem semantically identical,
and we can merge them to inspect “how VERB PRON VERB NOUN."
Let’s instead traverse down the second VERB
to understand what are the actions being queried.
Among the four child templates covering most queries,
“how VERB PRON cook NOUN” has the most indicative verb,
with all queries labeled and predicted to be food related.
“How VERB PRON do NOUN” has high error rate
with only one predicted label, none,
but almost all its queries are labeled as procedural.
Looks like the model underfits to this form of template pattern!
“make” has the highest coverage,
and the model predicts most of make queries to be food.
However, 6 of 34 queries are labeled as procedural.
If we look into the queries, again, the model is not strictly wrong,
because many queries are food related and could have both labels.
Finally, we also see a model error outlier in “how VERB PRON play NOUN.”
“chess” and “old maid” both being board games,
“how do you play old maid” is labeled as none but predicted as procedural,
whereas “chess” shows the reverse.
This means we should double check the labels
as well as the model stability on various similar nouns.
Our users made many more pretty interesting observations on
the distribution of a query dataset, labeling errors,
and model error patterns and outliers.
One highlight from this is,
you can see half of the errors our developers found were dataset related.
It shows the importance of data quality validation and data wrangling,
before any modeling.
Our user also commented that Tempura provided finer-grained analysis,
intuitive starting point, and made their process more effective.
Extending beyond analyzing just one dataset, we compare multiple datasets
to see if our structural templates can help reveal dataset characteristics.
We did it on three datasets.
MS MARCO and NQ are public search queries from Bing and Google Search Engine,
and Assistant is from our internal anonymized search queries.
For each dataset, we sample 10k queries,
extract and select overview templates from them,
and finally compare the template distribution.
Here I will just highlight the result from Assistant and MS MARCO.
The high level take away is, queries from different sources differ.
Even though “what be NOUN” is an important template in both Assistant and MS MARCO,
this template has a much higher coverage in the MS MARCO.
The selected templates diverge afterwards:
MS MARCO has many just “NOUN” queries
while Assistant has more “what do NOUN mean” queries.
This “NOUN” case is actually pretty interesting.
All the query datasets have a filtering step,
and the goal is to keep knowledge-seeking question queries,
but only MS MARCO shows "NOUN" queries.
just from this one template, it’s clear the dataset filtering strategies are different.
And this is not something you can observe from the question word distribution.
Tempura introduces the concept of structural templates,
summarizes a query dataset by selecting a representative subset of templates,
and also helps analysts navigate the template space interactively.
Our user study shows that Tempura can support data exploration and model error analysis.
Our case study show that tempura can
expose distributional differences between datasets.
Overall, dataset quality and distribution matters a lot,
and we encourage future explorations on data understanding and wrangling.
Our case study show that tempura can
We imagine Tempura, or essentially the structural template exploration approach,
to be useful in many more ways.
First, developers can mitigate the distribution difference through data augmentation.
To flag potential inconsistent labels,
the dataset requester can use the templates to
provide a global perspective in the data labeling process:
the system can easily compare a newly labeled query to
its structurally similar, previously labeled queries,
and warn the labeler if the labels do not match.
Also, quality analysts and designers can build template-based
diagnostic sets to address model biases.
Thanks for listening!