hello everyone i'm arjun srinivasan i'm a member of the research staff at tableau and
today i'll be talking about my work with vidya setlur on recommending utterances
for conversational visual analysis so to set up some context natural language interfaces
for visual analysis are gaining traction both within academic research as well as part of
commercial systems and with the advancements in nlp these systems have gotten much smarter when
it comes to language understanding and allowing people to create visualizations through language
but from a user experience standpoint especially in the context of exploratory visual analysis
there are two key challenges that still persist in these systems
the first one is knowing what aspects of the data to explore and analyze and the second is
knowing how to phrase the utterances that abide with the system's understanding capabilities
more specifically given a data set users need to decide what attributes they want to look at
what charts they want to consider and each of these decisions ultimately have implications
on the discoveries made during analysis and in other cases even when users know what they want
to visualize they may not be fully aware of the system's understanding capabilities and know how
to phrase the utterances so the system can best understand them so to address these challenges
we investigate how we can leverage utterance recommendations as a way to guide visual analysis
while also implicitly helping users discover and learn natural language capabilities of the system
before i jump into too many details i'll now give a quick demo of our prototype tool snowy
that presents utterance recommendations in the context of a visualization system
that supports both language input as well as manual view specification and for the
demo i'll be using this data set of movies where each row is a movie and attributes are
different properties of the movie including the release year genre worldwide gross and so on
and just to give a brief overview of the prototype's ui on the left you have the attributes
and view specification panel in the center you have the input natural language input region
and the visualization canvas and on the right you have the system generated recommendations
so now let's walk through an example of using snowy to work with the movies data set i just
showed so right off the bat here snowy uh presents utterance recommendations covering a
wide range of analytic intents so for instance this first one how many items exist for each
content reading uh gives you a sense for the distribution or something like the last utterance
here which is what is the trend of worldwide growth across over the years uh gives you a
sense for an attribute changes over time so if i go ahead and select this last recommendation
snowy treats that as my active input and creates a line chart in response
uh snowy now also presents some follow-up uh recommendations on the top right corner and you
can notice that some of these recommendations are much more terser so for instance for the first one
instead of something like show me worldwide gross over the years
for each content rating this recommendation just reads compare across content ratings
now if i select this the system then updates the line chart to a multi-series line chart
and you can notice the spike in the blue line starting around the year 2000 and turns out
snowy also proactively was looking at this chart as we were and suggests a follow-up utterance in
this case uh in case you want to drill down into that spike between 2000 and 2005. skimming through
the new inquiries that are suggested you can notice that the trend stays at the bottom because
we've issued that utterance a couple of times but the phrasing for that utterance now show changes
in production budget instead implies that you can use this utterance to update or swap the active
attribute in your chart so specifically if i go ahead and select this the view largely remains the
same except on the y-axis now instead of having the worldwide gross you have the production budget
now looking at the recommendations again i find this third one about uh comparing average imdb
ratings across major genres somewhat interesting because it's slightly different from what we've
been asking uh which is largely to look at the trends so i like this utterance but not
necessarily the attributes imdb ratings and genres or the combination there so i'll go ahead and
ask the system for more examples of different phrasings and attributes i can use to kind of
ask similar questions and i noticed this last one talking about the average gross across
genres and find it somewhat interesting so i'll go ahead and select this
and in response you can see that the system goes ahead and creates a bar chart and immediately
i can notice that action and adventure have pretty high values compared to the other genres
and snowy also makes this observation and suggests looking at just action adventure and musical
because those are the three highest grossing genres but instead of filtering for now i'll just
go ahead and select these three categories with my mouse to get a better sense for their values
and notice that the follow-up utterances now talk about deictic utterances that i
can issue to the system so in this case uh it's to compute the average values for the categories
that i selected so i picked that and snowy then computes the average just for the three bars that
i had selected now hopefully that demo gives you a little feel for what snowy's interface looks
like and the different types of recommendations that it presents i'm not going to jump into the
implementation details in depth here but i'll walk through an example to give you a high level
sense for how we generate the recommendations because that's really the novel piece here
so let's take the example of this system state from the demo i just showed
where i entered show average worldwide gross by major genres as my input query and then
the system return the bar chart as well as a set of recommendations on the right
now to generate these recommendations given the active utterance and chart
snowy first adds some more knowledge to its kind of context by
looking at the number of interactions that were previously performed with the different
attributes as well as the different analytic intents supported by the tool
then using this as the set of input snowy performs three steps to generate the actual recommendations
first the system considers all possible operations and eliminates those that don't apply to the
current interface state so specifically in this case the bar chart doesn't already have a filter
applied to it so modifying the filter doesn't make sense and gets eliminated then based on the active
interface state and the previous user interactions the system reorders the valid set of utterances
or valid set of operations and once it finalizes the categories it should show
it then goes ahead and parameterizes them to show the actual attributes or with the
actual attributes that would be shown so for instance in this first case of applying a filter
the system selects the attribute major genre because that's in the active chart and then action
adventure and musical as categories because those are the most dominant genres in the active chart
and as a final step the system then uses a set of predefined grammar rules
to phrase the actual utterances that get shown in the interface
now to verify the utility of the generated recommendations and get some feedback on
the prototype's usability we conducted a preliminary study with 10 participants
all sessions were conducted remotely with sessions lasting about 50 minutes on average
and during each session participants performed both targeted analysis as
well as open-ended exploration with two different data sets
the sessions also included a questionnaire and interview phase that focused specifically on
the use of recommendations and at a high level uh participants said that the recommendations didn't
intervene with their targeted analysis phase but in the case of exploratory uh data exploration
they the recommendations not only guided but in some cases even augmented their analytic workflow
however the feedback on the interestingness of recommendations was a little more mixed
primarily because participants had this notion that the system doesn't have enough knowledge to
preempt what they are actually interested in but as highlighted by the third comment here even in
cases where participants had this uh kind of mixed reaction to the interestingness they found the
having the phrasing to be useful and this appreciation for the recommendation
phrasings was in fact a general sentiment across participants suggesting there's more
value in investigating their impact on learnability specifically going forward
speaking of going forward one line of work that we're really excited about to explore
is to look at new scenarios and applications that utterance recommendations can be leveraged
so to give a more specific example snowy's current interface is very WIMP and desktop focused
but the same set of recommendations can be leveraged in other scenarios like mobile
devices and chat based visualization tools where they become much more vital and salient given the
limited screen space as well as the prominence of natural language as an input modality
so to summarize we describe how natural language interfaces for visualization
can generate and present utterance recommendations
and more specifically highlight how these recommendations can guide visual analysis
while implicitly aiding learning and discovery of natural language features
and with that i'd be happy to take any questions thank you