Developing a Conversational Recommendation System for Navigating Limited Options is a paper by Victor Bursztyn, Jennifer Healey, Eunyee Koh, Nedim Lipka, and Larry Birnbaum
in a collaboration between Adobe Research and Northwestern University.
Our background for this work is the rise of conversational interfaces, illustrated by voice-based assistants like Alexa and Google Home.
But when it comes to making recommendations, these systems usually fall into two categories:
They either ask users to fill out a predefined set of slots, like asking for price range and cuisine type when looking for restaurants;
Or they ask users to rate a small set of items as a way for inferring preferences.
But this is not how real conversations work.
Interactions with real sales agents tend to intertwine rounds of preference elicitation and recommendations,
with the agent proactively asking for feedback and inferring preferences from such feedback,
often in the form of a critique to a previous recommendation.
Our system tries to behave like a real sales agent in this sense.
One important component of this system is what we call open-ended critique understanding.
Here, this component transforms the statement “It's a little tacky for the price tag” into “I prefer a more stylish place” -- but how does it do that?
Our system uses a large language model called XLNet in a few-shot setting.
We insert in XLNet’s prompt a few examples of transformations in the restaurant domain.
Then we append the critique we’d like to transform plus the string “I prefer” and we sample the most likely continuation of that sentence.
Now that we have an open-ended preference inferred from a critique, the system can make recommendations also using open-ended statements.
Here’s how it goes from “more stylish” to “A previous customer wrote that Shanghai Terrace has a really amazing rooftop dining area."
We use customer reviews as unstructured repositories of knowledge,
and we define recommendations as positive review sentences with maximum specificity in relation to preference statements.
To extract these recommendations, first we use an emotion detection model to filter only the positive sentences from this set of customer reviews.
We then encode these positive sentences in an embedding space along with the preference statements.
With that, we rank each positive sentence in terms of their semantic similarity to the preferences and return our best match as a recommendation.
This is how we go from critique to preference, then from preference to recommendation in a completely open-ended way, like real conversations.
To test the hypothesis that this system would be better, we designed an experiment with two systems, A and B.
Both systems allow users to see nearby restaurants, drill down on an option, and open its menu, photos, and reviews.
If the user rejects an option, system A proactively asks for feedback and processes this feedback to make a new recommendation using the proposed methods.
System B, however, doesn't ask for feedback or use it in any way. It only displays requested information until the user is satisfied with an option.
The experience of system A is illustrated here, with the user being able to utter critiques like "I don't think that steak is their specialty" when prompted by the system.
Our experiment was conducted over Zoom with a sample of 16 participants from a university community.
Each participant was presented with four scenarios to motivate their journeys:
lunch under time constraint, lunch with kids, romantic dinner, and vegetarian dinner.
All scenarios were rotated to keep the order balanced.
We measured the number of queries uttered by the users to evaluate the efficiency of both systems, and we surveyed user confidence upon accepting a choice.
This table shows the average and median number of queries needed in each scenario, and the self-reported confidence also in each scenario.
We found that our system was more efficient in two scenarios with clear constraints: lunch with kids and vegetarian dinner. 0:04:41.800,1193:02:47.295 System B, however, only allows users to utter predefined commands to ask for information or accept an option.
But it was less efficient in the romantic dinner scenario.
And we found that our system inspired more confidence upon accepting a choice in three out of four scenarios, tying only in the hardest one: the vegetarian dinner.
Thank you for watching out presentation.