Hi, I'm Jason Portenoy, and today i'll be talking about our work on Bursting Scientific Filter
Bubbles and Boosting Innovation via Novel Author Discovery, which is joint work with
Marissa Radensky, Jevin West, Eric Horvitz, Dan Weld, and Tom Hope.
With millions of scientific papers coming out every year,
researchers are forced to allocate their attention to increasingly narrow areas.
This creates isolated silos of scientific knowledge, which limits our ability to discover
knowledge outside our bubbles. This presents a major challenge for the future of science,
since breakthroughs often emerge precisely from building these connections across different areas.
For example, big innovations in computer science like neural networks
or quantum computing were directly inspired by other areas of science.
Our goal in this work is to connect scientists to peers outside their bubbles,
to help find fellow researchers who spark new research ideas.
An existing example of an author recommendation system in the real world is a conference app that
recommends people you may be interested in meeting based on shared interests,
for example if you're both interested in machine learning. But the problem with
this methodology of recommendation is that it reinforces your filter bubbles,
showing you people in areas you're already familiar with. In our work we want to help
researchers find authors who make them think of new methods they can apply to their problems,
or new problems to work on, or new data sets to explore --- things that are both new and relevant.
The approach we designed for this goal involves both commonalities and contrasts.
We're looking for authors who share common facets with you,
who are similar to you in certain aspects, to ensure recommended authors are relevant to you.
For example authors who work on similar problems. But we also want these authors
to have contrasts along other dimensions. For example, they may work on similar tasks but use
dissimilar methods. This increases the chance they will be novel and unfamiliar to you.
For example, take Dan as an author.
We first extract these facets we care about from Dan's papers --- tasks, methods, and data sets
--- using information extraction models. We then find peers with commonalities along one dimension,
for example similar tasks, and then we find the contrasts: peers who not only share commonalities
but also have contrasts such as new methods that can potentially inform Dan's work in novel ways.
As an important technical note, measuring the similarity between raw spans of text extracted
from papers is far from trivial. In this work we apply an approach that worked well in our
experiments. We take a language model trained on many scientific papers and fine-tune it on
semantic similarity tasks to make the language model representation more suitable for measuring
distances between spans of text. This gives us a set of embeddings for terms in documents
to represent individual authors. We aggregate the appropriate terms,
for example all of the methods terms for that person's papers, weighted by a relevance score
that takes into account the author position and the importance of the paper. We can then
compute the similarity between two authors using these faceted author level embeddings.
An important structural observation we make is that researchers wear many hats: they have
different themes or lines of work, working with different people, and we'd like to be able to
capture these different personas. For example, one persona Dan has is around NLP and language models,
while another persona is more related to explainable AI and HCI
these are two real personas we automatically extracted from Dan's papers
which we found by clustering Dan's papers into cohesive subsets around certain topics.
Taken together, this representation of an author can be thought of as a matrix,
where the rows correspond to papers, the columns represent the facets, subsets of rows
correspond to personas or clusters of papers, and the intersection of rows and columns can,
for example, be all the mentions of tasks in a specific cluster of papers.
And given this representation, we can now perform author retrieval. We can obtain
vector representations for different slices of this matrix, for example taking all dimensions
of tasks in a specific cluster of papers (spans of text mentioning tasks), and aggregate them
into an embedding that can be used to retrieve authors who are similar along specific dimensions.
We use this author representation and retrieval capability to build a system we call Bridger,
where you can discover novel authors and research directions.
And now, I will hand it over to my co-author Marissa to talk about the
user studies we did to evaluate our work.
We conduct two user studies relevant to Bridger.
The first addresses how to depict recommended authors: Does describing authors tasks, methods,
and resources help users better understand recommended authors?
The second addresses how to rank the recommended authors:
Does ranking author recommendations using task-based commonalities and
method-based contrasts spark more ideas for new research directions?
To determine if describing recommended authors' tasks, methods, and resources
helps users better understand them, we conduct a study with 13 computer science researchers.
We show them facets generated by Bridger using papers pulled from the Semantic
Scholar scientific search engine, as well as topics from Microsoft Academic Graph,
for six authors --- one they know well, such as themselves or their advisor,
and five unknown recommended authors. We see that participants generally found tasks, methods,
and resources helpful to better understand what kind of research an unknown author does.
And looking at different ranking strategies for the known authors' tasks and methods,
participants overall preferred our relevance score ranking strategy which is detailed in the paper.
We then move on to evaluate our system Bridger.
We recruit 20 computer science researchers, spanning full professors to graduate students.
We pull their papers from Semantic Scholar and, as a baseline, we use a state of art
real world retrieval system used in practice by Semantic Scholar
for surfacing search results and recommendations. This baseline recommends authors based on the
aggregation of all the user's papers rather than specific dimensions of the user's work.
So let's say Dan is a participant in our study. Dan sees recommended authors,
recommended according to our condition or the baseline, and different personas
show different recommendations according to different lines of work.
Dan is instructed to examine the recommended authors --- look at their papers,
the topics and tasks they work on, and the methods and resources they use --- and check
items that he considers potentially interesting for forming new and relevant research directions.
We aggregate the proportion of checked items for our condition and for the baseline, and
find that overall about 80 percent of researchers preferred recommendations based on our approach of
commonalities and contrasts --- specifically similar tasks and distant methods --- for
discovering authors who are not only relevant but also spark ideas for new research directions.
Under the same similar tasks and distant methods condition, we also find that our new
persona based representation further boosted user preference for recommended authors by 75 percent.
Finally, a result we're particularly excited about is that when we look at our recommendations
versus those of the Semantic Scholar baseline, we find that we are able to to suggest authors
who are much more diverse much more distant to the end user, while still being relevant.
What you see on the left are plots of distance measures between the end user
and recommended authors, such as distances based on the citation graph, shared publication venues,
or distances in the co-authorship network. We find that our approach is able to find
substantially further-away authors, while sparking ideas for new methods or tasks
that are relevant to the user, which is empirical evidence we are able to make progress toward the
challenging task of bridging across diverse scientific communities, as we set out to do.
In conclusion, we present our work's main contributions:
A multi-dimensional author representation for matching authors along specific facets;
Boosting discovery of useful authors and ideas from novel areas;
Exploring how to effectively depict recommended authors;
And evidence of bridging across communities.
Thank you very much!