Hyeonsu Kang: Hi! My name is Hyeonsu Kang and today i'll be presenting our TOCHI invited paper, titled, Augmenting scientific creativity with an analogical search engine. This was a collaboration project with Xin Qian, Tom Hope, Dafna Shahaf, Joel Chan, and Aniket Kittur.
Hyeonsu Kang: To motivate this research, consider an example of successful analogical inspiration in science, which used to be a 50 year old, challenge in aerospace, engineering. And that is the problem of how we could transfer a large number of solar arrays into the space for use by telescopes, satellites and others.
Hyeonsu Kang: One solution might be to make the cells lighter, another might be to build a bigger rocket.
Hyeonsu Kang: But an astrophysicist working at NASA at the time, happened to be an origami expert as well, and he recognized the deeper structural similarity between folding Origami papers, and transporting and deploying large solar arrays in the space,
Hyeonsu Kang: despite the obvious surface level differences. This kind of recognizing deeper structures, despite the surface of the differences is what we term analogical processing. And analogical inspiration in sciences are where, partly because analogical processing is difficult.
Hyeonsu Kang: One reason it's difficult is human memory is really sensitive to surface similarity which leads us to faver near analogs. analytical processing that involves just a few relations at once can incur a heavy cognitive load and tax our working memory.
Hyeonsu Kang: In addition to that, the number of research papers is growing exponentially, and the specializations are getting deeper and deeper, making no single person to be an expert across many domains to make the connections.
Hyeonsu Kang: So in this project we tackled, whether development of an analogical search engine that moves beyond that surface similarity
Hyeonsu Kang: and processes, relations between many papers and various domains at once is a feasible achievement.
Hyeonsu Kang: This is challenging because one there is a lack of data which means no large corpus of gold standard analogical inspirations.
Hyeonsu Kang: exists. At the same time there's too much data with exponentially growing and literature, and deepening specializations making the connection morning challenging.
Hyeonsu Kang: There's also the complexity involved in the scientific research that makes analogical processing naturally more difficult than other previous domains, such as product innovation.
Hyeonsu Kang: And finally, there's a big time and resource commitment required to actualize these inspirations. The incentive structures may work against analogical inspirations, and make an evaluation of success tricky.
Hyeonsu Kang: Given these difficulties, we thought a successful first step towards supporting analogical inspirations in sciences was to build an analogical search engine for sciences that find structurally aligned papers. Specifically, papers that match on similar purposes as to query.
Hyeonsu Kang: Matching on the purpose and diversifying by mechanisms was previously shown to be a powerful strategy for facilitating creative education in products,
Hyeonsu Kang: and help scientists, come up with creative adaptation ideas using the search results.
Hyeonsu Kang: So our measure of success is facilitation of creative adaptation over direct application ideas. But what counts as creative adaptation or direct application? To define these types, let's consider an example of heat transfer in semiconductors focusing on interfacial thermal conductance.
Hyeonsu Kang: Here Paper A has a similar purpose to transferring heat but it looks at more directly controlling thermal metallic contact conductance. This may be a novel idea, but something that's more familiar to the user and the user can also anticipate a direct application out of it. So this is an example of direct application.
Hyeonsu Kang: On the other hand, Paper B uses different fin structures to to deal with this heat transfer problem. Though this is not exactly a familiar domain to the scientist, she can understand the gist of it and come up with a different idea, which is to design Nano fin structures for semiconductor walls.
Hyeonsu Kang: So characterized by the novelty and uncertainty, this is an example of Creative Adaptation.
Hyeonsu Kang: Our first probe system combined AI with human filtering to find structurally aligned papers, and empirically examined whether facilitating creative adaptation ideas with search results was feasible.
Hyeonsu Kang: To do that we crowdsourced, purpose and mechanism, annotations for 2,000 papers to train a sequence to sequence tagger which extracted purpose and mechanism tokens from a paper abstract. Then we applied this model to over 2 million paper abstracts to predict purpose and mechanisms,
Hyeonsu Kang: embedded the extracted purpose text to create a nearest neighbor similarity index for search.
Hyeonsu Kang: Using the similar similarity index, when the user query arrived at the engine, we first embedded the query with the same encoder, then searched through the index to find near purposes.
Hyeonsu Kang: We realized that due to noise in the automatic purpose extraction phase and the similarity embedding, there were results that were obviously too far to be useful in the control;ed experiment which we used the human filtering step to remove.
Hyeonsu Kang: Using the output of the AI + human filtering pipeline, we ran a controlled experiment, comparing the analogical search engine with a more conventional scholarly search engine using BM25 keyword search
Hyeonsu Kang: to find out the benefits and challenges. We created a blended list of analogical and keyword search papers, and asked the participants to read each result blind to condition to brainstorm new research ideas.
Hyeonsu Kang: Thus the main data source consisted of 267 transcribed think-alouds of brainstorming from 8 participants, which 2 coders coded into 4 types of ideation outcome and 3 degrees of purpose match. The middle level purpose match, denoted here as Part
Hyeonsu Kang: indicated that the idea had a similar high-level purpose, but differed on low level purposes.
Hyeonsu Kang: Okay. So for for the first results, we looked at whether analogy papers facilitated creative adaptation over direct application.
Hyeonsu Kang: We compared the analogical search papers to keyword search papers and creative adaptation ideas were 2.6 times more frequent among the analogy search papers than keyword papers.
Hyeonsu Kang: While most of the direct application ideas were associated with keyword papers.
Hyeonsu Kang: So analogical papers were more associated with creative adaptation ideas. But how does matching on purpose affect the ideation outcome?
Hyeonsu Kang: We found that the degree of purpose match mediated how the condition of a paper affected the creative adaptation ideation outcome.
Hyeonsu Kang: Specifically partial meaning, low-level purpose mismatches resulted in creative adaptation ideas more often in analogy search papers than keyword search, papers.
Hyeonsu Kang: Looking into the relationship between ideation outcome and the degree of purpose match in the keyword search condition when their purpose fully matched, 70% of the its outcome was the direct application type, while direct application ideas were only 6% among the partial purpose matched papers.
Hyeonsu Kang: As the purpose-mismatch increased the frequency of creative adaptation ideas also increased in the keyword search condition as expected. But when comparing the partially matching analogical versus keyword search papers, the frequency of creative adaptation ideas was more than double among the partially mismatching analogical search papers.
Hyeonsu Kang: However, partially mismatching the purpose can be a double-edged sword. Depending on how it mismatches, this may facilitate creative adaptation, or render the analog completely useless.
Hyeonsu Kang: So this led us to study what the challenges involved with knowing which mismatches were beneficial to creative adaptation.
Hyeonsu Kang: answers to which have implications for facilitating useful misalignments for ideation. So to understand the challenges in the process we conducted interviews with participants and thematically coded their transcripts. The first challenge was related to discovering sailent structural elements.
Hyeonsu Kang: When connecting with papers and different domains, participants recognized that their problems may be represented at different levels. But it was not always clear how to express the problem in advance.
Hyeonsu Kang: For example, the combination of structural elements or attributes sometimes resulted in an unintended intersection of domains. P3 said she felt like she was tricking the machine when using popular search engines, because combining terms like "thin film", "pressure" often led to solid thin films in manufacturing.
Hyeonsu Kang: But what she's actually interested in were liquid, and not solid thin films in the domain of heat transfer.
Hyeonsu Kang: Importantly, researchers had to see papers with mismatching structures to know which attributes were important and refine their problem representation accordingly.
Hyeonsu Kang: For example, after seeing a bunch of results related to all different aspects of the problem, such as the coating material, size, interface and fabrication processes, etc. did P6 realize that these were all important aspects that he didn't think of before that distinguished between the domain he's interested in from the rest.
Hyeonsu Kang: The second challenge was related to supporting superpositions and mismatches with just the right amount of variation to find useful diversity in the result.
Hyeonsu Kang: For example, looking for only the exact purpose matches may lead to missed opportunities. As P1 said here, even though fins represented an idea in a different scale for transferring heat, which prevented her from proactively searching for them, but having been exposed to the paper about fins forced her to think about novel designs for semiconductor walls.
Hyeonsu Kang: But at the same time too much deviation can make the papers useless.
Hyeonsu Kang: To conclude, we have investigated the opportunities and challenges involved with building an analogy search for scientific papers.
Hyeonsu Kang: Our human plus AI analogy search engine found novel papers that facilitated creative adaptation ideas, while purpose-match, mediated the effect of condition on the ideation outcome with some low-level mismatches in analogical search papers leading to more creative ideas.
Hyeonsu Kang: We also identified challenges involved with knowing what mismatches were beneficial to creative adaptation. Challenge One found the iterative nature of discovering salient structural elements. Scientists may be able to specify what aspects of the structure were important only after seeing examples of mismatching structures.
Hyeonsu Kang: Challenge Two found the difficulty of supporting superpositions of mismatches with just the right amount. While too much deviation may break the analogs, too little mismatches may not be sufficient to facilitate creative adaptation in ideation.
Hyeonsu Kang: Thank you.