(audience applauds)
- Great, thanks so much.
We're definitely in, like the most fun session.
So you did a good job picking.
So I'm really excited to talk to you about our work
on metaphor writing, and creativity,
and writing support more generally.
But first I want to motivate a little bit,
why we're super interested in metaphor,
which is a very specific writing activity,
or rhetorical device.
So there's two main questions
that we're really excited about.
The first is about co-creative tools,
and this idea that we're starting to design programs
that are really interactive
and actually might be contributing,
like more actively contributing, to the creative process.
So these are, this is Drawing Apprentice,
which is a really cool drawing app
where you draw, and thenthe computer draws with you
based on principles of abstract drawing.
The picture below is from a CHI paper from last year
which is a turn taking drawing app with an AI agent.
So this is starting to become really exciting,
it's pretty new, and I think there's a lot
of really big questions
about how you do this kind of collaboration with a computer.
But the second is just generally
about how we support creative writing,
because most creativity support in the past
has focused on the visual arts and music,
and there's a lot of really amazing work in these areas,
but there's just relatively little in writing,
even though we're a hyperliterate society,
and especially with the Internet and social media.
Like, writing is one of the main ways
that we're communicating with people these days.
So Metaphoria is this project in the middle, right?
It's a co-creative tool,
but it's really focused on creative writing.
So metaphor in particular is a really nice creative task,
because it's very challenging,
so it's really hard to come up
with a super interesting and new metaphor,
but it's also really open-ended,
so there's no one best way
to describe something metaphorically,
or one best way to connect two ideas.
So we set up this problem,
as this idea of what's a metaphorical connection
between two nouns?
And so normally the way we do this
is we pick an abstract word,
something like anger that doesn't really have
a physical embodiment,
and that's kind of complex,
and you want to find some way to describe it.
And then we take something really concrete like wood
that has a really strong physical embodiment,
and we say, how is anger like wood?
And then when you ask this question you end up coming up
with a really interesting metaphor.
And so you can pair these words all day long,
and it kind of creates a little puzzle
for you to do on your own, and they're pretty challenging.
Like, how is peace like a window,
or how is gratitude like a stream?
And what's cool is that it's not obvious
how you would relate these,
and there's a lot of different ways
that you can relate them, right?
Okay, so, anger burns when lit,
peace lets in the sunshine,
gratitude flows over you like a stream, right?
Just immediately these kind of simple answers,
if you sat here and just tried to think of something,
they start to sound really beautiful.
So this is a really nice framework,
and a really nice testbed
because you can just randomly pair these words all day
and you keep getting new problems.
So I want to spend the next five minutes or so
talking about the system design,
'cause the main thing is that we built a real thing,
it's online right now if you want to go play with it,
metaphor.ga, and the main idea
is that the user puts in the word on the left,
so in this case peace.
You can type in whatever word you want
and then it pairs it with a random concrete noun
which is like, they're mostly nature-related but not always,
and then the system comes up with 10 potential ways
that they might be related.
So this is an idea generation thing,
it's giving suggestions to the user.
But how do we do this?
So for this example, how is anger like wood?
How do we come up with this list of 10 ways
that these two words might be related?
The first thing that we do is we get a long list
of properties of wood from ConceptNet.
So ConceptNet is an open source knowledge graph,
it's kind of a long standing AI project
to encode a bunch of common sense knowledge
in a big graph network.
And so you can query this, you can download it all,
it's open source, we use a specific set of queries,
and these are the kinds of things you get.
So they're pretty good,
they're like, varied as any long standing AI project.
Quality is touch and go,
but with the nature things it's pretty good.
So wood fence is in a property, burns when lit,
you can compost it, it feels rough,
we're like, okay these are definitely things about wood,
so good so far.
But then what we want to do is find which ones of these
might also be properties of anger.
And so we use this proxy of semantic similarities.
So the basic idea is we have all these properties of wood,
let's find the ones that are most related to anger.
And then hopefully they'll kind of make sense,
'cause if they're related to anger,
maybe they're a property of anger,
they make sense in the context of anger.
So to do this, we build off word embeddings,
which are a really popular technique
in natural language processing,
where you learn a vector representation for every word.
So you're embedding these words
in a really high dimensional space,
and words that are very semantically similar,
that mean similar things are nearby in this space.
So we use a algorithm called the Word Mover's Distance,
which will calculate a semantic similarity measure
for sets of words, so it's like a document-level algorithm.
And so this allows us to order these properties of wood
by how similar they are to anger.
And so you can see in the example it's doing pretty good,
so like burning when lit, making a fire, feeling rough.
These are like, whoa, sorry,
this is a very sensitive microphone.
These things seem to be pretty related to anger,
whereas something like fencing in a property
or building a boat are maybe less related to anger.
And then the final thing we do
is we want to select the top 10,
but we want to make sure that they're distinct.
So a lot things in ConceptNet are pretty similar.
So like, burning when lit, making a fire,
and burn in a fireplace are all pretty similar,
we don't want to return them all to the user.
So again we use a semantic similarity measure,
but this time we want to check that all of the properties
are above some certain thresholds,
so that they're all distinct enough
to be providing new ideas to the user.
So we take this algorithm and we embed it
in a web application
where the user can put in the word on the left,
pairs it with this nature word,
and it gives you this set of 10 ways
that they might be related.
There's also this expansion feature
where you can actually, for each suggestion,
click on the plus and it'll give you some extra ideas
about how these things might be related.
So it explicitly puts the two words
into two parallel sentences
to make the analogy of the metaphor really explicit,
and it comes up with these other words
that might be related.
So garden is for growing wildflowers,
whereas that jealousy is for growing sorrow.
Not going to go into exactly how we come up with these words
'cause that would take a long time,
but the details are in the paper,
and I'm happy to talk about it in the questions.
So I want to spend the rest of the time
talking about the evaluation,
'cause one of the really important things
about co-creativity is actually
how the person is collaborating with the tool.
So we have three studies,
the first one is an intrinsic study
where we just look at kind of how well
the algorithm is performing on its own,
and then the other two studies
are actually like user studies,
where we had people use this system
to do particular things to try to figure out
what the collaboration was looking like.
So in this first study we just wanted to ask like,
are these suggestions high quality?
Like, do they make sense, are they good metaphors?
Outside of the context of being used
for any particular thing.
So we had two expert writers, so these were people
who had a Masters in Fine Arts
and Creative Writing who were published.
They do this kind of work for a living,
annotate a bunch of metaphors.
Both metaphors that were generated by our system,
as well as these two other systems.
Thesaurus Rex is a system for making analogies
between things from morph and the AI community.
Intersecting Word Vectors is another algorithm morph
in the natural language processing community.
So we had metaphors from all of these
based on these same two seeds,
so each algorithm we would feed the same seed
and take out some of the results,
and then we shuffled them all up,
gave them to these expert writers,
and we asked them were the metaphors apt,
specific, and imageable?
So apt is like the ground floor of metaphors,
does this metaphor make sense?
Is this a connection that relates these two things
as opposed to just any kind of random word to connect them?
Specific has to do with if the connection in the metaphor
is something pretty specific about the thing
you're trying to describe.
So one really good metaphor algorithm
that would always be very apt
is if you always said the two things were entities,
so you would say oh, anger is like wood,
in that they're both entities.
So it will always be very apt,
and you get a very high score,
but it wouldn't be very interesting metaphor.
We actually wanted to say something specific about anger.
So that was our second quality
that we thought was really important.
And then the final one is imageable.
So oftentimes the idea of a metaphor
is to make something really visual,
to take something really abstract
and give us some image to associate with it.
So that was the third thing we asked about.
So you can see that we're doing pretty well,
so Thesaurus Rex has also got some really good scores,
but our system seems to be very specific.
So already we see okay, this seems to be pretty good,
it seems to be comparable to this other work
that people are doing with metaphor creation.
And then we ran a user study where the primary question
we want to ask was, well, do writers use these?
So isolated they might be interesting,
but given a task when people actually have
to do some amount of writing,
are they going to take these suggestions?
So we had 16 undergraduates write a metaphor
to show how these pairs of words were related,
we asked them to write a couple sentences.
So you know, gratitude is a stream,
could you explain in a couple sentences
how this metaphor would work or what it means?
So they each wrote for these six metaphors,
these were their prompts, and half the time,
for half their prompts they'd have access
to Metaphoria and the suggestions,
and then our control was them not having access to anything,
so they were just doing the task on their own.
So in the case where they had access to Metaphoria,
75% of the writers used the suggestions.
This was based on retrospective interviews
and also just looking at a screen capture
of what they were doing,
and kind of seeing what they were mousing over.
Of the four who didn't use the suggestions,
one said that they thought the suggestions were pretty good
and they liked them, but they really wanted
to do the task on their own,
like they thought of it as a challenge
that they didn't want help for.
So we're gonna get back to this towards the end.
And then the other three,
they did say that they thought
they didn't make that much sense,
or that they were too literal
and actually didn't think that they were that metaphorical.
So for the most part it's doing pretty good,
but obviously there's room for improvement.
But we were also curious
if we were encouraging divergent writing.
So one problem that might happen,
even if the suggestions are really good,
is that everyone ends up writing something really similar.
Like they take the same suggestion,
and they end up writing something
that's just rephrasing it,
and then that's not really a great creativity support tool.
So we had a look at if they were more divergent,
in that people wrote more different things from each other
when they had access to the suggestions,
versus when they were doing it on their own,
and in five of six cases they were more divergent.
And so you can see here,
one participant took this suggestion
and talked more about this idea
of how a stream can clean your body,
in a way that gratitude cleans your body,
whereas another participant took this
and talked about how it can sweep you away.
Right, and so you might ask, like,
how does that relate to bathing?
Like, were they really using this suggestion?
And so when we talked to them, they said,
well I was thinking about bathing,
and the actual physicality of being in a stream
and when I thought about that,
I thought about all these other impacts
of physically being there.
Like the stream could knock you over,
and so that's how they got to this idea of sweeping away.
And so this is kind of the ideal use case, right?
Which is the system inspired them to think about something
that they might not have thought about otherwise.
But this wasn't always the case.
So in this one prompt almost everyone
picked this suggestion,
or they said that they were inspired by this suggestion,
that jealousy can irritate the skin,
and almost everyone literally used the word irritate
and talked about how it gets stuck to you,
and stuck in your clothing.
Okay, so, for the last study we really wanted to see
how Metaphoria is used in a really open-ended writing task.
So most people who do creative writing,
they don't give themselves these little prompts,
and definitely not this specific metaphor prompt, right?
They come with their own ideas
of what they're trying to write, especially with poets.
It's often very expressive, they want to be writing
about the things that they are feeling.
So we wanted to see how poets use Metaphoria.
So we had three poets, we gave them a tutorial of the system
and then we said write a poem for the next 30 to 40 minutes.
This is an excerpt from each of them.
They each wrote at least double this.
And in this table the red are words
that they input into the system.
So these are things they searched in Metaphoria,
and then the greener things that they took
or were inspired by in the suggestions.
So you can already see that they're using it
in super different ways.
So this person right, on the left she would put in a word
that she was trying to describe,
or that she wanted to talk about,
went off and looked for a suggestion that she thought worked
and then she would move on.
This person in the middle, she put in this word, work,
and then she sat with those suggestions
for a really long time while she kept writing,
drawing from multiple different suggestions
for work as she was writing,
and then she put in some other words, swaying, and storm,
that she didn't find anything interesting,
and then she put in the word, she,
which we were like, why would you put in that word?
And she was like, well I was curious,
what's something that was really abstract
and pretty devoid of meaning, how it would work.
That actually came up with this suggestion.
She is used for currency and jewelry,
which she really loved,
and actually I was the experimenter for this,
and she audibly gasped and was like,
that's a really good line.
But this actually made her very uncomfortable,
so we'll talk about that in the next slide.
And then the final poet on the right was totally different.
She was more collecting, she put in lots and lots of words,
very associative, probably five times as many words
as ended up in the poem.
And she would then look for something
that was a really interesting rephrase of the suggestion,
write it and keep going.
It was really this more of a collage effort.
And so one question we have is what cognitive model
are people projecting onto this system?
Like, how did they view it?
And so the person on the far left
really saw it as a cognitive offloading tool,
she actually interviewed, talked about it,
like a calculator for words.
This person in the middle seemed to be treating the system
with a lot of agency,
and sometimes that agency threatened
her sense of ownership over the whole thing,
and then finally the person on the right saw it
as a casual creator,
so an algorithm that could explore an infinite space,
and her job was to navigate and find
interesting parts of the space.
So we really want to design tools that support writers,
and not threaten them.
So one thing that's kind of cool
about the ownership issues is it's a sense
that this system was doing really well.
So it's a nice problem to have,
but there's lots of ways we can mitigate it, right?
By modeling on existing relationships
and figuring out how those work
and how the system can align with them,
or leverage and expanding existing tools
that writers are already really comfortable with.
So this is a quick summary slide
about what we talked about today,
and there's a demo online if you want to go play with it.
It's pretty fun and you can get a feel for how it's working.
I'm happy to take any questions.
(audience applauds)
- [Coordinator] Any questions, yeah?
Yeah! (mumbles)
Oh, sorry!
- [Ramos] Hi thank you, Gosan Ramos, Microsoft Research.
I'm curious if you were able to observe other
sensitivity or concern about biases in the word embedding?
For example, if you can say,
how are rocks and strength alike?
Well, because they're both masculine, right?
Or how are doctors and such and such?
I wonder if that's something that was a concern for people,
or just out of sight, out of mind?
- That was definitely a concern for us
developing this system,
because bias in word embeddings is a really big problem
across all kinds of different axes.
In the kind of way in which we set it up
in our test bed it didn't come up that much,
we never really saw anything that seemed problematic.
We definitely had some filtering on the backend,
to make sure that no slurs or nothing
that was like really explicitly problematic
would ever be returned by the system,
but we didn't do anything to directly address
more subtle kinds of bias,
and it didn't come up in any of the studies,
but that's a really good point
and something that we were definitely thinking about.
- [Coordinator] We have time for one more question.
- [Man] I have a question.
So, should I speak into the mic?
I'll speak into the microphone.
So we recently saw that the open AI research lab
released examples of how AIs could produce text
that is eerily hard to distinguish from human text.
Have you in any way experimented with or thought
about how you might integrate these notions
of more creative writing,
or using metaphors in AI generated text?
- Yeah, that's a really great question.
Most of the creative writing support that uses AI
is based on this model of neural networks
that do text generation
often in a sentence completion paradigm,
where you're writing something
and it'll complete your sentence for you,
and it's definitely really interesting,
that there are, like, I know one computational poet
who's currently working with the GDPT algorithm
or whatever to generate poetry.
I think we were really interested in focusing on
what I want to call traditional,
or like intentioned typical writers who really come to it
with their own expression that they want to write about
and a lot of those kind of sentence finishing things
are really, they lack context not just for what's going on
in what someone's writing, which,
they're probably going to get better at,
but they lack the context
of what the writer is trying to write about.
Like how would it have any idea
where the writer is looking to go?
And so one of the main things,
reasons we designed this in this way
where the user would put in the word
that they wanted to write about,
is we were really concerned about making sure
that it was contextually relevant
for what the writer wanted to do.
So I definitely think there's space for those technologies,
but you have to be really careful
about what the writer's intentions are.
- [Coordinator] Yes, thank you once again.
(audience applauds)