CHRISTOPHER POTTS: Hello, everyone.
Welcome to Part 2 of our series of screencasts on distributed
word representations.
The focus of this screencast will be on matrix designs.
Let's start with the word-by-word design
that we concentrated on in Part 1.
So here again, we have a vocabulary along the rows.
That same vocabulary is repeated along the columns,
and the cell values captured a number of times
that each row word co-occurred with each column word
in some large collection of texts.
This matrix will have two properties
that I think make it noteworthy for developing
semantic representations.
The first, is it will be very dense.
And as we bring in more data from ever-larger corpora,
it will get denser and denser in virtue of the fact
that more words will tend to co-occur with more other words,
in this ever-larger collection of documents.
The second is that it kind of has the nice property
that its dimensionality will remain fixed,
even as we bring in more data.
As long as we decide on the vocabulary ahead of time,
all we'll be doing is incrementing individual cell
values.
And so we could bring in as much data as we want,
but without changing the fundamental design
of the object.
Both of those things are points of contrast
with another common design that you see in the literature,
especially in information retrieval,
and that is the word-by-document design.
For this design, again, I have words along the rows,
but my columns are now individual documents,
and the cell values capture the number of times
that each word occurs in each one of those documents.
As you can imagine, this is a very sparse matrix
in contrast to the word-by-word one
that we just looked at in virtue of the fact
that most words don't appear in most documents.
And we'll also have the property that as we
bring in more data in the form of more documents,
the shape of the matrix will change,
we'll be adding column dimensions
for each new document that we bring in to the space,
and that could really affect the kind of computations
that we can do.
The only thing that balances against the ever-increasing
size of this matrix is that because it is so sparse,
we might have some easy and efficient ways of storing
it efficiently, putting it on par with a much more compact
but dense word-by-word matrix that I showed you before.
Now, those are two very common designs
that you see in the literature, but I
want you to think creatively and align
your matrix design with whatever problem you're trying to solve.
So let me show you one that's really radically different.
This is what I've called the word by discourse context
matrix.
I derived this from the Switchboard Dialog Act Corpus,
which is the Switchboard corpus, where each dialog act has been
annotated by an expert annotator with the sort of dialog
act or speech act that was performed by that utterance.
What that allows us to do is collect a matrix
where the rows are, again, words,
but the columns are those individual labels
the annotators assigned.
I think this is a really interesting matrix.
I think if you appear even at this small fragment,
you can see some interesting information emerging.
So for example, "absolutely" occurs a lot
in acceptance dialog acts, whereas more hedged words
like "actually" and "anyway" are more
common in things like rejecting part of a previous utterance.
And I'm sure there are lots of other interesting patterns
in this matrix.
And of course, that's just a glimpse
of the many other design choices that you could make.
Again, think creatively.
You could have something like adjective by modified noun.
This would probably capture some very local, syntactic
information or collocational information.
We could generalize that a bit to word by syntactic context
to explicitly try to model how words
associated with specific syntactic structures.
It would be very different from our usual semantic goals
for this course.
Word by search query might be a design
that you use in information retrieval.
We don't even have to limit this to linguistic objects.
Word by person can capture the number of times
that each person purchased a specific set of products,
and then we could cluster people or products on that basis.
We could also mix linguistic and non-linguistic things,
so word by person might capture different usage
patterns for individual speakers and again,
allow us to do some kind of interesting clustering of words
or of people.
We can also break out of two dimensions.
We can have something like word by word by pattern
or verb by subject by object.
Many of the methods that we cover in this unit
are easily generalized to more than two dimensions,
so you could have that in mind.
And of course, as I said, think creatively, and think
in particular about how your matrix design is aligned
with whatever modeling goal you have
or whatever hypothesis you're pursuing.
Another connection that I want to make
is that even though this feels like a kind of modern idea
in NLP, vector representations of words or of objects,
are actually pervasive, not only throughout machine learning,
but also throughout science, right?
So think back to older modes of NLP
where we would write a lot of feature functions.
We'll be exploring such techniques
that can be quite powerful.
Even though they feel very different
from the distributional hypotheses
that we've been pursuing, in fact,
they also represent individual data points as vectors.
So for example, given the text like the movie was horrible,
I might reduce that with my feature functions
to a vector that looks like this.
And I might know as a human that 4 captures
the number of words, 0 captures the number of proper names,
and 1 over 4 captures the percentage of negative words
according to some sentiment lexicon, that's a human level
understanding of this.
In fact, those dimensions will acquire a meaning to the extent
that they assemble them into a vector space model
and the column-wise elements are compared with each other.
So even though the origins of the data
are very different, in fact, this
is just like vector representations
of words in the way we've been discussing it.
The same thing happens in experimental sciences
where you might have an experimental subject come in
and perform some act in the lab.
They do a complicated physical and human thing,
and you reduce it down to a couple of numbers
like a choice they made or a reaction time in a choice
and so forth.
We might model entire humans or entire organisms
with a vector of numbers representing
their physical characteristics, and perspectives, and outlooks
and so forth.
Again, we might know what these individual column dimensions
mean, but they acquire a meaning when we're doing modeling only
to the extent that they are embedded in a matrix
and can be compared to each other across the columns
and so forth.
There are many other examples of this, where essentially,
fundamentally, all of our representations
are vector representations.
So maybe the far-out idea for this unit
is just that we can gather interesting vector
representations without all of the hand
work that goes into the examples on the slide right now.
A final technical point question that you
should ask that's separate from your particular matrix
design, what is going to count as co-occurrence?
So I think there are at least two design
choices that are really important when
answering this question.
To illustrate them, let's use this small example.
So I have this text "from swerve of shore
to bend of bay, brings."
And imagine that our focus word at our particular point
of analysis is this token of the word "to."
And these indices here indicate going
left and right, the distance by counts
from that particular focus word.
The first question that you want to decide
is, what's your window of co-occurrences going to be?
So for example, if you set your window to 3,
then the things that are within 3 distance of your focus word
will co-occur with that word, and everything falling outside
of that window will not co-occur with that word,
according to your analysis.
If you make your window really big,
it might encompass the entire document.
If you make it very small, it might
encompass only a very local kind of collocational information.
So you can bet that that's going to be meaningful.
There's a separate choice that you
can make falling under the heading of scaling.
I think a default choice for scaling
is to just call it flat.
So what you're saying there is something
is going to co-occur once with your focus word
if it's in the window that you've specified,
and that would kind of equally weight all of the things that
are in the window.
You can also decide to scale, a common scaling pattern
would be 1 over n, where n is the distance by word
from your focus word.
That would have the effect that things
occur that occur close to the word of interest, co-occur
with it more than things that are at the edges, that
are near the end of the window.
Those choices are going to have really profound effects
on the kinds of representations that you develop.
Here are some generalizations I can offer.
Larger, flatter windows will capture
more semantic information.
As the window gets very large to encompass for example,
the entire document, you'll be capturing, essentially,
topical information.
In contrast, if you make your window very small and scaled,
you'll tend to capture more syntactic or collocational
information.
Independently of these choices, you
could decide how text boundaries are going to be involved.
So a text boundary at the level of a sentence or a paragraph
or a document or a corpus could be a hard boundary that's
independent of your window, or you
could decide that you're going to allow your window
to go across different notions of segment that you have.
That's really up to you.
And again, I think it will have major consequences
for downstream tasks involving the representations that you've
created.
To help you begin exploring this space,
the associated code released for this course, the associated
notebooks, provide you with four word-by-word matrices,
and they have a few things that allow you to do comparisons.
First, there are two matrices that
were developed from the Yelp Academic
Dataset, which is a lot of reviews
of products and services.
And there are two matrices that come from Gigaword,
which is news wire text.
So there's fundamentally a real difference
in the genre of text involved.
In addition, for each of those corpora,
we have two different designs--
window size of 5 and scaling of 1 over n, which ought to,
by my hypotheses, deliver a lot of kind
of collocational or syntactic information.
And a window size of 20 and scaling of flat,
a very large window, lots of things
co-occurring with lots of other things, that might be
a better basis for semantics.
And you have those two points of variation,
both for Yelp and for Gigaword, and I'm
hoping that kind of gives you a sense for how these design
choices affect the representations that you're
able to develop with methods that we're
going to cover in later parts of the screencast series.