hey everyone welcome back features are
the way you represent your knowledge
about the world for the classifier and
today I'll walk you through techniques
you can use to represent your features
and utilities tensorflow provides to
help will use a data set from the US
Census as an example and the goal is to
predict if someone's income is greater
than $50,000 based on attributes like
their age and occupation the data set is
stored as a CSV file and previously
we've seen how to use the column values
directly as features but today we'll use
feature engineering to transform them
into a more useful representations as we
go I'll visualize what these
transformations do using a tool called
facets and you can find a link to it in
the description you'll also find
complete code to train a tensor flow
estimator on this dataset okay let's get
started
let's begin with a numeric attribute
like age and think about how we can use
it to predict income now if you think
about how age correlates with income our
first intuition is that as age increases
usually so does income and the simplest
way to represent this would just be to
take the raw numeric value and use that
as a feature here we're building a list
of features we use to Train the model
and each of these is stored as a feature
column this contains data about the
column from the CSV file and how to
represent it here we'll write a feature
that just uses the raw value of age and
this string corresponds to a column in
the CSV file now what can go wrong with
this approach well if we think more
closely about age we realize it's not in
a linear relationship with income the
curve might look something like this
it's flat for children that increases
during working age and decreases during
retirement a linear classifier for
example is unable to capture this
relationship that's because it learns a
single weight for each feature to make
it easier for the classifier one thing
we can do is bucket the feature and
bucketing transforms a numeric feature
into several categorical ones based on
the range it falls into and each of
these new features indicate whether a
person's age falls
to that range and now a linear model can
capture the relationship by learning
different weights for each bucket let's
see how this looks in facets
conveniently there's a live demo that
runs in the browser with our census data
pre-loaded and each individual from the
CSV is visualized as a dot colored by
income if you click on a dot you can see
stats about the person
now let's bucket by age and you can
adjust the number of buckets to make it
more or less granular how you choose the
number of buckets is up to you and
ideally you'd want to use your knowledge
of the problem to do this well in
tensorflow we can create a bucket eyes
feature by wrapping a numeric column
from the CSV and here we're specifying
the number and the ranges of the buckets
we'd like created once this is done we
can add the bucket eyes feature to the
list used to train our model now let's
see how to represent a categorical
feature and I'll use the education
column as an example because there are
only a few values the best way to
represent this is just to use the raw
value and here we'll create a feature
column that says education can be a
single value from this list of course
you could also read the values from a
file on disk rather than writing a
mountain code now using the raw value is
the right thing to do when there are
only a small number of possibilities
we'll cover the case where there are
thousands of possibilities in a moment
first let's take a look at feature
crossing feature crossing is a way to
create new features that are
combinations of existing ones and these
can be especially helpful to linear
classifiers which can't model
interactions between features here's
what this looks like in facets I'll take
our age buckets from before and cross
them with education under the hood you
can think of a true/false feature being
created for each bucket that tells the
classifier whether an individual falls
into that range now these buckets can be
informative and here we see some groups
are likely to have a high income and
others low in code using a feature cross
works the same way as before we'll cross
our age buckets with education and add
it to the list of features to use the
feature cross can generate many
possibilities quickly which is why
they're often represented under the hood
with
a hashed feature column is one way to
efficiently represent a categorical
feature with a large vocabulary more
importantly you can use these as a way
to make your data easier to work with
because they free you from having to
provide a vocabulary list in this
example will represent the occupation
column from our CSV file by using a hash
with 1000 possible values notice we
don't have to provide a vocabulary list
and to avoid collisions I've set the
hash size so it's larger than the number
of items in the vocabulary here's how
this works under the hood normally a
categorical feature is represented as a
one hot encoding that means there's one
bit for each possible value in the
vocabulary and we can create a lookup
because we know the vocabulary list in
advance now if we don't know the vocab
we can use a hash function to compute
the bit automatically the downside is
there could be collisions meaning
different items are mapped to the same
value hashes can also be used to limit
memory usage at the cost of adding some
noise to your training data if you have
a large vocabulary it can be memory
intensive to use that as input to a
neural network a hash column can be used
to limit the maximum number of
possibilities but I prefer them simply
as a tool to save you programming time
finally I'd like to mention embeddings
and these can be less intuitive than the
other techniques but they're a powerful
way to work with categorical data in a
deep learning setting you can think of
an embedding as a vector that represents
the meaning of a word and we can
visualize the data set of word
embeddings using the tensorflow
embedding projector and there's an
online demo you can find in the
description here we're looking at a data
set of 10,000 words each of which is
represented by a vector with many
dimensions projected down to 3d so we
can see them you can search for words in
the box to the right and if you
experiment a bit you'll find similar
words are often close together for
example all of the words in this cluster
are cities what's neat about embeddings
is that they're learned automatically in
the process of training at DNN and to
make that happen all you need to do is
write an embedding column here will
create an embedding for education with
ten dimensions now embeddings are
helpful if you have a categorical column
with a large vocabulary and you want to
compress the
representation so the classifier learns
general concepts rather than memorizing
the meaning of specific words for
example imagine if the census data had a
column called job title there are
thousands of different jobs and an
embedding could be used to help your
classifier learn that words like
programmer and software engineer often
mean the same thing okay I hope this was
a helpful intro and thinking about how
to represent your features is one of the
most important contributions you can
make to a machine learning experiment
feature columns are great because they
let you experiment with different
representations in code and make
advanced features like embeddings
accessible as a next step I'd recommend
you try the code in the description and
see if you can modify it for a problem
you care about thanks for watching
everyone
and I'll see you next time
[Music]
[Applause]