[MUSIC PLAYING]
LAURENCE MORONEY: So Josh, welcome to Coffee
With A Googler.
JOSH DILLON: Thank you.
Thanks for having me.
LAURENCE MORONEY: Now you work on something
called the Distributions API, and that
brings about probabilistic programming.
What's that all about?
JOSH DILLON: Yeah, so this is something
that myself and my team were really excited about.
So I think a little history is in order.
So TensorFlow did a number of amazing things.
Maybe one of the most amazing is how
they made training deep neural nets easy-- or relatively easy.
LAURENCE MORONEY: It's a lot easier than it used to be.
JOSH DILLON: It's a lot easier than it used to be.
And this has resulted in a flurry of research
and exciting results and solving problems
that otherwise seemed impossible, or at least,
very difficult to solve.
LAURENCE MORONEY: Right.
JOSH DILLON: So before neural nets
were all the hotness, many machine learning folks,
including myself, sort of consider
ourselves probabilistic machine learning as, like,
a specialization.
And we want to be a part of the new hotness.
LAURENCE MORONEY: Who doesn't?
JOSH DILLON: Who doesn't?
LAURENCE MORONEY: I got the logo.
JOSH DILLON: So that's where Distributions enters the story.
We think that TensorFlow doesn't have
to be just a language for fitting neural nets.
It can be a language for statisticians.
It can be a language for doing probability first machine
learning, for fusing neural nets with sort of older ideas
and getting something that's greater
than the sum of its parts.
So Distributions is a tiny Lego piece
that tries to make that just a little bit easier.
LAURENCE MORONEY: Now I noticed you recently
published a paper on the Distributions API
explaining what it's all about.
Could you give us the brief summary?
JOSH DILLON: Yeah.
So we wrote just a white paper sort
of showing all of the cool things that
can be done when you mix TensorFlow intrinsics
and this Distributions API.
It's actually pretty simple.
But I think the real power comes from the breadth of the API,
in terms of the number of Distributions
that are included, as well as the way they sort of plug
and play with TensorFlow workflows.
LAURENCE MORONEY: Cool.
JOSH DILLON: Let's call it.
LAURENCE MORONEY: Now I'll be honest with you.
When I first started seeing something called
a Distributions API, I was like, hey cool, something
I can use to distribute my models.
But--
JOSH DILLON: Yeah, maybe we should have renamed it.
LAURENCE MORONEY: Shall we start refactoring?
JOSH DILLON: If you want to refactor, by all means.
LAURENCE MORONEY: It's going to be a bit of work.
But it's not about distributing apps, right?
JOSH DILLON: It's not about distributing apps, no.
It's about encoding properties of random variables
in an easy to use framework.
So a random variable is a way to represent uncertainty,
let's say.
And there are several useful properties that it has.
You may ask, what's the average under this random variable?
You may ask, what's the variance under this random variable?
So distribution just collects these properties.
That's not super exciting.
I think what's slightly more exciting
is the way we organize the library automatically takes
advantage of modern hardware.
LAURENCE MORONEY: OK.
JOSH DILLON: So we have something
we call "batch semantics," which is a TensorFlow idea.
And it means that you can parameterize
one distribution in many different ways
and take advantage of SIMD instructions.
LAURENCE MORONEY: Right.
So I noticed in your paper, you spoke a little bit
about Distributions.
But you also mentioned these things called "bijectors."
JOSH DILLON: Right.
LAURENCE MORONEY: That's a fabulous name,
but what exactly is a bijector?
JOSH DILLON: So we actually named it a bijector so,
that way, if you Google searched for it,
you would only find that word.
A bijector is a very simple idea.
It is a function of a tensor.
And where this fits into the Distribution
API is, presumably, that tensor is itself
a sample from some distribution.
So a bijector is a way of transforming one distribution
into another distribution.
LAURENCE MORONEY: OK.
JOSH DILLON: That's useful, because if we
didn't have that concept, we have like 60 distributions now.
We may have 600 distributions if we
tried to implement all transformations
of all distributions.
There's something called a lognormal, which
is E to the normal distribution, let's say.
And we don't actually implement that as a first class citizen.
To achieve that, you would transform a Gaussian
distribution with the exponential bijector.
And now you have lognormal.
LAURENCE MORONEY: I see.
So it's a way that you can combine and chop and change
distributions.
JOSH DILLON: Exactly.
Yeah.
Exactly.
The idea is, distributions are sources of stochasticity
or randomness, and bijectors are deterministic transformations
of those sources of randomness.
So there is a kind of dichotomy there.
LAURENCE MORONEY: Interesting.
So just taking a step back for a second and just think,
why is probabilistic programming, in your view--
why is it necessary, and what does it
bring to the table for developers?
JOSH DILLON: Yeah that's an exciting question.
So I like to think of probabilistic programming
as being a formalism for reasoning under uncertainty.
LAURENCE MORONEY: OK.
JOSH DILLON: I like that definition,
because I can't think of something that doesn't
fit into that definition.
LAURENCE MORONEY: Those are the best definitions.
JOSH DILLON: Those are the best definitions.
Science is you make a hypothesis,
you verify that hypothesis with some empirical study.
But we might imagine that the third time
you ran your experiment, you got a totally different result.
It doesn't mean your hypothesis is wrong.
It just means that, guess what, nature has randomness.
So having a way to make it easy to specify uncertainty
and to perform inference and reasoning on it
means that you can sort of tease out the signal from the noise.
And doing so in a formal way keeps you honest.
It means that you can't claim "I have a cure for cancer,"
when in fact, it works only one out of a million.
LAURENCE MORONEY: Got it.
Got it.
And also for safety reasons, right, if you're
doing something like--
JOSH DILLON: Absolutely.
LAURENCE MORONEY: This is a green light.
JOSH DILLON: Absolutely.
Suppose you have a machine learning
algorithm trying to detect if it's a green light.
If you were only 2% sure that it's a green light,
you probably should stop.
LAURENCE MORONEY: Exactly, or don't go.
JOSH DILLON: Don't go, yeah.
LAURENCE MORONEY: Right.
But then if it's 97% or if it's 98%--
JOSH DILLON: So that's the question.
LAURENCE MORONEY: --that's when you start.
JOSH DILLON: The nice thing about having
some notion of uncertainty in your predictive system
is you can weigh the opportunity cost of action or inaction.
LAURENCE MORONEY: And it might seem like a lot of this
is very theoretical and very mathematical,
but it's also very concrete, right.
As soon as you build your first neural network,
as soon as you build a neuron, that's
entirely based on probability.
JOSH DILLON: Yeah, it's interesting you
say it's theoretical, because machine learning likes to boil
things down to accuracy.
I have some algorithm, and it achieves 97% accuracy
on detecting handwritten digits.
That's not how the real world actually works.
The whole point of having a machine-learned model
is you're going to make some choice.
And so I think it's immensely practical to have a system that
tells you, look, this is the choice I think you should make,
but this is how much you should trust me.
That seems like actually not theoretical.
That seems like what we all really want or maybe even need.
And so if you take a probabilistic first viewpoint,
you can start to design systems that have this capability.
LAURENCE MORONEY: Right, and to be
able to get that probability, you
need like things like loss functions,
and you plug a distribution into that.
So by having tons of distributions,
it gives you the ability to chop and change
to see what works best for you.
JOSH DILLON: And not just tons of distributions, but ways
to change them while preserving the computational properties.
LAURENCE MORONEY: And that's the bijectors.
JOSH DILLON: And that's the bijector, exactly.
LAURENCE MORONEY: Cool.
So say I'm getting started in machine learning,
and even getting started in machine learning
is a hard enough thing.
But what advice would you give me
to be able to optimize what I'm doing?
Any good pointers that you would have from your background?
JOSH DILLON: So I do not have neural network as a background.
But I like to think of neural nets--
although this may be heretical-- as very fancy forms
of regression.
Regression is a relatively old--
actually, an old idea in statistics,
and relatively old in machine learning,
in which you try to predict a response, given some evidence.
That's what a neural net does, too.
So I think to get started, we have this thing called
trainable distributions, which basically take a tensor that
is your evidence, and it gives you back a distribution
from which you could plug in, say, minus distribution dot log
prob of your labels as a loss into an optimizer.
So the examples we have there show just how easy it is
to do regression in TensorFlow.
And the nice part about that is, as you add more
sophistication-- different layers, in TensorFlow parlance,
with different activations, nonlinear functions
of the output of those layers--
now you can get state of the art machine learning
sophistication.
LAURENCE MORONEY: Cool, cool.
And obviously, if I'm going beyond beginning,
and I want to start getting into Distributions what--
I mean, I guess read your paper would be good advice.
JOSH DILLON: Yeah, so, the white paper
has a lot of examples that show just the power of this really
simple idea.
And those are my favorite kinds of ideas,
because I actually get them.
LAURENCE MORONEY: That's always the best kind, right?
JOSH DILLON: That's the best kind of idea.
LAURENCE MORONEY: No point having an idea
that nobody understands.
JOSH DILLON: Especially me.
LAURENCE MORONEY: Well, thank you so much, Josh.
This has been so much fun, and it's really, really digging
down into stuff that I formerly really didn't understand.
But having read your paper and having
played with some of this stuff, I
can really begin to see where this is going.
JOSH DILLON: I'm happy to hear that.
LAURENCE MORONEY: So thanks very much, Josh.
That was fascinating.
And thank you, too, for watching this episode of Coffee
With A Googler.
If you have any questions for me,
or if you have any questions for Josh,
please leave them in the comments below.
And don't forget to hit that subscribe button.
Oh, and by the way, be sure to check out our brand
new TensorFlow channel on YouTube.
Thank you.