So, let's go ahead and fit our collected data to an exponential distribution,
we'll adjust our analysis graph as such.
Now, as you recall from your introduction to statistics class,
the form of an exponential distribution is 1 over beta times e to
the power minus x over beta.
Thus, fitting an exponential curve to the earlier histogram,
means finding the parameter beta forming the PDF from which
the observed data is most likely to have been produced.
Now this gets into something called a maximum likelihood estimation.
This is something that another Udacity class goes into a lot of detail about,
we've put a link to that lecture in the instructor's notes.
But very briefly,
the maximum likelihood estimation strategy takes the following approach.
Let's suppose that I've got a collection of data points, data point d1,
d2, through dN.
The maximum likelihood estimation approach,
asks the question, what's the probability that I saw each of
these observed data points all at the same time.
I don't know the specific values of these probabilities, but I do know the form
of them, because I know that they conform to the exponential distribution.
The only thing I don't know is beta.
And, so I can write this expression as a function of beta.
When I maximize this,
I can then find the value of beta that gives me that maximum.
That's maximum likelihood estimation in a nutshell.
Now, what does this get us?
If it is the case that an exponential process is a good model for
describing tweeting behavior, then we can use it to predict the next tweet time.
So this comes actually from a simple expectation calculation.
Let's suppose that we already know the parameter beta.
Let's define that x is the random variable time until next tweet,
a.k.a inter-tweet time.
Now the PDF for x at time t is just the exponential evaluated at t.
The expected value of the distribution, it's just this integral here,
which should look very familiar, which is equal to the parameter data.
So let's think about what this means fora second.
If we have a fitted exponential distribution meaning we
know what the parameter data is, then it turns out we can use the parameter data
as a prediction of the number of seconds to wait until the next tweet.
Now, Let's go ahead and find the beta parameter using Python Notebook.
Let's go ahead and find the beta parameter using IPython Notebook.
So here we have again the same histogram describing intertweet times that
we saw earlier.
So let's first think about what we're trying to fix.
We want to find the beta parameter which produces a curve which
most looks like this.
In order to perform the exponential fit,
we're going to use a function from the scipy package called curve_fit.
Let's just go ahead and import that.
So curve_fit does the following.
Let's suppose that we got some values which we're trying to reproduce.
And we got a callable function here called fitFunc,
which you can pass an independent variable t and a parameter,
any number of parameters, but in this case we've just got 1, the beta parameter.
And as you can see,
all that does is just evaluate an exponential at the appropriate value.
You can see here that the number b is defined as the inverse of what it
would typically be defined at in an exponential distribution.
So instead of 1 over beta, it just appears as b.
So you can, anywhere you see b, you can think of that as 1 divided by beta.
So again, fitFunc is just a callable function which takes
the input independent variable time t and
the beta parameter represented here as its inverse just as the letter b.
And then evaluates the exponential function at the appropriate time.
So if you hand curve fit that callable function and
the values which are trying to reproduce here captured in the variable count.
Then the curved fit finds the value of beta to put into fit func that
best matches the count values, it does this using mean squared error loss.
Let's go ahead and evaluate that.
You can see here that curve_fit actually returns two parameters,
fitParams and fitCov.
The first one, fitParams is just the beta parameter that we're interested in,
and you can see that the actual beta parameter is
one over the parameter that we specified.
So, 1,451.5 seconds, but curve_fit also returns a covariance matrix.
The second thing that cur fit returns is a covariance matrix.
There's one entry in the covariance matrix for
each pair of parameters that we're trying to fit for.
So in our case, we have only single parameter, b, that we're trying to fit for,
and so the covariance matrix with only a single entry.
Although in general there is more.
So what does the matrix mean here in this case.
Well, what curve_fit actually did,
is it tried to the best value of b over many bootstrapped samples.
This number here which amounts to the variance of that
number describes how much b changed over those bootstrapped samples.
So, the higher this covariance value here means,
the more that be changed over the bootstrap samples.
That means that we should be less certain of what the parameter's value is.
As you can see here,
the covariance value is quite low, 1.04 times 10 to the negative ninth.
That translates to high confidence.
So, we got our beta value, 1451 and a half seconds.
Let's take a look at how well this fit does.
You can see, here, in blue, is the original intertweet times.
The exponential fit that we just did, is displayed in yellow.
So we got our beta value, 1451 and a half seconds.
Recall from earlier that if we are going to use an exponential fit as our model,
then we use the beta parameter as the number of seconds to
guess until the next time that the person tweets.
And so, the model that we would use in this case is that we're going to
guess 1451.5 seconds, until this person tweets next.
Let's take a look at a histogram of the signed error of
always making that guess.
So, here, we're looping through each value in timeUntilNext.
And evaluating the difference between that and our guess.
Now, let's look at a histogram, that's very interesting.
In the next few videos,
we'll take a deeper look into how well this guest turned out.
But, before we do that, let's just summarize where we are.