Now if you recall,
when we looked at our histogram of signed error, using our guess of 1,451 and
a half seconds, we saw that most of the histogram occurs before zero.
This suggests that if we adjusted our model just a little bit
by perhaps adding an offset, we might be able to improve our accuracy.
So let's think about that problem a little bit.
So our initial guess, of the exponential fit, says that our initial for
the PDF of inter-tweet times has this form.
One over beta, times E to the power of -t over beta.
And then we found the value of beta,
which is most likely to have produced the data that we saw.
Now, our insight that we might be able to add an offset,
means that we're suggesting the model.
Might look, something like this instead, and then,
in addition to finding a value of beta, we should also look for a value of C,
that then makes it most likely to have produced the data that we saw.
We can actually generalize this,
to a slightly larger class of exponential models.
In addition to a constant c, we can also think about another constant, A.
Where A and C are constants.
So what does this mean, exactly?
Well, we're still trying to solve the same problem.
We have a collection of data points,
describing a histogram and we're trying to find a curve that best fits it.
So previously, we just used beta.
And we used the knob of the beta parameter,
until we found a PDF that best fit the curve.
Now, in addition to using the knob of beta,
we're going to attempt to use two additional knobs, A and C.
And the thought is, can we use the three of these together to get a fit for
the PDF, which is more accurate?
let's find out.
So we're going to use precisely the same method in i pi fun
notebook to perform a fit.
Specifically we're again going to use psy pi's curved fit in order to
find the best parameters.
This time however.
Fitfunc is gong to have a couple of extra parameters.
So in addition to just having the constant B,
now we're reasoning about the constant A and the offset C.
And so fitfunc let's you pass to it in addition to
the independent parameter time, the parameter which we had before, B.
But as well as that A and C.
Other than that, it's exactly the same.
We're setting to curve_fit, we've got this callable function
with independent value t and a bunch of parameters, and we're going to feed it,
these values that I want you to fit to.
Now given that, find me the parameters which best fits those values and give me
the covariance matrix which describes how certain we are about those values.
Let's go ahead and do that.
So, here are the parameters that it came up with for A, B and C.
So curve fit says the value for a is 3.34 times 10 to the negative 1, or 0.334.
We have a new B value, of 2.172 times 10 to the negative 3.
Which means, in order to get our beta value,
we just find the reciprocal of this and then the offset value C, is quite small.
3.185 times 10 to the negative 6.
Pretty much negligible.
So, what does that mean?
It means that the adjusted model, looks something like this.
Recall that the generalized form of the adjusted exponential model.
Is a times 1 over beta, times E to the power of minus T over beta,
plus offset C.
Except that now, we have actual values for A, beta and C.
So, let's just plug those in.
So, the result of our new fit, looks like this.
It says that, our new model is approximately 3.34 x 10 to the negative 1,
times 2.172 times 10 to the negative 3, times E to the power of
minus 2.172 times 10 to the minus 3 times t, plus an offset that is nearly zero.
So recall that once we've got our model,
in order to go from our model to our guess, of the number of seconds to wait.
We take the expected value of this.
So what we want is expected value of inter-tweet time,
given that the above is our model.
And so if we are to evaluate that, that comes to about 153.83 seconds.
So that's to be compared against our initial value of 1,451 and
a half seconds, quite smaller.
Now let's see how well this new guess of 153.83 seconds works out.
So first thing's first, let's evaluate our adjusted histogram.
So just as before, the blue indicates the original intertweet times.
And the yellow, is the initial exponential fit that we did.
To that graph, we've added the red,
which is the generalized exponential fit, using the A and C parameters.
You can see that it's a little bit of a tighter fit, just sort of qualitatively.
Now, let's form a histogram of the signed error for
our updated guess of 153.83 seconds.
So again here, we're collecting signed differences and absolute values of
differences, just now for our updated guess instead of the original guess.
So we see that the mean of the absolute error, has decreased to 1401 seconds
from 4446.8 seconds, as it was before.
So it's gone down quite a bit.
In addition to that, we've gone from being within about 1,437 seconds 75%
of the time to being within 676 seconds 75% of the time,
of the actual inter-tweet time.
So that's saying that with our adjusted model were good to
within about 11 minutes, 75% of the time, which is actually not terrible, given
that we still don't really know anything about the dynamics of the problem.
So, before we proceed, let's just summarize where we are.
So we decided to build a regression estimator to estimate the number of
seconds since the last tweet.
We decided to model this regression estimator, using an exponential fit.
Our initial fit.
Suggested always guessing 1451.5 seconds.
After forming some confidence bands on this value of beta and taking a look
at the absolute and assigned error, we decided we want to take a step back and
look at a slightly more generalized version of that model.
With a greater number of parameters.
After fitting the new generalized model, we came to a new guess, of 153.83
seconds, and discovered that it performed a bit better than the initial fit.
Now, in the next few videos, we're going to dig deeper into this data, and
the dynamics of the problem, as we do a more formal investigation.