In this video, I'll talk about how to calculate the steady-state distribution of an ergodic
Markov chain and use it for decision making.
In a previous video, we talked about how to calculate n-step transition probabilities.
We can repeatedly apply the formula and see what will happen when we increase the value
of n.
We introduced the toothpaste brand example in a previous video.
Suppose two toothpaste brands are available on the market.
This is the original 1-step transition probability matrix.
When a person just purchased brand 1, there is a 90% chance that he or she will purchase
brand 1 next time.
There is a 10% chance that he or she will purchase brand 2 next time.
When a person just purchased brand 2, there is an 80% chance that he or she will purchase
brand 2 next time.
There is a 20% chance that he or she will purchase brand 1 next time.
For 2-step transitions, this is the transition matrix.
P to the power of 2.
Let's continue to increase the value of n.
We find that after n is greater than or equal to 20, there is little change in the transition
probabilities.
It means, after a long time, the probability that a person’s next toothpaste purchase
would be brand 1 is 67%, and the probability that a person’s next toothpaste purchase
would be brand 2 with a probability 33%.
These probabilities do not depend on whether the person initially purchased brand 1 or
brand 2.
These probabilities are called the steady-state probabilities, which can be used to describe
the long-term behavior of a Markov chain.
The problem reaches the steady state after around 20 transitions.
In general, if the one-step transition matrix P contains very few entries that are near
0 or 1, the steady state is usually reached very quickly.
The behavior of a Markov chain before reaching the steady state is called transient (or short-run)
behavior.
Now, let me introduce the steady-state theorem.
Given enough time, every ergodic Markov chain will reach the steady state.
Let P be the transition matrix for an s-state ergodic Markov chain.
Then there exists a vector π from π1 through πs, which is called the steady-state distribution,
or equilibrium distribution, such that when n is approaching positive infinity, the n-step
transition matrix is equal to a matrix with this π vector repeated for s rows, and the
n-step transition probability from i to j is equal to the jth element in this π vector.
The sum of the elements in the steady-state distribution is 1.
New state distribution at step n+1 is equal to the old state distribution at step n time
the one-step transition P. When n approaches positive infinity, the steady-state
distribution is equal to the steady-state distribution times the one-step transition
matrix P. Let's find the steady-state distribution for
the toothpaste example.
Apply the steady-state distribution property.
The steady-state distribution is equal to the steady-state distribution times the one-step
transition matrix.
From this equation, we can get two equations.
Plus the third equation, the sum of the steady-state distribution elements is 1.
After we solve this system of equations, we find that the steady-state distribution is
π1=2/3, and π2 =1/3.
The steady-state probabilities can be used in the decision making process.
Let's revisit the toothpaste brand example.
Assume there are 100 million customers.
Each customer purchases one tube of toothpaste every month.
One tube of toothpaste costs $1 to produce and is sold for $2, so the profit is $1.
For $50 million/year, an advertising firm guarantees to decrease the fraction of brand
1 customers who switch to brand 2 after a purchase from 10% to 5%.
Should the company that makes brand 1 toothpaste hire the firm?
Let's calculate the steady-state profits before and after advertising.
Before advertising, this is the original transition matrix, and this is the original steady-state
distribution.
That means brand 1 will have a market share of 2/3 and brand 2 will have a market share
of 1/3.
The profit for the toothpaste company that makes brand 1 is equal to 2/3 times 100 million
customers times 1 tube per month times 12 months times the selling price $2 minus the
cost $1.
It is equal to $800 million per year.
The advertising firm is offering to change the P matrix to this new one, where the first
row is changed to 0.95 and 0.05.
Following the procedure for calculating the steady-state distribution, we find that the
new steady state is 0.8 and 0.2.
That means brand 1 will have a market share of 80% and brand 2 will have a market share
of 20%.
The profit for the toothpaste company that makes brand 1 is equal to 0.8 times 100 million
customers times 1 tube per month times 12 months times the selling price $2 minus the
cost $1 minus the advertising cost $50 million.
It is equal to $910 million per year.
910 is greater than 800.
So it is totally worth it to hire the advertising firm.
Okay, that is how to calculate the steady-state distribution of an ergodic Markov chain and
use it for decision making.
Thanks for watching.