In this video I will be talking about building an automated trader
that is able to decide when to hedge a European call option contract
in the presence of transaction costs
using reinforcement learning.
Think of hedging as home insurance,
but in finance we use hedging to reduce risk from stock price movement.
The amount of stocks traded at each time step
is calculated using delta from the Black-Scholes formula.
Therefore,
if the call option is for 100 shares of MLB stock
and delta is 0.1
the trader needs to short 10 shares of MLB.
In a real-world scenario
where transaction costs exist,
it becomes critical while observing the market,
to know when to hedge during the life of the option
to have this tradeoff between trading costs
and hedging risk.
Let’s see how we apply Reinforcement Learning in hedging.
The agent is observing the financial market inputs
like the stock price, time to maturity, the option price, the stock holdings
and takes an action of whether to hedge or not.
If the agent hedges:
the new stock holdings are -51
calculated using delta.
Therefore,
the agent has sold 6 stocks resulting in $3 in transaction costs.
If the agent doesn’t hedge,
the holdings remain the same.
After one period,
the stock price moves to $99.4.
The agent observes a total profit and loss
resulting from the change of the stock price,
the change of the option price,
and transaction costs paid
and a reward related to the total profit and loss.
In Reinforcement Learning,
the agent will learn by trial and error
to maximize the cumulative reward
by choosing when to hedge during the life of the option.
The reinforcement learning setup
is composed of an agent
and an environment.
The environment sends a state to the agent,
which takes an action in response.
The agent will evaluate its last action
based on the reward it returned.
The loop keeps going
until the environment sends a terminal state,
say the maturity of the option,
which ends to episode.
After each episode,
the agent will learn to take actions
that maximize the cumulative reward.
MATLAB made it easier to set up the environment
and design the reinforcement learning components.
The reset function,
returns the initial state of the environment,
it is called at the beginning of each training episode.
The step function,
specifies how the environment advances to the next state
based on which action the agent takes.
The observations include:
the stock price, time to maturity, agent’s stock holdings...
The actions is whether to hedge or not.
The observations, actions,
reset and step functions
form the environment.
The agent consists of a policy
and a Reinforcement Learning algorithm.
The policy is a mapping function between observations and action.
It can be a neural network designed by specifying the layers,
activation functions
and neurons.
The Reinforcement Learning algorithm
continuously updates the policy parameters
and will find an optimal policy
that maximizes the cumulative reward.
We train the agent for an hour
and we can see that the agent has learned to maximize the cumulative reward over time.
As a result,
a trained agent
outperformed a trader who used delta hedging
and another
who decides not to hedge at all.
We simulate a random stock path
when all periods are hedged,
the loss is $120
the agent hedged 38 periods
and didn't trade for 12 times
and the loss is $55.
Thank you for watching.