(lighthearted music)
- Hi, this is Junjie Li from Microsoft Research Asia.
I'm a senior researcher SDE in the Machine Learning Group.
My recent work has been focused on game AI
with deep reinforcement learning.
This covers distributed system, Machine Learning
and AI capabilities infused gaming.
Mahjong is a very popular tile-based table game
with over 90 million players all over the world.
Even in the United States,
the National Mahjong League has over 500k members.
Mahjong has 136 tiles.
Players cannot see the hidden tiles of other players,
and cannot see the hidden tiles in the wall either.
The permutation of these hidden tiles
can lead to a huge number of hidden states.
If we compare games
according to the number of hidden states,
Mahjong is much more complicated than Go and Poker.
Moreover, different permutations of these hidden tiles
mean different levels of luck to the players,
and sometimes luck may dominate the result.
This is why Mahjong is so fascinating,
because it's closer to real-world applications.
For example, in the financial market,
the black swan events also bring uncertainty
and luck to some people,
and the other traders' strategies
are part of the private hidden information.
The huge state space, together with rich hidden information,
brings huge uncertainties
and makes it extremely difficult for AI
to discover the logical links
between the small set of public information
and the optimal playing strategy.
But once successful,
it could harness AI with new capabilities
that its predecessors never had.
Our team at Microsoft Research Asia
was very interested to explore this challenge.
We therefore decided to tackle imperfect-information games
using Mahjong AI.
Due to Mahjong's unique challenges,
previously developed algorithms for Go and Poker
would not work any longer, and we need new technologies.
In the past several years, we have developed
a novel system called Suphx.
It is based on deep reinforcement learning plus self-play,
and on top of that, global reward predictor, oracle guiding
and policy adaptation are the key technologies.
With reinforcement learning,
we can build a strong Mahjong AI
even without good Mahjong skills ourselves.
To evaluate Suphx's performance,
we got onboard Tenhou,
which is the most famous Mahjong platform in Japan.
After playing over 5000 games in the expert room,
which is the only room AI is allowed to play in,
Suphx successfully achieved 10 DAN,
the highest level in that room.
If we look at the stable ranking,
which is an index provided by Tenhou
to evaluate the performance of Mahjong players,
Suphx is much more stronger than top human players.
This is Suphx's decision flow in Mahjong.
We can see that the rules in Mahjong are complicated.
the playing order is not fixed,
and can be easily interrupted by chow/pong/Kong, or winning.
Besides the complicated rules,
Mahjong contains different kinds of information.
For example, tile set, tile sequence, integer information,
and categorical information.
How can we design the state representation
that our model can better understand
Mahjong complicated rules and state?
To make our model better understanding Mahjong state,
we design channel based features.
It encodes all observable information
into multiple channels.
Take the private hand tiles as an example,
it uses 4 channels to represent this private hand tiles.
Besides this private hand tiles,
we also do a short-term search
and summarized as look-ahead features.
This can help our model
better doing score calculation and planning.
We train the model with supervised learning
by expert data from Tenhou.
The accuracy of our model
significantly outperforms existing work on
Mahjong action prediction.
Suphx is first trained by supervised learning,
and then boosted through reinforcement learning.
We build a highly parallel
and scalable distributed RL system.
Which can support up to 70 GPUs for parallel training,
more than 2.5M of self-play games per day.
To ensure reproducibility, we also build scripts
for job submission, resource allocation,
model and log management.
Let's talk about the Global Reward Predictor.
In Mahjong, there are both round scores and game rewards.
Round score is determined by the winning hand of each round.
And a game reward is determined
by the final rank of accumulated round scores.
However, the game reward cannot differentiate
between well-played rounds and poorly played rounds.
And the round score is not well aligned
with the final game reward.
So, we design a reward predictor
to provide immediate reward for each round,
while optimizing for the long-term game reward.
The input of the reward predictor
includes the score information of current k rounds,
dealer and out of the game information.
The output is the final game reward.
The training signal for our reinforcement learning
is the marginal reward,
which is the difference of predicted reward
between the k and k-1 rounds.
Next, let me introduce the oracle guiding.
Oracle guiding uses an oracle agent that can see
the perfect information to self-play.
The perfect information includes private hand tiles
from other players and wall tiles.
It first uses all perfect information to self-play,
and gradually dropping out the normal features.
After dropping out to normal feature,
it continues training with one tenth of learning rate
and reject some state action pairs
if the importance ratio is larger than a threshold.
With oracle guiding,
we can achieve Suphx's performance much faster.
Last, let's talk about Policy Adaptation
In Mahjong, different permutations of the hidden tiles
actually correspond to totally different games.
What we do during offline training
is to find an AI model that can perform well
on many different games.
However "one size fits all" is always difficult.
On the other hand, during the online playing stage,
we are dealing with one specific game,
in which the permutation of the hidden tiles is fixed
although invisible.
In this case, the real state space shrinks a lot
and it should be easier to adapt our AI model
to fit this specific game.
Policy adaptation like Monte Carlo Tree Search,
cannot be directly applied to Mahjong
due to many rollout branches and irregular playing order.
In Suphx, we take a very different approach.
It adapts the offline-trained policy
to a given initial hand as follows:
First is simulation.
We sample many permutations of the hidden wall tiles.
Each permutation of the wall tiles
are the same as real initial hand tiles.
We use these wall tiles to self-play and get reward signals.
After simulations,
we use the trajectories and rewards to
fine-tune the policy.
And then we use the fine-tuned policy to play this round.
By doing so, we can expect that
the offline training policy can
better adapt to current score and initial hand tiles.
To demonstrate the value of each component in Suphx,
we trained several agents:
SL is the supervised learning model.
RL-basic uses a round score as the reward.
RL-1 uses the reward predictor as the reward.
RL-2 further uses oracle guiding
during Reinforcing Learning.
From the figure, we can see that
both global reward predictor and oracle guiding
can significantly improve the performance of our models.
In addition to testing,
the enhancement to offline reinforcement learning,
we also tested the online policy adaption.
The result shows that, about 66% of the initial hands tiles
can get improvement with policy adaption.
Finally, we provide more resource for Suphx.
First is our paper, which is currently public on arxiv.
Second is the game replays of Suphx.
Third is a search link that presents more articles
and videos for Suphx.
Finally, if you can read Japanese,
please check-out the book called "Suphx's Impact",
which is a best seller Mahjong tactic book,
written by one of Japan's highest-ranking players.
Thank you and feel free to contact us for any questions.
(lighthearted music)
- We have to actually re-think the entire AI stack.
- What is Reinforcement Learning?
Less look at an interactive panel.
- All our code is now open source.