Hello everyone. I'm Omid Mohaddesi, and in this presentation, I will talk about our paper "To
Trust or to Stockpile: Modeling Human-Simulation Interaction in Supply Chain Shortages", on behalf
of our research team at Northeastern University Simulation games are used as a research
environment to study human decisions, methods such as gaming simulation and participatory
simulation, and game-based simulation environments such as Gamettes have shown
great promise in studying human behavior. Researchers usually use these environments to
validate and improve the underlying simulations by observing or modeling human behavior.
However, a key challenge in modeling human behavior in these environments
is the system’s dynamic nature. As a result, there is a trade-off,
where the state of the system, affects humans behavior, and this state can change due to
human interaction with the simulation. We also usually use these environments
to generate and test hypotheses and in doing that we create manipulations that may have
some unexpected effects on human behavior. So, when we combine these manipulations with
a dynamic system teasing out their effect becomes more complicated.
Our goal is to extend existing research in behavioral modeling within game-based
simulation environments by characterizing the human-simulation interaction.
We do this in three steps. First, by characterizing the simulation.
Then characterizing the behavior. And finally by characterizing the interaction, while
taking experimental manipulations into account. To achieve this goal, we use a gamette environment
as described in our previous work to put human participants into the loop of a simulation that
replicates a supply chain environment we considered a supply chain, where
players play the role of a wholesaler and need to order from a manufacturer and allocate their
inventory to downstream health centers with a goal to minimize costs and maximize profit.
In addition, at some point in the game, a disruption happens that limits their supply.
After a tutorial, players start making decisions on week 21 of the simulation
and continue playing for 35 game weeks. On week 28, players are notified about a disruption.
Moving forward, we had two hypotheses. First we hypothesized that if we provide
order suggestions to players they deviate more from these suggestions when facing
a disruption compared to a normal condition. In addition, we hypothesized that if we share
supply chain information with players it will reduce their deviations from those suggestions.
To test the second hypothesis we considered a manipulation where, for some players we provided
information on their supplier inventory. We asked 121 online users to play the game
and randomly assigned each one to either an information sharing or no
information sharing condition. For characterizing the system,
we used principal component analysis to reduce the dimensionality of the simulation.
Here we consider the state of the system for player’s role which is the wholesaler,
so we considered inventory, demand, backlog, receive shipments, and on-order.
We found that two principal components explain 77% of the variance in the
data for these supply chain variables. Here you see the scores from these two
principal components. Each point here represents the state of the system for each player and in
a certain week. By performing hierarchical clustering, we found three system states.
These states represent situations where the system is either stable,
or it is in a supply disruptions state where players are unable to satisfy their demand,
or it is recovering from disruption and players receive large shipments after a shortage period.
By considering the states that most players experience every week we divided the gameplay
into five phases. We will use these five phases to characterize the interaction
of players with the simulation. To characterize the behavior,
we used a Hidden Markov Model or HMM in short. An HMM describes a Markov process where
a set of observed outputs can be explained by a set of hidden states.
What we're trying to do here is to infer this set of hidden states.
We considered our observed outputs for the HMM to be the players’ deviations from order suggestions,
and we tried to find response modes that explain the amount of deviation.
Using HMM, we found two modes of negative adjustment, a control mode,
and five modes of positive adjustment, where, in each response mode, players show different
levels of deviation from order suggestions. On the left, you see a snapshot of ordering
scene that shows order suggestion, and the plot on the right shows the inferred
response modes from the HMM, and how each one corresponds to a specific range of deviation.
Now imagine that each player can have a different response mode at each time point during the game.
So, we can consider the sequences of players’ response modes and compare
these sequences across all players. By doing this, we found three types of
players who were showing different patterns in their decision-making. We called them hoarders,
reactors, and followers. Hoarders tend to stockpile more frequently and are more
frequently in positive adjustment modes. Reactors follow order suggestions before
the disruption but react to the news about disruption and transit to other response modes.
And finally, followers almost always follow the order suggestions.
Now that we have characterized both the system and the behavioral responses from players,
we can look at the behavior over the system phases that we described earlier.
We can do this by counting the frequencies of response modes across hoarders, reactors,
and followers, and over the phases of the system,
and separately for players who received information sharing and those who didn’t.
We can see that hoarders and reactors deviate more from order suggestions when facing the disruption,
especially in phase two and during the shortage in phase three. This supports our first hypothesis.
And while hoarders tend to stockpile more than other players,
those who receive the information sharing deviated less by being in mode of P1 more frequently.
This shows that information sharing reduces order deviations, which supports our second hypothesis.
An interesting but unexpected finding here is that
information sharing leads hoarders to hoard even more after the shortage period is over.
Most simulations have a high dimensional space and can generate different dynamics,
especially when they are a model of a complex system such as a supply chain.
Games are complex systems too and can generate different experiences that result in different
behaviors. Including manipulations may also result in unexpected interactions. These are these are
all reasons that make game-based simulations complex environments for behavioral modeling.
Through characterization of the system, the behavior, and the interaction, we try to shift
the focus to human-simulation interaction as a research area within HCI and show how these
characterizations help for making sense of human behavior in such complex environments.
Future research can extend our approach by extending the system and behavior
characterizations. For characterizing the system, we can differentiate between dynamics
that are driven by the simulation or are the results of human behavior.
For characterizing the behavior, we analyzed observable response
by considering in-game decisions, but future research can study non-observable responses,
such as mental models or belief updates. And with that, we would like to thank the
anonymous reviewers for their comments and CHI ’22 organizers for the opportunity to present
this work. Please check out our website for more information and feel free to reach out via email
or Twitter if you have any questions. Thanks for listening!