Hi everyone! Our work is on rethinking probabilistic models of human behavior in the context of
robotic applications.
Whether it’s quadrotors , autonomous vehicles, or manipulation, when robots interact with
humans , they need models of human behavior for both inferring human goals and preferences,
and predicting what people will do. In such interactions, robots typically assume
that human goals or preferences are captured via a reward function R , defined on trajectories
\xi via features of the trajectory \phi – for example, things like distances to obstacles
or efficiency. This reward is typically parametrized by \theta,
which are weights encoding linear tradeoffs between features or even weights in a neural
network. In the Boltzmann model, this reward induces
a probability distribution over trajectories, proportional to their exponentiated reward.
This models that people aren’t perfectly optimal, but the higher cumulative reward
a trajectory has the exponentially more likely it is.
Boltzmann rationality has seen great successes in all previously mentioned domains, in both
inference and prediction. So although what we focus on is very particular,
this model’s widespread usage in robotics motivates how fundamental to the science of
interaction it is to get it right.
But, where does it come from? Let’s look back at Boltzmann’s origins: its precursor,
the Luce choice axiom. Imagine that you have to choose how to commute:
you could either drive or take the train. In your mind, each option has a value. Perhaps
you prefer taking the train, so its value is larger than the car’s.
Luce’s axiom simply states that each option will be chosen with probability proportional
to its value. If we further assume that each option has
some underlying reward, the Luce-Shepard choice rule assigns it a probability exponentially
increasing with its reward.
Both formulations are simple and intuitive; however, they suffer from a subtle but fundamental
shortcoming.
Suppose you don’t have a preference for driving or taking the train; Luce would predict
equal probability for both options. Now, if you have to choose between two cars
that differ only in their color, and the train, Luce predicts equal probabilities once again.
Lastly, imagine 100 of the same car and one train. The more cars are available, the less
likelihood is assigned to picking the train. That seems wrong.
This simple example showcases the fundamental issue with these models, called the duplicates
problem: should the addition of these duplicate cars reduce the other option’s probability
that much?
Now, what does this have to do with robotics? Do we have duplicates in robotic tasks? It
turns out, not quite, but what we have is a lot worse.
Let’s have a look at a task where we might want to use such a model to infer the human’s
preferences.
Imagine we have a person demonstrating a trajectory to the robot for how they prefer it to do
a manipulation task, like carrying a cup. To apply the Boltzmann model, we assume that
out of the space of all trajectories , the person prefers some over others and that influences
their choice – here is a simplified representation of that on the bottom.
Boltzmann, as an extension of Luce-Shepard to continuous spaces, values each trajectory
in proportion to its exponentiated reward against the alternatives.
In continuous trajectory spaces, which are typical to robotics problems, we don’t have
duplicate options, but we have a continuum of options with varying degrees of similarity
to each other. Nothing is a duplicate, but everything is more or less similar to everything
else! This is why we think that Boltzmann is a naïve generalization of the Luce axiom
to continuous spaces. Our insight is that we need to rethink how
to generalize the Luce axiom to trajectory spaces, and account for how similarity in
trajectories should influence their probability.
With Boltzmann , the likelihood of choosing a trajectory is proportional to its exponentiated
reward against the alternatives, with no regards to how similar to one another these alternatives
are. We propose that we limit errors due to similar
selections with our LESS formula. We introduce a similarity score between features
of trajectories. Thus, each exponentiated reward is divided by the total similarity
between the trajectory and its alternatives.
Let’s now tease out the difference LESS makes in a concrete example.
In this gridworld, the person is trying to teach a robot to navigate from the orange
circle to the cross by providing 10 demonstrations \xi_H. The human’s reward trades off between
two features, the distances to the two objects. In this situation, the person wants the robot
to stay close to both objects. In this second grid, however, the person prefers
the robot to stay away from them. For both grids, we can plot the feature density
for the distance from the top left object. When we additionally plot the human samples,
we see that for the first grid the demonstrations fall in a dense region , because there are
many trajectories between the two objects that have similar feature values. Meanwhile,
for the second grid, we see that the trajectories fall into sparser areas of the feature density
because there are fewer trajectories that are farthest away from the object.
So, how does this lead to Boltzmann being wrong?
To see how Boltzmann suffers from the similarity problem, let’s attempt to perform inference
with these demonstrations. Here is the first gridworld, with demonstrations
falling in a high-density area. When we perform Boltzmann inference to determine which theta
produced these demonstrations , we see that the posterior probability is high entropy:
there is no clear \theta being inferred. In this example, Boltzmann inference under-learns
because there are many high-reward alternatives, which lowers the probability of the demonstrations.
In the second grid with the 10 trajectories lying in a low-density area, we see the opposite
effect where Boltzmann inference is extremely confident that the true objective is to stay
far away from both obstacles, although some of the trajectories clearly stay close to
the second obstacle. It turns out that in this example Boltzmann inference over-learns,
because observing such rare demonstrations seems like an extraordinary occurrence. Even
worse, the Boltzmann inference learns much more from the 6 trajectories that lie in the
low-density area, essentially ignoring the other 4 trajectories that lie in slightly
denser regions. In summary, Boltzmann, by being sensitive to feature densities, can
under- or over-learn.
Now, the way LESS solves this problem is by flattening the curve, regulating the effect
that similarity between trajectories has on their probabilities of being chosen.
The result is that in the first grid, LESS confidently identifies the ground truth because
it doesn’t under-learn, whereas in the second grid LESS doesn’t overlearn and recognizes
that the trajectories give evidence for two objectives.
Armed with the intuition behind our model, we put it to test in a mechanical turk experiment
with real people, gridworld simulations, and a real world high-dimensional scenario with
a robotic manipulator and real human demonstrations.
First we checked if people choose trajectories based on LESS or on Boltzmann.
We ran a Mechanical Turk study with 80 people, split into two groups.
The task was to teach a robot how to navigate while using the least number of tiles and
avoiding the obstacle. In the control group, the participants saw
two equally good trajectories, Left and Right, and one bad one. This gave us control proportions
of Left versus Right trajectories being chosen. In the experimental group, we added two Right
trajectories, very similar to the original one.
Here we show the predicted proportion of Left versus Right choices using Boltzmann and LESS,
respectively, as well as the observed proportion in the experimental trial, with a dotted line.
Our main observation in this experiment is that the Boltzmann proportion prediction is
significantly different from the observed, whereas LESS predictions provide a tighter
equivalence bound to the observed.
Now, if people do act more like LESS than like Boltzmann, how much does this impact
the robot’s reward inference when using this model?
We used the same gridworld described previously, and we simulated sampled human demonstrations
according to 8 different preferences \theta. We see that when the person behaves according
to LESS, inference quality is significantly higher with LESS than with Boltzmann.
This plot suggests that if people are LESS, then it’s important to model them as LESS.
We lastly performed an experiment with a real world high-dimensional scenario with a 7 degrees-of-freedom
robotic manipulator and real human demonstrations, where one cannot have access to the full trajectory
space, nor the ground truth reward. Returning to our models, we see that in the
real world with infinite-dimensional trajectory spaces it is intractable to compute the integrals
in the denominators. Thus, in practice, we need to sample the trajectory
space to transform the integrals into finite sums.
We’ve previously teased that Boltzmann inference performance is highly dependent on the structure
of the environment, specifically on the feature space density induced by all possible trajectories.
Now, we hypothesize that different sampling sets will result in vastly varying inference
results for Boltzmann, whereas LESS will be more robust.
To this end, we collected demonstrations from 12 users for 3 robot manipulation tasks: carrying
the cup while keeping it close to the table, away from the laptop, and away from the human’s
body. To test our hypothesis that LESS produces
more robust inference than Boltzmann, we sampled 10 different trajectory sets to emulate the
continuous and infinite trajectory space. This plot showcases that performing inference
with LESS across multiple trajectory sets in a real-world setting results indeed in
higher robustness than inference with Boltzmann.
Together, our results provide evidence that LESS is a better model for explaining people’s
choices, performing high-quality inference, as well as robustly inferring preferences
in the real world. Thank you very much!