Hi, my name is Maciej Korzepa and I'm from Technical University of Denmark. I will
talk about how we can use simulation environments to guide the design of
contextual personalization systems using hearing aids domain as an example.
First, let's consider why we need context-aware personalized systems for hearing
aids users.
Hearing loss is usually represented by an audiogram which describes loss of sensitivity to sounds of different
frequencies. When fitting a hearing aid for a given user to compensate for the
hearing loss their audiogram is translated into frequency amplification curve
and possibly some other settings responsible for directionality and noise
reduction. These settings are usually fine-tuned for speech in our scenario
which represents the most common hearing difficulty. However the same setting
often fails to provide benefit in many other situations where user intention is
for example ignoring surrounding sounds and focusing on work and enjoying subtle
sounds of nature attending a lively discussion at work or exercising.
So diversity of contexts is not really taken into account and typically only a
single setting is offered to users. Moreover the different needs arise not
only from different situations but also from differences in cognition between
users. Two users with identical audiograms can have vastly different perception of
sounds due to some more complex cognitive factors that are not measured
because of technical or practical limitations.
So how can we personalize settings for different users and different contexts?
et's make a context-aware in recommender system. But where to start?
Let's consider typical recommender systems. They mainly focus on recommending disparate items such as
videos or songs, they used learning signals such as watch time or whether
item was bought or not, they're trained on huge static data sets which are in fact
very limited and context-aware data sets are even more rare. And as the data sets
are static these systems can't really learn dynamic user behaviors.
How would the situation look like for context our hearing aid setting recommender system?
There are absolutely no training datasets, recommendation space is
completely different, standard learning signals are not applicable and reaction
to dynamic user behavior is very important in this domain. So we have to
start from scratch. Normally we would run a user study. First we would design and
implement the method then we would recruit some users to evaluate it and we
would hope that it works. However as this is a new method in a new domain
probably it won't work so we need to reiterate after improving the method.
This approach has a few serious issues. First of all, it's very time consuming.
Designing a user study can take lots of time and running it can take even longer.
Secondly it can be very costly and Finally it's very limited in size so we
might not get even enough data out of it. So what are the alternatives?
We propose that we should simulate a user study or even 1,000 of them. The main idea is to
create a model of domain-specific environment including elements such as
users, context, preferences and interaction interface between
environment and agent, recommender system. The environment should also include
dynamic changes in these elements. We can model them by using available data to
limit the discrepancy between the simulation and the real world and when
the data is not available or sufficient we can use common sense assumptions
about their characteristics. We should make the assumptions adjustable such
that agents can be evaluated under a wide range of conditions.
Our goal is to be able to cheaply and quickly evaluate different solutions under a
wide range of conditions allowing agile iterative design. It is important to say
that the goal is NOT to create a very detailed and accurate model of the real
environment. But even if the agent works in the simulation environment how can we
be sure that it will work in the real world? Of course we cannot be sure about
this. However if the agent fails in the
simulation environment, it will almost certainly fail in the real world.
Our goal is to answer questions such as can the agent actually learn preferences of
individual users in realistic time before users lose interest in the system?
Or whether it can exploit similarities in preferences between users to speed up
the learning process? Whether it can respond to dynamic changes in user
behavior or preferences? Or how sensitive it is to incomplete information about
relevant context or user characteristics which is very often the case due to
technical limitations or privacy privacy concerns. The more thoroughly we test the
agent the bigger the chance it will succeed in real world.
I will give now a high-level overview of the simulation. We have an agent that has access to setting
space from which it will choose settings for recommendation. We have a user
sampler which generates or selects a user for an iteration of the simulation
and the context sampler corresponding to the user. From them we obtain current
user and context features that condition user preference model.
The agent observes only the observed components of user and context features
and based them it offers a setting recommendation to the user.
The user evaluates the recommendation sending a noisy response back to the
agent which updates its knowledge about user preference. After interacting with
the agent user state might also undergo a change for example in engagement which
will have an impact on future interactions. As simulation might be
viewed as a generative model we need to define these blocks in a probabilistic
manner. In this work we focus on generating possible preferences using
Gaussian Processes that give us a high level of control over assumptions we
want to put on these preferences. As not everyone might be familiar with Gaussian
Processes I will now give a quick primer.
Gaussian process is a nonparametric model allowing to impose certain structural
characteristics on a given function using Gaussian priors. Mathematically, the
values of function f are drawn from a normal distribution with covariance
matrix K defined through kernel that measures correlation between function
values as similarity between the corresponding inputs.
Here we can see the function samples for the most common RBF kernel we can control the smoothness of
these functions using lambda hyperparameter.
The lower the lambda the quicker the function will change. Depending on the
choice of kernel the supported functions might look completely different.
Here we see for example Brownian motion kernel and quadratic kernel.
We can also do predictions using Gaussian Processes. By observing some data points we narrow
down the range of plausible functions and can obtain mean prediction and
uncertainty around it.
I will now talk about context and user models and how we can use them to generate meaningful preferences.
We need to be able to generate context that is structurally similar to the one that we
can observe in the real world. It might be helpful to consider which context
types are actually relevant for a given application. For example in hearing aids
domain we might consider acoustic environment at low level such as
frequency composition or high level such as restaurant or natural sounds.
We can consider activity, location or even time. we can create a generative model for
context using existing data if available or create one from scratch using some
simple assumptions. In general we assume that the more similar two contexts are
the more correlated corresponding preferred settings.
Moreover we assume that context consists of observed and hidden components and
that the agent can access only the observed component. Finally we want to be
able to control complexity of preferences as a function of context as
well as its sensitivity to context observed and hidden components.
We achieve this by context kernel that we define as product of two RBF kernels
operating on observed and hidden components. By plugging the context kernel into a
Gaussian Process we can take samples of preferred setting. In these samples we
use only two context features one observed and one hidden.
By modifying the value of lambdas we can control the complexity of preference.
The more complex the preferences are the harder it will be for the agent to learn them.
We can also modify only one lambda at the time to control the contribution
coming from observed or hidden features. Here we can see how the preferences
change from being completely defined by observed features to being completely
defined by hidden features. The user model specifies user dependent
characteristics the together with context has an impact on preferences.
For hearing aid users it can be for example age, audiogram or more complex hearing
loss characteristics. Similarly to context we assume that the more similar
features are, the more correlated preferences will be. User characteristics
also consists of observed and hidden components and we want to be able to
control diversity of preferences across users as a function of user characteristics
We achieve this by defining user kernel that is weighted
sum of two RBF kernels where lambda parameters control the diversity of
preferences and theta controls the balance between the contributions of
observed and hidden components. In this example we sampled preferences for 8
users with covariances defined as in the figure. We can see that there are 3
clusters of users that have no correlation between them but there are
correlations within each cluster to a varying degree. We take a single sample
which looks like this. We can see that the first 3 users are indeed very
correlated, same as next 2 users and the last 3
users are also correlated but with the 8th user being a bit less correlated
to 7th and 6th user as can be also seen in the correlation matrix.
The sampled settings are the preferred settings only and in simulation we want
to be able to evaluate the degree of user preference towards any setting in
any context. For that we assume that the further a given setting is from the
preferred setting in specific context the lower degree of preference the user
has towards that setting. To achieve that we use Gaussian process regression with
RBF like kernels using sampled preferred settings as above as training data.
As a result we can assign some degree of preference to any setting within the
full space of settings as shown here for an example where the setting ranges
between 10 and -10. Finally we want to be able to map between user's latent
preference f in given contexts and settings to a noisy user response y.
In this example we assume that user responds with a score in range from 0 to
1 where 0 and 1 correspond to maximally disliking and liking the
recommended setting and 0.5 indicates neutral preference. We sample user
response from beta distribution with mean parametrized by user's latent
preference f and variance v indicating to what extent the user response is noisy.
Here we show the shape of the distribution for a few different values
of user's latent preference. Let's see a simple example of a simulation.
We consider a single user and two scenarios for contexts : one with full observability
with the context fixed and one with partial observability. The recommendation
space is one dimensional between minus 10 and 10 and we limit the
number of interactions with between users and agent to 200. We test two
agents: one oracle mean agent which is the baseline predicting single setting
that on average has the highest preference and UCB agent which a Bayesian
optimization based agent using Upper Confidence Bound to balance between
exploration of new settings an exploitation of already tested settings
that obtain high score from user. We measure the performance by cumulative
regret at interaction t which is simply how far the agent is from optimal recommendation.
In the full observability scenario we can see that the UCB agent
starts with lots of exploration but very quickly finds the reasonable settings
and overtakes oracle mean agent. In partial observability scenario we can
see that due to insufficient knowledge UCB agent takes more time to find good
settings but it still can improve over oracle mean agent in a reasonable number
of interactions.
The presented simulation framework creates a sort of self reinforcing loop of design.
We start with designing agents that we then evaluate in our simulation environment. Based on the simulation results we can improve
the agents and evaluate them again and repeat this process until we're happy with the result.
Then we deploy the agent in the real world
and based on the collected results we can improve the simulation
environment and revise underlying assumptions. This will allow us to create
even better agents in a new revised environment that matches the real world closer.
As future work we consider modeling dynamic aspects such as user engagement
the users community float and the effects they have on consistency of
responses or frequency of interactions with the agent. Also, we consider modeling
agents that can capture and act on patterns of similarities in preferences
across across users, can adjust to dynamic characteristics of the environment,
minimize the number of interactions needed to improve user
satisfaction with recommendations and can learn in a privacy-preserving manner.
Thank you for your attention and I would be happy to answer your questions.