We propose two online learning algorithms
for modeling the personal preferences of users of interactive systems
The proposed algorithms leverage user feedback to estimate user behavior
and provide personalized adaptive recommendation
for supporting context-dependent decision making
Context dependent decision making is a cognitive process of
selecting an action depending on the current situational context
In the daily use, smart phones users often need context-dependent
decision making to achieve their goals
For example, users repeatedly select their photos,
and add aesthetic image filters, then upload them to their SMS accounts.
Repeated decision making increases cognitive load and can cause decision fatigue
In this work, we propose online preference modeling for adaptive
recommendation to support repeated user selection tasks
We particularly focused on a common application of image filter selection
We take a data-driven approach for predicting target user selections
by using a database of other users' selections
The proposed approach needs to collect pairs of images and
and its selected aesthetic image filters from multiple users
We then apply the paired data to supervised learning algorithms
to generate policies of past other users
A policy can predict a past user's filter selection for given images
The predicted filters are supposed to be similar with the past user
We collected selection data from multiple users and generated a set of policies for our online preference modeling
We formulate preference modeling as online prediction algorithms over a set of learned policies
The algorithms adapt to a target user by learning the policy that best predict the user's selection
First, our online learner receives an image for filter selection
Each policy predicts suitable filter for the given image
The user then selects a filter from the candidates for the image
Online learner can access the user's selection
In this example, the user selects the pink filter from three candidates for the dog image
The online learner gives rewards to policies in the set that predicted the correct filter
Our approach allows application designers to customize reward functions
In this case, the reward function adds just one reward value to policies
The online learner accumulate reward values for each policy over time steps
The learner uses the current optimal policy to predict user selection
and visualize personalized adaptive recommendation at each time step
We consider two settings of this online prediction problem
The first setting is already described in the previous slide
The first setting is allowed to obtain prediction results from all policies in a set
It can update the weights of all the policies sequentially to find an optimal policy for the given target user
We call this online prediction setting All Prediction
We also consider the more challenging setting
In this setting, the online learner is not able to access all predictions from a set of policies
This setting easily occurs when an online learner has a large number of
policies in a set or computational resources are limited such as on mobile devices
In such a case, the learner is allowed to obtain a limited number of predictions in each time step
We call this setting Limited Prediction
In the second setting, the online learner can access the numbers of M predictions from number of N policies
M is larger than 1 and smaller than N
We aim to design this algorithm to efficiently search for an optimal policy from limited M observations
Unlike the first setting, the online learner is not allowed to access all prediction results in each timestep
To address this problem, we used methods of the multi-armed bandit problem to efficiently explore better policies in a set
The online learner chooses M policies based on scores of bandit algorithms such as the KL-UCB algorithm
We show an example of the limited observation setting
In this case, the online learner chooses a policy from a set of six policies in each timestep
The online learner selects three policies based on bandit scores for exploration exploitation balancing
The online learner obtains three predictions from the selected three policies
The predictions are probabilities of each image filters for given images
The online learner computes exponentially weighted averages of the probabilities based on rewards of policies
The rationale is to combine multiple policies to generate personalized preference models for target users
The online learner then obtain user filter selection to update the preference models
In this case, the user selects the pink filter
The online learner then updates the rewards of the selected policy
In this case, Policy A and C predicted the pink filter as Top1
and thus receive reward value 1
The online learner continuously accumulates reward values for personalized modeling
Again, the application developer freely designs the reward functions
We apply the proposed algorithm to an image filter selection application
In this scenario, users were asked to select a suitable filter for a given image based on their preferences
This application can present the top-3 recommendations of image filters from the proposed algorithm
The algorithm sequentially learns target user preferences based on the features of the given images and user filter selections
The figure shows our GUI interface for image filter selection
The interface contains several GUI components such as image preview,
filter view with the top-3 recommendations, and image sequence buttons
The interface has the original image and eight image filters on the candidate filter view
Users can select a filter by clicking on it with a mouse
Image preview then shows an image with the selected filter applied
Users can freely compare filters to make a selection
We built a dataset on the image filter application to evaluate our algorithms
We recruited 16 university students
Seven of the participants had never selected image filters for photo sharing SMS due to use of supervised learning for policy generation
A small amount of selection data makes policies with poor performance
Thus we decided to collect 300 selections from each participant to learn a policy that can reflect the participant's preference
We thus obtained 300 user selections of image filters from all 16 participants
Totally 4800 pairs of images and filters for the dataset
Note that the application did not show any recommendations during the data collection process
We investigated how accurately the proposed algorithms are able to predict user selections
and how filter recommendation affects user selection and behavior
In this presentation, we describe the overview of the evaluations
Please read the paper for the details of three evaluations if you have interest
The first evaluation compared our algorithm with baselines in All Prediction setting
We used supervised learning and bandit algorithm as baselines
The evaluation shows that our online learner made better personal preference models for each user dataset over baselines on several metrics
We also changed reward functions for updating policies
The results showed that the prediction performance was tunable by changing the reward function
The second evaluation investigated the impact of numbers of learned policies on All Prediction setting
We changed numbers of policies in a set for our method
The results show that prediction performance was able to improve as the number of policies in a set increased
The third evaluation investigated prediction performance in Limited Prediction setting
We compared our method with two previous methods
The previous method is similar with our method
but there is difference about use of selected M policies
Both baselines select M policies and then use 1 policy for prediction and update rewards of the selected M policies
The results showed that our algorithms are robust under challenging Limited Prediction settings
in which a smaller number of expert policies is assumed
This slide shows a brief summary of our pilot user study
We investigated how visualizing recommendation affects users' filter selection behaviors
We recruited 16 participants
in the user study, we compared our visualized condition with no visualization in 100 filter selection tasks
The results indicated no significance in a metric of selection time
However, in subjective metrics, the participants overall liked our visualized condition
The subjective results indicate that our recommendations reflect target users' preferences
and give participants cues to make satisfying filter selections
We summarize the main contributions
First, we propose a preference modeling algorithm for adaptive recommendation
formulated as online prediction problems using past users' selection data
Second, we extend our algorithm for limited observation setting
in which the online learner is allowed to access a limited number of predictions
Finally, we develop a photo filter application with filter suggestion and create filter selection dataset for our evaluation
Thank you very much for your attention