Hello everyone, I’m Zhi from Stony Brook University. In this video,
I will introduce our work Modeling Touch-based Menu Selection Performance
of Blind Users via Reinforcement Learning. As a fundamental problem in HCI, menu selection
behavior modeling has been extensively studied. However, most existing studies have focused on
sighted users, leaving blind users’ menu selection under-studied. A computational model that can
simulate blind users’ menu selection behavior would be a valuable tool for designing, adapting
and evaluating accessible user interfaces. It also provides a chance to infer user-specific
characteristics. Although menu selection may appear easy for sighted users, it involves
rather a complex behavior for blind users. To select a menu item, a blind user navigates
to the item with only auditory feedback from a screen reader.
Based on the auditory feedback, the user uses swiping, gliding, and direct touch
actions to move the focus to the target menu item, and double-tap to select it. However,
each action has its own strengths and weaknesses. A blind user may choose any of the three
navigation actions or a combination of them to search for and then select the target menu item.
In this paper, we build a computational model to simulate how blind users select
menu items using the four actions. We model the menu selection process as a stochastic
sequential decision process, in particular, as a Partially Observable Markov Decision Process,
where the partial observability comes from that the user can only hear the auditory feedback.
We propose a model of long-term memory which describes how the blind user recalls and
forgets the menu item position. We then solve the problem using reinforcement learning.
In the reinforcement learning framework, the environment simulates the menu system, and the
agent plays as the blind user. The agent learns the optimal policy, that is, how the blind user
selects menu items, via simulated interactions between the agent and the environment.
At every time step, the environment has a state, including the position of the target menu item,
the user’s finger position, and the focused menu item position.
The agent then partially observes the state. We assume that the agent can estimate the
finger position, and the focused item position by estimating the relative
position of the finger on the phone screen. The agent then updates the position memory
based on the observation. For a menu item, the position memory assumes that its recalled
position follows a Gaussian distribution, where the mean is its real position,
and the standard deviation is determined by the total memory B(m_i) and a parameter sigma PM.
After that, the agent computes the position belief by discretizing the Gaussian distribution.
At the same time, we also consider the effect of menu arrangement. For example,
in a grouped menu, the agent can determine in which group the target menu item is,
and set the probability of the items in other groups as 0.
Then the agent uses a Deep-Q Network to map the position belief to one of the four actions that
leads to the fastest selection, and then executes it in the environment.
For the direct touch action, as there is no visual feedback, we assume that the real landing item can
be modeled by a Gaussian distribution, where the mean is the intended landing menu item position
and sigma DTU is a parameter to be determined. After executing an action, the agent receives
a reward immediately. The reward is defined as the negative value of the estimated time cost
of the action. For the selection action, we also add extra incentives for successful selections.
Before we can train the model, we need to determine the time cost model for the four
actions, and the parameters sigma PM in the position memory and sigma DTU in the direct
touch uncertainty model. However, there is no existing model in previous work that we can use,
so we conducted a menu selection user study first.
In the user study, we considered two different menu lengths of a linear menu,
each of them with 3 different menu item arrangements. For each condition,
the blind user selected 60 menu items. We recorded every accessibility event for further analysis.
Based on the user study data, we first created time cost models for the four
actions. We assumed that the time cost could be modeled by a Gaussian distribution
and estimated the mean and standard deviation based on the user study data.
We then determined the parameters sigma PM and sigma DTU using grid search. We optimized
them together by minimizing the mean absolute percentage error between the selection time and
action composition predicted by the model and in the user study. Here the action composition
means the percentage of actions that were used by the users, as shown in the figure on the right.
The summarize, we only used the time cost for actions to determine the time cost
model and the overall selection time and action composition to determine
parameters in the position memory and the direct touch uncertainty model. We did not
use any of the user’s action sequence in the training process, and the optimal policy was
learned based on simulated interactions between the agent and the environment.
We also observed that in the user study some users did not use the swiping action at all,
as user 2, 6, 7 shown in the figure. Based on this observation, we adopted two training strategies.
The first strategy is called one-group model, where we trained a model for all users. The second
strategy is called two-subgroup model, where we trained a model for each subgroup of users
according to whether they used the swiping action. Now, let’s look at the model evaluation result.
For the selection time, we can see that both the one-group model and the
two-subgroup model achieved high prediction accuracy. Especially for the two-subgroup.
Our model captured the effect that longer menus have longer selection times. For the 10-item menu,
the random menu has the longest selection time and the grouped menu has the shortest selection time.
We can also see that our model overestimated the effect of menu arrangement on the selection
time for 6-item menus. This means that the benefit of a grouped menu layout may
not be generalizable to short menus. We also compared our model with input
action model based methods. In these methods, the user uses only one of the direct touch,
gliding and swiping actions to move the focus, and uses a double-tap to select the focused item.
The results showed that our two-subgroup model has the best performance in terms of
predicting the selection time. In addition to that, our model can also capture the learning
effect and the menu selection strategy which the input action model based methods failed to do.
To check the learning effect during the menu selection process, we manually divided the 60
trials into 6 blocks. We can see that in the figure, our model could successfully
predict that the selection time decreases with more trials that the user selected.
In addition to selection time, as shown in the following figures, our model can also
capture the action composition that was used to select menu items with a low prediction error.
Furthermore, our model can also capture the selection strategies that were used
by the users. We observed three different selection strategies in the user study.
We could see in the figure that our model could successfully predict the percentage of selection
strategies used by the users. To further evaluate our model,
we also conducted train-test split evaluations where we estimated the parameters for the actions
and memory models with 80% of the user study data and held 20% for testing. The results
were similar to the previous evaluation. Please refer to our paper for more details.
To conclude, in this paper, we proposed a computational model for blind users’ menu
selection behavior. We modeled the menu selection problem as a Partially Observable Markov Decision
Process. A key aspect in our model is a position memory model which takes long-term interaction
history into account to describe how the blind user recalls and forgets the menu
item position. Our model successfully captured the effect of menu length and menu arrangement,
the percentage of actions used by the user, and the user’s selection strategy.
Thank you!