Hi, I'm Yichi, a phd student at Aalto university. I'm going to talk about our paper investigating
positive and negative qualities of human-in-the-loop optimization for designing interaction techniques.
Designers often need to explore vast design spaces providing the base parameter setting
for an interaction technique. For example, designing a vibration on a smartwatch needs to configure the
vibration amplitude, duration, and repetition. For any kinds of game a designer has to determine
some factors, such as the latency of the input the density of the obstacles, and speed, etc. And all
kinds of input have some internal parameters that require a base setup. Such design optimization is a
common step in all design workflows. And, so far, this process is majorly guided by the designers.
Typically, a designer will propose a design parameter x, which is based on
their knowledge and intuition, then the interface will be configured according to a parameter
users then interact with the interface to acquire some major performances based on
that, the designers then proposes a next design x. This continues until a satisfying design is made.
The major pros of this designer-led process is that the designer has a stronger sense
of agency. However, maybe they may be subject to confirmation bias or suffer for design fixation.
thus miss the good design candidates. On the other hand, optimization methods have been advanced a lot
in recent years. So, another potential design process is instead of having human designers
to ideate a design candidate, you can use optimizer to search for the optimal designs.
In this framework, an optimizer will propose a design and the interface will be configured
accordingly. Again, the user will interact with it, and the measurement will be provided to
the optimizer, then the optimizer will observe the performances and propose
another design parameter. All these form a human- in-the-loop optimization. So far, no study has
compared the optimizer-driven design process to the designer-led design process. We hypothesize
that optimizer will avoid design fixation and search the design space more systematically.
But, in the same time, the designer will lose the agency and engagement from the design process.
Our paper contributes an experiment that compares these two conditions on 3D touch interaction
with 40 novice designers. Before diving into the study, we must mention the optimization method
we employed. We chose multiple objective Bayesian optimization. There are several reasons. First,
Bayesian optimization is designed to optimize for a black-box function, so it requires very little
assumption. Secondly, previous works have shown that among all black-box optimization methods Bayesian optimization
has very promising performances. Lastly, it is able to capture the noisy behaviors
from users. Specifically, we chose multi-objective Bayesian optimization, also known as MOBO.
The reason is that a lot of time, designers deal with not just singular objectives but multiple.
MOBO is ideal to deal with such scenarios. The basic principle is that instead of searching
for a single optimal point, MOBO searches for the Pareto optimal designs, which are the best trade-off
of all the designs. Now, we can move on to the study. A quick summary of the major findings: Optimizer-driven
design delivers better outcome in terms of the major performances. However, the subjective experience of the designer is worse. Also, MOBO promotes wider exploration of the design space.
We selected a 3d touch interaction as a design challenge. Specifically, we chose the parameter
in Go-go technique to be optimized. It is a classic work defines a transfer function that maps
a real arm vector to a virtual reality cursor. One important parameter in this Go-go technique is D,
which is a threshold distance. If the real-hand position is within D, the virtual cursor is at
the same position with the rear hand. However, once the real arm exceeds the threshold D, there will
be an additional parameter k to determine how fast the virtual hand will be stretched out.
Besides the k and d in Go-go technique, we further introduced two parameters that control
the vibration of the controller when the virtual cursor contacts the target in virtual reality.
We have configured the MOBO as follows: The four design parameters are as introduced earlier.
And the range is defined by a pilot study. The two objective functions we made is speed,
which basically maps the completion time and accuracy which linearly mapped to the selection
spatial error. We configured MOBO as such, but please check our paper for more details.
Our experiment setup is 40 novice designers were invited to perform an optimization task where they
are their own users. And they are asked to optimize for the two objectives as we just introduced.
It is a between-subject study design. Besides measuring the performances, we further measure
the creativity and workflow they perceive by questionnaires. Here are the results.
To summarize, while both conditions led to comparable speed, MOBO resulted in better
accuracy. Further, MOBO successfully reduced the mental load and efforts during the design process.
On the other hand, designer-led process has higher rating in subjective feedback. It has
significantly higher agency and ownership, and also leads to higher expressiveness. From our experiment,
we've learned some interesting challenges for future research to tackle.
First, we encourage future research to investigate the potential of allowing the designers to steer the optimizer if needed.
As we learned from the study the designer felt loss of agency. Future process should allow the designer to
prioritize the design dimensions or directions if they want.
Second, is the mixed-initiative interaction.
That is, pushing the optimizer to the background, and only make suggestions or recommendations.
Third, improving the transparency. Our study showed that the designer felt uncertain
with the optimizer as they cannot reason what the optimizer is attempting to do.
A potential direction is to allow the optimizer to communicate whether it is exploring or exploiting to the users.
Lastly, we anticipate that allowing the users to actively guide MOBO performing exploration
or exploitation will lead to better agency and higher searching efficiency.
To conclude, our paper contributes a novel experiment where optimizer-driven design
process is compared to designer-lead process. And the result shows that the optimization leads to
better performances that sacrifice the agency and expressiveness. The implication is that designers
can indeed benefit from Bayesian optimization, but future research is needed.
That's all my presentation. Thank you for your attention.
Please check our paper for more details.