in this work we propose interactive
hyper parameter optimization and
demonstrate the dashboard prototype with
paintable timelines
hyperparameter optimization is an
important but time-consuming task in the
development of machine learning
applications automatic optimization can
free a developer from tedious trial and
error processes
in automatic hyperparameter optimization
an optimizer searches for parameter
values that maximize a black box
function
in our expert interview we found that
automatic optimization has several
challenges
in automatic optimization human experts
sometimes have their own domain
knowledge about parameter distributions
on specific problems
however it is difficult to leverage the
domain knowledge
existing automatic optimization
frameworks do not have enough supports
for human intervention during
optimization
in this work we propose interactive
hyperparameter optimization that
introduces interactivity into optimizer
for parameter turning
we present an interactive dashboard to
allow the user to set search ranges
during the search to leverage domain
knowledge of the user to facilitate the
search
the dashboard visualizes status of
optimization processes
and provides painting interface to set
search range
this is a screenshot of our system
schedule panel shows the schedule score
panel shows the score at each iteration
parameter panel supports painting
interface
it shows the parameter values tested in
the past and to be tested in the future
horizontal axis shows iterations of
parameter tuning
vertical axis shows result score and
parameters of an iteration
the system visualizes the progress of
the optimization as in standard
dashboard
the user pauses the search at any time
and then sets the search range in the
parameter panel
the user resumes the search and the
system searches the parameters within
the given range
we provide three ways to set the range
directly draw a new bounding box
grab and drag the boundary
or selecting past samples
the user can also fix a parameter
after setting the range the user can
split the range by specifying the number
of splits
we show the interface design of the
dashboard
in all panels we encode score values
into color information
we also use the vertical axis for scores
in score panel
the user see tested parameter values and
its scores in parameter panel
this interface adapts high-dimensional
parameter optimization such as learning
deep learning models
a notable feature of the paintable
timeline is that it not only visualizes
the ranges but also allows the user to
modify the ranges by several painting
operations
first the user can use the paintable
timeline to gradually narrow the search
space during optimization
second the user can fix a parameter
value during the search if he or she
judges that it is no longer necessary to
try different parameter values
third the user can schedule future
searches by specifying which parameter
space to be explored when
fourth the user can select past trials
in the score panel with rubber banding
to set the ranges of the selected trials
parameters
this is effective to detect promising
parameter ranges after several trials
we implemented our prototype system on
an existing optimization framework
optuna our interactive sampler
communicates with the interactive
dashboard via the websocket-based
communication protocol
with our interactive sampler users can
run interactive hyper-parameter
optimization by modifying few lines from
the original optuna code
we designed the representative three
cases where interactive optimization
with prior knowledge in parameter
distributions can outperform fully
automatic search
first example shows a case where the
user knows that the optimum is at the
diagonal region
in this case the user limits the search
range of x and y to similar values after
observing the initial results
the next example shows a case where x is
dominant and smooth while y is
subordinate and random
in this case the user limits the search
range of x observing initial results
the next example shows a case where the
optimum is inside a basin
in this case the user limits the search
range of x and y at a low score region
after observing initial results
we run a user study to examine the
possible effectiveness of interactive
optimization using the paintable
timeline
our main goal is to check whether
interactive optimization with prior
knowledge can outperform fully automatic
optimization in specific cases
to do so we compared our interactive
optimization with fully automatic
optimization
we recruited seven machine learning
experts who have sufficient experience
in research and development
in the study we assume that the user is
already familiar with the target problem
type and has worked on it several times
we used three illustrative examples that
each example has the same characteristic
but variations about score distributions
as the result we confirmed that
interactive optimization outperforms
automatic optimization for all tested
functions
we measured relative scores by dividing
the respective scores with the
corresponding true optimum
we obtained positive feedback about
interactive optimization concept
in contrast experts also identified
limitations and challenges of the
current dashboard and the study
this slide shows main limitations and
future works of this study we tested
interactive optimization with very
specific cases in our user study
we had the strong assumptions that
experts had prior knowledge about score
distribution
in the future we hope to investigate the
effectiveness of interactive
optimization in practice
as real-world optimization problems are
diverse and take considerable time
replicating them in a controlled user
study is difficult
to run a large-scale study we need to
develop a more robust and functional
implementation and distribute it to the
real users
the current implementation is still an
early prototype and lacks many
fundamental functionalities
we plan to run further interaction
design process to improve the dashboard
interface the extensions are already
tested as a proof of concept
implementation
multiple processes work on different
locations along the timeline in parallel
the user can change the location by
dragging the knob
the user presses the estimate button and
the system shows internal score
estimation of the sampler
the user presses notify button and sets
when to send a notification
we currently support desktop
notification
in conclusion this work presents the
concept of interactive hyper parameter
optimization and developed a prototype
implementation of an interactive
dashboard
based on optuna framework the user study
demonstrates that interactive
optimization with appropriate search
strategy can outperform fully automatic
optimization in the three specific cases
thank you very much please read the
paper for more details if you have
interest