Hi
I'm Nikhil Singh and I'm presenting on
the paper the sound sketch pad
expressively combining large and diverse
audio collections
this project pertains to audio based
recombinant music which I use here to
describe several rich traditions of
music in which sounds form the basic
units
rather than musical notes often these
kinds of music freely combine elements
of recorded audio content into new
expressive forms
over the years they have grown in
complexity as we have more recorded
audio content available than ever before
traditional music production tools such
as digital audio workstations are
designed to manage small audio
collections in precise detail
having emerged from hardware paradigms
such as the tape machine and audio
mixing console
they make working with much larger audio
recordings and collections very
cognitively effortful and time consuming
working with more sound requires expert
knowledge of the tools and highly
optimized workflows in addition to this
the sound sketchpad project focuses on
interactively composing sound units into
new sound collages controlled by sonic
and graphical inputs
the sonic input acts as a guide for what
the output should sound like while the
graphical control allows refinement of
various aesthetic qualities of the
resulting collage over time
in this implementation the graphical
controls have four types energy density
weight and variance
we refer to the combination of these two
inputs as a sketch which the system
works to resolve into a sound collage
to begin with the input audio is
windowed and time varying features are
then extracted
a feature of interest pitch by default
is used to search the database to find
suitable sensor combination up to 100 by
default
some of the input graphical paths are
then interpreted as audio feature
trajectories replacing analyzed features
in the time feature matrix
the remaining control inputs are mapped
to audio processing functions
these can be arbitrary functions to take
as arguments the audio and control
trajectories and produce audio as the
output
to resolve sketches into sound
combinations candidate sounds are first
time shifted to make multiple copies
we extract multi-feature matrices from
each sound and a linear combination of
these represents the collage which we
try to maximize in similarity to the
input sketch
then one of two methods are used either
sounds are greedily selected at each
iteration to minimize the euclidean
distance between the source sketch and
target collage multi-feature matrices
which we call the sparse method or
sounds are assigned continuously varying
amplitudes that are optimized via
simulated annealing
with multi-feature kl divergence as the
objective function
we refer to this as the dense method
which produces mixtures of sounds rather
than just combinations
in this case a density parameter can be
assigned to filter out sound
assigned to low amplitude let's listen
to a quick example here's an example of
an input audio guide consisting of me
whistling
now here is a constructed collage based
on this input without really tweaking
the graphical inputs
oh
this approach supports flexible
iteration even with large audio
collections
as well as combinatorial creativity at
scale in comparison with daws which are
designed for small audio collections
audio database solutions which require
extensive manual browsing
and other purpose-built approaches such
as corpus based concatenate of synthesis
it also supports easy extensibility in
that new parameters for control can
easily be added to map to target audio
features or audio processing units
the system is also learnable it requires
no particular expertise or experience to
use
as well as being consistent in behavior
and interpretable
thank you for listening