Hello everyone my name is Bryan Wang from the university of Toronto
thank you for attending the talk from all over the world today I'm going to introduce our paper
Soloist generating mixed-initiative tutorials from existing guitar instructional videos
through audio processing this is joint work with Mengyu Yang and my advisor Tovi Grossman
Learning instruments using online instructional videos has become increasingly popular.
However, learning with pre-recorded video does not provide immediate feedback
and existing video navigations are not optimized for instrumental learning,
making the learning experience encumbered to address these limitations we present soloist a
full-stack framework that automatically turns off the shelf video lessons into interactive tutorials
So, how can we create an interactive tutorial from a pre-recorded video?
Firstly, Soloist uses a series of audio processing to extract musical information from the video.
This information is then presented on a front-end interface for the user to interact with.
The user can record their practice and Soloist would automatically analyze their performances and
compare them with the video. To provide a guided learning experience Soloist also provides three
types of learning feedback including audio replay melody visualization and the note correctness
score which we will discuss later let's first take a look at how soloist enables video navigations
tailored for instrument learning. Soloist uses a waveform navigator to visualize the instructor's
narration voice and the instrument sound, helping the user locate where the guitar is being played.
Moreover, Soloist automatically segment the video into demonstration regions; clicking on the region
will play the specified interval and the user can drag the boundaries to adjust the start and end of
a region. Soloist provides a set of navigational tools to facilitate music learning with videos.
For example, users can click on the loop button to play a region repeatedly;
clicking on the preview button and hovering the cursor on different
regions will display their melodic lines in each region.
Users can select a region and press the query button,
Soloist will then display the regions containing similar melodies and hide the remaining ones
Another important feature of Soloist is to provide immediate feedback on the user's practice.
To practice along a region, the user can select it and press the record button Soloist will then
analyze the user's performance and provide various feedback
The first type of feedback is melody visualization which visualizes the melodic
lines of both the user's recording and the instructor's demonstration
Many common mistakes of guitar performance, such as missing vibrato, can be therefore identified.
Another type of feedback is audio replay, the user can play their recordings and the reference
audio simultaneously and a mixing slider allows them to adjust the relative volume
Soloist also provides a score assessment based on the note correctness of the user's performance.
Since the models we use for audio processing may have inaccurate predictions,
we adopted a mixed-initiative design where the user can overwrite the score
if they believe it's inaccurate.
To evaluate Soloist we conducted a technical evaluation on the video segmentation
by comparing the segmentation results with human labels. The evaluation shows that
our algorithm is reasonably accurate and outperformed all baselines across metrics.
We also conducted a qualitative user study remotely with 8 participants to
obtain their high-level feedback. The results showed that the participants
unanimously preferred practicing with Soloist compared to using traditional youtube videos.
With that, I wrap up my talk and I'm happy to answer any questions, thank you.