controlled hierarchical summarization for long form spoken dialogue every day,
we are surrounded by various forms of spoken dialogue, interviews,
debates, meetings and conversations with friends and mentors.
These are all rich sources of information.
However, accessing this information is difficult because these sources invariably
contained the shared challenge of interpreting an auditory rich stream of information.
Now imagine receiving an email from a colleague about an exciting interview on YouTube.
It's 23 minutes long. You're not sure if you want
to commit to watching the whole thing but you want
to know if there's anything you are interesting in it.
The goal of this system is to help people explore
audio and find key takeaways quickly, automatically summarizing and distilling
content to the most interesting points would be an ideal solution.
We can create such a system by combining a speech
to text model with a text summarization model by processing
audio into transcripts, inserting punctuation and deploying a text summarization model.
We should theoretically be able to output condensed text representations
of a long audio file.
However, simply using pre existing solutions is not sufficient and
as practical usage concerns, automatic speech recognition has word recognition
errors that can mislead users in this example, whether, as
in whether or not should be a seasonal weather predicted
punctuation errors can split off sentences, creating incomplete context.
In this example, the sentence is split into two, creating new meaning.
Unfortunately, speech training data is not easily obtainable summarization models
are trained on clean structured text and are not well
suited towards speech, which contains its own set of challenges
such as disfluencies and lack of structure.
This leads to hallucinations or incorrectly out putting unseen words.
Speech may also be ambiguous, confusing summarization output.
Ultimately, these errors prevent practical usage.
Still, we explore these models for feasibility in a novel
speech summarization system, investigating the exact areas, we discover that
some language models may be reused.
We find that models trained on typical data such as news are not suitable.
However, models trained on paraphrasing noisy text show promising results
but still do not reach a usability standard due to
the nature of summarization, a model must understand the underlying
semantic content, otherwise the output will be illogical.
Our paraphrasing model still suffers from this.
However, semantically grouping the input into chunks helps model comprehension
by presenting the input text is discreet and unique topics.
The model does not need to reason between multiple ideas.
Nonetheless, machine learning systems are imperfect and impose additional design considerations.
In our system, we consider the following user intent as
users have different goals in understanding information, content, user navigation
to support different browsing habits and user error correction to
recover comprehension due to incorrect ai generated summaries.
The system presents summarization in a hierarchical manner.
Starting from the ASR segments we recursive lee combined similar
summaries to obtain increasingly abstract ones, providing users varied levels of detail.
Here we see how the information is presented, clicking on.
The short summary, gives a high level idea of this
audio segment and each subsequent level provides increasing detail, allowing
the user to read as deep as they want.
Lastly, the users the option to read the speech to
text transcript and listen to the original audio segments.
Our system supports two forms of navigation by presenting short summary.
Start to finish in a linear fashion.
Users can reach short summaries to get an idea of the speakers conversational flow.
Alternatively, users can use short summaries as indexing tools to
select segments they are interested in.
Our system also affords user air recovery from machine learning errors.
Using tiered information. Users can understand that the partner experiences
referenced in the short summaries are within the context of a retail Starbucks store.
In this example, we see how the ASR segment outputs.
We're in the midst of what's probably one of the
largest reckonings around racial equity by listening to the audio.
We can conclude that the rectangle openings should be the
word reckoning For our experiments.
We evaluated our system on 10 participants.
Users individually scored 556 summaries and we found that 71
were immediately understandable and 98 were understandable upon further investigation
Out of all users, we find they spend 27 of
the time they would have spent compared to listening to the full audio file.
Lastly we go over qualitative user feedback.
You can scan a 20 minute interview in a minute
and find the parts you find interesting.
I would get this as a Youtube plugging.
I found that I could use the short summaries as
subheadings and individual bullet points.
Its structures information well by length and complexity, automatic SparkNotes for boring talks.
So in conclusion, our system affords accessibility, user intent, user control and error recovery.
Thank you for listening.