[Music]
Hello everyone. My name is anam ahmad
khan and i am a researcher at the
university of Melbourne. Today I will be
talking about how gaze and speech can be
effectively combined for enabling
implicit interactions.
Multi-modal interaction refer to those
that utilize multiple modalities to
exchange information.
One such potent interaction is produced
when gase and speech are combined.
Most gaze and speech interaction
techniques use data from these
modalities as explicit input where
speech is used to communicate user's
intent while the gaze is used to
disambiguate the target of a speech
command by requiring the user to direct
their eyes as pointer. Even though the
resultant interactions are robust, users
are likely to deviate from their natural
Behavior. This is because users are more
likely to simplify their speech command
while the use of gaze as a direct input
requires conscious user effort that can
induce undesirable cognitive workload.
Thus, to enable true natural multi-modal
Interaction, we need methods for
integrating these modalities.
Therefore, we explore how natural gaze
and speech could be integrated for
enabling implicit interaction.
We demonstrate our approach in the
challenge of anchoring voice note which
relates to the problem of mapping a
voice note to the digital region to
which it refers.
Our prior work attempt to solve the
challenge of anchoring voice notes by
leveraging uses natural gaze behavior.
Our findings highlighted that the gaze-based
approach does not work for some specific
scenario such as whenever user depicts
similar gaze pattern in two adjacent text
Region.
We argue that a multi-modal approach
combining the content of user utterances
and their gaze behavior can better
overcome the challenge of anchoring
voice note.
This is because of two reasons, first
while uttering notes users often tend to
comment about the content they are
Reading. Hence the note content can have
relevance to the text regarding which a
note is made.
Second while uttering voice notes users
often directly look at their focus of
Interest. Hence the speech often include
terms which explicitly refer to the
items of interest.
Therefore we performed a combined
analysis of user's natural gaze and
speech behavior and present a machine
learning based pipeline that builds
a prediction model which indicates the
relevant region for anchoring notes.
For this purpose, we employed our
previously collected data set that
contains gaze and speech of users while
reading and recording notes. The input to
the machine learning pipeline was the
voice note which has to be anchored, the
candidate text message the participant
had read during note taking and the gaze
data depicted by the participant.
The input was first used to engineer
lexical features. For this purpose, we
first obtained the textual content of
the voice note.
The candidate text passage and the
transcribed note were then fed into a
sentence BERT model which represents
semantically similar sentences by
similar vector.
We then use these vector embeddings to
find the cosine similarity between the
transcribed node and the text passage.
The obtained scores were then sorted to
obtain the semantic rank of each text
passage.
Our exploratory data analysis revealed
that there were some notes that
reflected participants own opinion and
as such their content is likely to be
unrelated to the text passage.
Therefore, the relevance rank calculated
for the text passage of these notes
would not be useful and should not
influence a classification pipeline. To
identify the pattern of such notes, we
calculated a p-value for the semantic
relevance rank of each text passage.
This was done by first randomly
selecting 100 text passage which were
unrelated to the document being read by
the participants. We then found the
cosine scores between the unrelated text
passage and the voice note to create a
sampling distribution of cosine scores.
After building the distribution of
similarity scores we obtain the p-value
feature for the observed cosine scores
of text passages.
Our exploratory analysis suggests that users
often made voice note in which implicit
references were made to data displayed
on the screen.
Therefore to capture these patterns we
focused on jointly analyzing data from
both input modalities to engineer
features that can identify users visual
attention while making implicit
reference to content.
For this, we analyzed each note content
to observe whether demonstrated pronouns
occurred in it. Whenever a note
contain these pronouns we expected an
implicit reference to the content
displayed on the screen would be made.
Hence for each voice note we analyzed
the gaze during a specific window and
engineered gaze features which reflected
which textual content users looked while
uttering an implicit reference.
Lastl, we extracted reading pattern
based feature that captures user global
gaze behavior while composing voice note.
For this, we leverage our previously
proposed annotation classifier to obtain
the probability for the candidate text
message being predicted as annotated
based on users global gaze pattern while
they read and recorded voice notes.
After engineering various features we
combine them in a logistic regression
classifier that could predict referent
text passage for anchoring voice notes.
Our findings show that the proposed
multimodal approach combining uses gaze
and speech
outperform the unimodal approach by an
average precision of 12.
This is because our pipeline leverages
data from both gaze and speech modality
which captures different dimensions
of user's input.
Speech capture what the user is speaking
While Gaze debate the global reading
pattern of the users while reading.
Additionally, the leverage synchronicity
between gaze and speech input captures the
user's visual attention while implicit
reference were uttered.
Thus, when data from both modalities are
combined they complemented each other
and are more informative for anchoring
voice note.
overall, our work contributes a machine
learning pipeline for implicitly
annotating voice note by combining users
natural gaze activity, the semantic
knowledge from their vocal utterances
and the synchronicity between gaze and
speech input. Through such a pipeline, our work
enables design of multi-modal system that
Combine natural gaze and speech pattern
to enable novel natural user experiences.
That's all from my side. Thank you for
paying attention.
[Music]