Hello, my name is Yaohua bu,
I’m a phd student from Academy of Arts & Design,
Tsinghua University.
A majority of L2 English learners often encounter
difficulties in improving their pronunciation.
On the one hand, second language learners cause
multiple mispronunciations that can
hardly be noticed by themselves.
On the other hand, we are lack of professional
native-speaking English teachers.
In offline English classes, exaggerated
feedback has been an effective feedback strategy
for teachers to rectify the pronunciation.
The literature have proven that visually exaggerated
speech is capable of promoting the perceptual
ability of learners. However, this direction has
rarely been explored in the field of HCI.
In this paper, we provide a personalized
exaggerated feedback in a CAPT system
named Pronunciation Teacher.
Specifically , our system evaluates learners’ pronunciation with
real-time mispronunciation detection and diagnosis
algorithm. Afterwards, personalized feedback
is provided by the audio and visual exaggeration
generators based on the MDD results.
The whole working flow of PTeacher consists of
two components: pronunciation training courses
and audio-visual corrective feedback generator.
This photo is showing the working flow
of the audio exaggeration generator.
TTS model is used to synthesize
neutral speech with given text.
The exaggeration level is determined by
the accumulated score in the MDD results.
Then pitch, duration and energy of the selected
phoneme are exaggerated with the parameters of the
corresponding exaggeration level with PyWorld.
In this part, I introduce the realization of
the Visual exaggeration.
We generate the exaggerated articulatory animation
using the following methods:
increase the amplitude of key articulatory movement.
We modify the colour of the tongue.
Auxiliary graphics and supplemental texts are finally added.
In the audio experiment, to determine
the optimum audio exaggeration ratios,
we synthesize 800 speeches
with different exaggeration level including zero, slight,
medium and strong as test materials.
Distinguishability rate, which indicates whether
a listener can easily distinguish the exaggerated
phoneme in the exaggerated audio. The distribution
and average of distinguishability rates
demonstrates that higher exaggeration level
produces higher distinguishability rate.
Understandability rate indicates whether
the learner can still understand the exaggerated speech
without confusion. The result shows that,
for the learners with high proficiency or low proficiency,
the optimum exaggeration level is slight.
Perceptibility score, which evaluates exaggerated
speech’s perceptual quality.
Participants are asked to give opinion scores
ranging from 0 to 5
in terms of perceptual exaggeration level.
This result indicates that neither slight exaggeration
nor strong exaggeration is good enough.
The experiments on distinguishability rates,
understandability rates, and perceptibility
scores apply a non-linear fitting to find
the optimum exaggeration ratios with reference to
different phoneme types. The result is shown that
all the best exaggeration ratios
fall into the range of medium exaggeration.
30 L2 learners are invited to participate
in the visual perceptibility experiment. We find that
the optimum articulatory movements exaggeration
level is a slight exaggeration for learners with
high proficiency,
strong exaggeration for learners
with low proficiency.
The optimum tongue colour is slight
for learners with high proficiency,
while strong for learners with low proficiency.
We compare our system to 3 other systems.
We find that the improvement rate of learners
who are trained with PTeacher is much higher than
those who are trained with the non-exaggeration system
or the non-personalizing system,
and is comparable to those who are trained
with human feedback.
This is the demo.
In free text practice lessons,
Learners customize the lessons base on their needs.
This lessons enable learner to train
their pronunciation by recording their pronunciation,
checking their pronunciation diagnosis,
and providing exaggerated feedback.