In this research we present EnglishBot, an AI-powered conversational system for
second language learning. I'm Sherry Ruan, a recent phd graduate in computer science from
Stanford University. This work was conducted in collaboration with researchers from Tsinghua
University and University of Washington. Why is English learning such an important area?
According to the British Council's report, English is currently studied by 1.5 billion people
all over the world. This means that one out of every five people on the planet is an ESL learner.
However, one-on-one tutors for language learning can be inaccessible and expensive. Although there
exist some inexpensive self-study resources online, most of the online resources for
speaking are listen-and-repeat based, as shown in the two screenshots here.
We hypothesize that chatbots can be a promising tool to address this and scale up language
learning for the following reasons. First, deep learning-enabled speech recognition advances have
significantly improved chatbot technology. Second, chatbots have enhanced distributed cognition and
social interaction abilities according to learning scientists. Third, chatbots could also provide a
friendly and non-intimidating setting for spoken language practice for shy language learners.
And lastly, chabots have proven to be more engaging and effective than
traditional interfaces in certain domains, such as mathematics and factual knowledge learning.
So, can we replicate the effectiveness and engagement of chatbots in language learning?
With that, we built EnglishBot, a language learning chatbot that converses with
native Chinese speakers interactively on college-related topics. We provide speech
recognition powered by deep learning, adaptive feedback on learners' natural language inputs,
and hints in learners' native language ,which is Chinese in our case.
Now, I'm going to work through each component of the EnglishBot system. First, we have a learning
material pane, which is comprised of two sections, vocabulary and sentence structures, designed to
assist users in completing the conversation with the chatbot. The conversation practice pane,
shown on the right here, serves as a major learning window where most of the learning actions
and interactions take place as well as all conversations between the user and EnglishBot.
EnglishBot's dialog bubbles indicated by the yellow boxes here consist of chatbot
utterances. The chatbot also comes with an audio play button which is used to replay the audio of
the English sentences and a translation button which translates English sentences into Chinese.
The user's dialog bubbles indicated by the dark grey boxes here contain the user's replies,
adaptive feedback from EnglishBot, a reference answer which can be clicked by the user to
display the reference answer, and the audio replay button that reads a reference answer to the user.
The next section indicated by the orange box here is a hint bar containing hints in the
user's native language. The bottom section indicated by the two dark blue boxes here
is the operation panel where the user records, edits, and sends replies to EnglishBot's prompts.
6a shows the first phase prompting the user to speak out loud to record
their response. The speech engine then transcribes user's responses to text.
6b shows a transcribed text. The user can click on the mic button to continue recording or
editing the text field because speech recognition may not be 100 percent accurate. In fact,
in our user study, the speech recognition engine achieved a 94.5 percent accuracy.
Here's how the complete interface looks like on a student's screen. There are
already two exchanges of conversations here. The current dialogue here is the fifth exchange.
To compare EnglishBot to more standard approaches,
we implemented a traditional style listen-and-repeat interface based on
popular speaking English learning software in china such as 7 English and Little English.
As shown here, this listen-and-repeat interface was designed to have the same color scheme and
a layout as EnglishBot. However, users cannot provide any conversational inputs to the system
which is a major constraint of the popular English learning software systems nowadays.
Hence, the only independent variable between the two systems is the interface: conversational UI
versus a traditional UI. We conducted two user studies in two different learning scenarios,
fixed usage and free usage, where users were randomly assigned to one of the two systems
and only used that system in their six day learning period so that we can trace people's
speaking English improvement to one of the two systems. Here's the actual user study procedure.
Users took a series of pre-tests before getting started including a free form and a script-based
speaking test. In the fixed usage study users were asked to complete all conversations in one unit
each day by following a strict order from beginning to end with no repetition.
This study may mimic the use of EnglishBot in a classroom setting.
In the free usage study, users were given access to one unit each day and allowed to complete as
many or as few conversations in the unit as they desired. This study examines the effectiveness of
EnglishBot in more real-world conditions and may better simulate the use of such a system outside
the classroom, such as during self-study. Here's the demographic information of users we recruited.
There were 56 college students with an average age of 23.4. They came from 22
different universities and 30 different majors. All were native Chinese speakers.
45 had never been in English-speaking countries before and 11 had been abroad for short term
travel purposes only. And lastly, user's average total time in English-speaking countries is 1.3
months. We aim to answer the following three research questions with our two user studies.
First, how engaging is the EnglishBot to learners in comparison to the listen-and-repeat system?
Second, given fixed usage, which system is more effective? And lastly, given voluntary usage,
which system is more effective? Now I'm going to present the results of our first user study,
the fixed usage study, where 28 people participated. First, we found that there
was no difference in engagement between the two systems in the fixed usage setting.
Users' speaking improvements were evaluated by speaking tests with graders. The grading was done
independently according to a grading rubric adapted from the IELTS speaking test across
four different metrics: fluency and coherence, grammatical range and accuracy, lexical resources,
and pronunciation. Here we found that there was no difference in speaking English improvement
between the two systems in any of the four measures in the fixed usage setting. Moreover,
EnglishBot took 2.5 times longer to practice than the traditional system when the usage was fixed.
So far, we haven't obtained much beneficial results for EnglishBot. Now I'd like to present
our results from the second user study, the free usage study, where 28 people participated.
First, we found that EnglishBot was more engaging than the traditional system in the
free usage setting, and the difference here was statistically significant.
Of their own violation, EnglishBot users spent 2.1 times more time than the traditional system users
and the difference here was statistically significant with a large effect size.
There was a significant improvement in learning gains on the script-based fluency
evaluation measure but not on the rest. This is a speculation but the results could have to do
with the duration of our study. Our study was only operated over a short time span,
and we know that language learning takes a lot of time to make progress.
Our results suggest more longitudinal work is needed to understand how to best create
and use interactive chatbot technology to support efficient and effective foreign language learning.
To summarize, in this work, we built a voice-based chatbot that helps college students improve their
spoken language and we conducted two user studies under free-usage and fixed-usage conditions.
We found that compared to the traditional system, EnglishBot engages students more
and motivates them to double the time spent in the free usage setting.
EnglishBot also improves students' speaking fluency more in the free usage setting. However,
under the fixed usage setting, EnglishBot achieves similar learning and engagement outcomes and is
less efficient than the traditional system. Our results suggest that conversational interfaces may
benefit foreign learners oral language learning, particularly under casual learning settings.
To learn more about our research, please visit hci.stanford.edu/research/smartprimer. Thank you!