Hi, my name is Christina and I'm presenting to you on behalf of my co-authors our work
"BrainCoDe" - electroencephalography-based
comprehension detection during foreign language reading and listening.
We live in a world where we have easy access to media content in several languages.
People use those to learn and improve their language.
They read Japanese books or watch English TV series to extend their vocabulary knowledge.
Media content is shown to be a feasible tool for language learning.
Thanks to the globalization, foreign media formats are easily accessible for us.
We can choose from a large number of books and movies to pick the content of our choice
which will in itself increase our motivation to engage with the language.
Let's look at an example:
Imagine, you're a big fan of Italian culture and there's a Netflix documentary on the history of Pizza
- sounds pretty amazing to me -
To engage with the content you choose to watch it in Italian.
You know a little Italian and you enable the Italian subtitles to better understand the dialects.
Engaging with such content is always a balancing act between looking up every unknown word,
which is time-consuming and interrupting,
and just trying to follow the story and trying to guess everything you don't understand from the context of the scene.
The chances are high that there will be words and phrases that you will not understand.
So we asked ourselves: How can we support users to achieve effective language learning with media content?
The basis of every language learning is learning the vocabulary.
If you want to help the user and improve their vocabulary knowledge,
it is essential to find out which words they know and which they don't know
In the case of using media content to learn that means we need a method to assess the users' comprehension
while they engage with the content. This method needs to be implicit
so the user should not get interrupted
as interruptions can lead to problems in both the tasks you currently perform but also in the task you are interrupted with.
Also in the best case it's independent from a certain modality.
It means you can support reading or watching movies or listening to audiobooks likewise
and it does not only classify general language proficiency
so determining if you can or cannot speak a language, but can detect individual unknown words within one language.
For this purpose, we chose to look into the workings of EEG - Electroencephalography.
It's a non-invasive way to investigate the brain's responses that can work equally well for visual as well as auditory processing.
Over the last decade, EEG devices have shown the potential for non-medical use.
The development of research and consumer devices has increased and we now have wearables such as the EmotivEpoc (in the upper picture),
or self-made research prototypes such as the one by Bleichner and colleagues (in the lower picture).
Research achieved important improvements in building small, wireless, low-cost EEG devices with printed electrodes
or the integration into a baseball cap as you see here.
That makes an application of EEG in real-world scenarios sound less futuristic.
There are two main paths for evaluating EEG signals: one the change in frequency bands and two event-related potentials:
signal peaks occurring after a certain stimulus.
We decided to focus on those ERPs.
As you see here ERP signals are waveforms representing positive negative voltage fluctuations of the brain,
and as I said, they're always responses to given stimuli.
Stimuli can be sensory, for example hearing a sound,
motory, moving hand, or even cognitive events like induced cognitive load during a task.
Prior work in neuroscience has shown that a negative peak in the EEG signal after 100 milliseconds,
a so-called N100, can be an indicator for an unexpected auditory stimuli.
Similar, a negative peak around 400 milliseconds can mean
that a person has problems integrating a word into the context of a sentence.
In both cases, so-called pseudowords can be the reason for such brain response
pseudowords are letter combinations that sound like words but do not have actual meaning
like shum, dake, laip, or Jabberwocky.
We consider those two ERP components very promising
to also work for detecting unknown words during second language media consumption.
So our research question now is can we use EEG,
and in particular, the N100 and N400 component, to detect individual unknown words.
And does it work well across modalities? So can we achieve equally good results for reading
for example, subtitles during movies, as well as listening?
For this purpose, we performed a user study and we focused on two main conditions: reading and listening.
To see if our approach works equally well for both.
Participants read text on the screen in a format similar to subtitles and listened to audio content via headphones.
We picked a set of standardized texts and translated a subset in the participants' native language German to work as a baseline.
For the audio condition, we used a text-to-speech engine to transfer the standardized text into the audio form.
We manipulated the English texts so that every second or third sentence contained a difficult word.
We wanted to create a realistic scenario in which the participants did not lose overall text understanding
but are rather confronted with individual difficult words.
We confirmed our manipulations through two types of vocabulary translation tests:
one in which individual words needed to be translated and
one in which the words are shown in the context of the sentence to see if the meaning could be guessed from the context.
Here are two sentences as excerpts from our text so that you can understand our image manipulations even better.
One is: "Sometimes the Inuit seethed their food but often it was not cooked at all."
While "seethed" means something like "simmer",
it is not a very common word for non-native English speakers.
For this sentence here, it could have easily been something like chopping or preparing the food.
The second example would be:
"For two hundred years French became the language used by English peerage"
While peerage is another term for English nobility, it's also not very frequently used by non-native English speakers.
We recruited 16 participants for our user study aged between 20 and 55.
They went through a reading and a listening condition
and were always at first presented with a native baseline, then read and listened to each two texts.
The order of the text was randomized between the participants.
We furthermore presented a set of German and English individual easy and difficult words
to recognize individual differences in the onset of the ERPs between the participants.
We used a 32 channel EEG device and tracked the users' gaze during reading.
We used four of the electrodes for the EOG.
We took them out of the 10-20 alignment and attached them around the eyes of the participants.
They are used to detect noise in the signals originating in electric potentials generated by the muscles around the eye
which are used to move the eyes from left to right or up and down.
And they interfere with the actual signal so we cleaned the data based on the EOG signals we received.
The data was then bandpass filtered: high pass with 0.2 Hz and low pass with 35 Hz.
We sliced the resulting signal stream into the individual epochs for each perceived words;
for audio based on the timestamps of the audio file, and for reading based on the gaze data.
We removed all epochs containing faulty word difficulty assessments
that means all words we considered easy, which were actually difficult for the participants,
and all words we considered difficult but that the participants could actually translate.
For the analysis, we decided to use only one electrode the Cz in the middle of the head
to create a realistic use case since one electrode could in the future be easily embedded
for example in headphones or in a baseball cap.
Our results now show that we can actually measure
a significant difference in the amplitudes of the ERP signals in this case for the auditory condition,
the N100, between words the user knows and the user doesn't know.
You can see in the figure here,
that we have a significant difference in the negativity of the peaks around 100 milliseconds after the stimulus onset.
The same could be shown for the reading condition.
You here have the figure of the N400 - 400 milliseconds after the stimulus
we have a lower amplitude for unknown words in the mean as for known words.
To evaluate the feasibility of ERPs for real-time application, we trained a classifier to detect the unknown words.
We trained a support vector machine (SVN) with a radial kernel and performed a K = 5 cross-validation
where K-1 folds were iteratively used for training and the remaining for evaluation.
As classification features, we used the signal amplitude in each epoch at 100 ms for the audio
and 400 ms for the visual condition.
With this classifier, we are able to detect unknown words with a mean accuracy of
82% for listening
and 87% for reading
However, we have a slight misclassification in the cases marked in red here
where we sometimes classify words the user knows as actually unknown.
But this misclassification is actually neglectable
since in language learning that would just result in repeating a word that the user already knows
and repetition in itself is good for language learning.
Our BrainCoDe approach is currently facing a set of limitations:
On the one hand, EEG hardware and software still needs to improve when we aim to apply it in a real-world environment.
We have currently performed a user study in the lab environment where it was quiet and participants were shielded from other external stimuli.
We still need to confirm if our approach would work equally well for a more complex situation.
Bleichner and colleagues already showed some promising results in detecting other ERP components,
in particular, the N300 which could be assessed in real-world situations with pretty good results.
Also, we still have a user-dependent training phase,
so right now we need to record user data while we know that some words are difficult and then can apply
our measurements to the actual data
but single-trial ERP analysis is advancing really quickly
and shows promising results so that we can hope to apply this in the future and get rid of this training phase.
So now that we can reliably detect unknown words using BrainCoDe,
we aim to in the future apply our approach for three main scenarios:
One to provide personalized learning content such as vocabulary lists that were generated
from the comprehension problems during your last for a movie.
Two: to build proficiency-aware interfaces.
For example, to provide playback options
if we detect a series of comprehension problems during the movie you've been watching and
And to implement content recommender systems based on your proficiency level.
For example, if you're new to a language
you could rather watch Friends,
which has pretty easy conversations and if you progress you can watch more complex content
Thank you very much for your attention.
For more information on our work please refer to our paper or contact us.
Thank you very much