Hello to everyone.
My name is Alessio Palmero Aprosio from Fondazione Bruno Kessler in Trento
and I'm going to present you a joint work with Stefano Menini and Sara Tonelli
which title is "Adaptive Complex Word Identification through False Friend Detection"
Let's build inclusive societies
one of the main steps to reach this goal
is to allow citizens to interact with the society in which they live
In particular, with the public administration
In this paradigm, citizens need to understand documents
even if they are not native speakers
Often, citizens with lower proficiency (for example, migrants)
are those that most need access to services
We focus on the task of complex word identification, that is:
given a text, predict which words challenge a target given population
Therefore, to let people have a better experience
in interacting with the public administrations
one way is to build systems that amend the texts
which need to be understood
by editing them or making them clearer
The goal of our work is to develop a system
that accepts as input a text in Italian
and the user mother language
There are also many shared tasks on the topic of complex word identification
A shared task is a competition
where teams of researchers are invited to solve a problem
and then a ranking is done and a winner is nominated
Standard approaches to complex word identification
only consider frequency of lemmas in the language
as an evaluation of complexity
But native and non-native speakers
consider different words as complex
We describe a novel, user-oriented and adaptive approach
that considers form similarity between words in different languages
Our targets are, in particular, non-native languages
We describe an approach based on false friends and cognates
False friends are words in two different languages
that are similar in form but semantically divergent
For example: "library" in English and "libreria" in Italian
Cognates, on the opposite, are words with the same meaning
and a similar spelling in two or more languages
For example: "colour" in English and "couleur" in French
In this paradigm, a user speaking English as first language
would consider the Italian word libreria as more difficult
than rarer words that have similar sound in English
This is why in our study
we consider false friends as complex words
This picture shows the architecture of our system.
Some parts of the pipeline need some pre-processing steps
that would be infeasible on-the-fly
and these are the parts with the blue background
Using the models generated by the offline phase
the pipeline can parse documents in real time
The white background graph shows the processing
that is done each time a new document needs to be parsed
First of all, a text in language L1
in our case Italian
and the user language L2 are given as input
Then, the text is pre-processed to get lemmas and content words
using a Natural Language Processing pipeline
The system then outputs the complex words found in the document
The yellow part of the pipeline is divided into pre-processing and online phases
In the pre-processing phase
for each pair of lemmas in L1 and L2, the two languages
we compute similarity measures using XXDICE, NED, and Jaro/Winkler
To do so, we use monolingual lists of lemmas
and then the similarity values are calculated all tokens against all tokens
resulting in millions of values for each pair of languages
This is why this computation is done offline
Each of the three measures has advantages and disadvantages
For example, Jaro/Winkler metric
boosts the similarity of words with the same root
As a result, two words that are close
according to a metric can be very different using another metric
To overcome this limitation
we balance the three metrics
by computing a coefficient for each of the three scores
tuning them on our manually annotated set
Then, for each content word, we identify a list of lemmas
that are similar in the two languages
In the green part
we train a Support Vector Machine model
The first phase is to align
the English, French, German and Spanish monolingual embeddings
to the Italian ones
using fastText vectors and multilingual dictionaries
Then, synonym lists from a dictionary are given
Then, the cosine similarity between the word in Italian
and the word in the language is calculated
as a measure of the semantic similarity between the two words
In addition to the cosine between two words
we also calculate the cosine
between the synonyms of the Italian word
and - respectively - the original Italian word
and the word in the user mother language
We then consider both the nearest
where the cosine is maximised
and the average value of the cosines of all the synonyms
Then a SVM classifier is trained
and is tuned using a 10-fold cross-validation
for every language pair
and the best model for every language
is used in the production pipeline
More details about the implementation can be found in the paper
Four datasets were manually created by a linguist
with experience in teaching a language for second learners
The datasets are then manually checked by another domain expert
and each dataset contains pairs of orthographically similar words
marked as either cognates or false friends
The language considered are English, French, German, and Spanish
with Italian as the main language
and all data is available on Github
This table shows the content of the dataset
for each language
and the number of cognates and false friends tokens in the dataset
whith the total, the sum of the two values
Obviously, the list of cognates is not meant to be exhaustive
This table shows the different accuracies
in the different configurations of our SVM classifier
In feature selection, Romance languages boost the accuracy of the classifier
As said, we use 10-fold cross-validation
to tune our parameters and extract these results
As you can see, if you use all the features the accuracy boosts
In this slide, you can see the results on a test set
in the various languages, with precision, recall, and F1
and obviously accuracy
We can see that our system works better on Spanish and English
This is due to the similarity between Italian and Spanish
and because the English dataset is bigger than the other ones
probably covering almost all the more common false friends
We randomly collect 500K sentences from a multi-domain Italian corpus
we remove all sentences containing at least one word from our training sets
resulting in a set of 24K sentences
LBI is the lexicon of Basic Italian, written by De Mauro
It contains the 7,500 easiest Italian words
split into three categories
fundamental Italian vocabulary, most frequently used, and highly familiar
The LBI-small row represents only the first set
Generally, the vocabulary-based approaches are less conservative
and award less frequent words
A strategy targeting the language of Italian non-native speakers
instead, seems to be more focused
This slide shows some example sentences
with different complex word identification strartegies
LBI-small output is in bold
while the result of our system is shown into brackets
In this example, "responsabile", "colpevole", "omicidio"
and "attualmente", "pausa", and "tournée"
are considered difficult by the LBI-small dataset
while "rende" and "attualmente" are false friends in English
and are considered complex by our system
For example "rende" is similar to the verb "to rend"
that is "rompere" in English
and "attulmente" is similar to "actually", that is a false friend in English
Also "prendendo" is a false friend in Spanish
as it means "accendere"
An online demo is available at this address
and is part of the framework for text simplification
developed in the Simpatico EU project
We can see it
Here you can enter some text, for example
"Oggi vado in libreria con alcuni amici"
You can click on "process text"
and then you can see a lot of linguistic annotations
Here there is the lexical simplification
You can see that "libreria"
is shown as a complex word, for example in English
if you click on English
if you click on French
the word "libreria" disappeared because
is not considered a complex word in Frensh
as is not a false friend
In conclusion we present an integrated pipeline
to identify complex words for language learners
adapting to the native language of the users
We describe a supervised approach to automatically detect
false friends in four language pairs
We show that standard frequency-based approaches
do not address the needs of users with different native languages
In the future
a more complex setting using the context would increase precision
We want to add more languages to our system
We also want to consider multiword expressions
as for now the system only consider single token expressions
And we want to involve users in the evaluation of the pipeline
for example allowing them to submit a feedback
to compare the standard and the personalised complex word identification
Thank you for your attention