- Hello, and welcome everyone.
So this is work I did together with my colleagues
Santanu Pal and Josef Van Genabith,
as well as Antonio Kruger.
We were dealing with the translation process.
Currently, when you do translation,
what you usually do is you take an input text
in source language, you give it to professional
and highly trained translator and he gives you the output
in a different language.
Since neural machine translation has gotten very good
in recent years, what you can also do
is take the same text, pass it to machine translation
and already get a very high quality proposal
in the target language.
Of course, you cannot publish that right away.
It's not that good yet,
and you still need a human to actually confirm
that it is correct.
But what you can do is pass this text alongside the source
to the human professional translator
and he just basically edits the mistakes
made by the machine translation.
This process is called post-editing
and it was shown to not only save time,
but to increase the quality of the final translation.
Post-editing is still done in interfaces
that are called computer-aided translation interfaces
and these come from the time well from translation
from stretch so on the left side you see
the source language segments,
on the right side you see the target
where you basically type the text.
Around that you have several support tools,
like translation memories,
machine translations, and all that.
As you see, these interfaces are still very focusing
on mouse and keyboard which made sense
because you were mainly producing text.
However it was shown that the time savings you achieve
for post-editing strongly depend on the interface.
If you do, for example,
post-editing on paper or on a screen.
Also, in post-editing, you don't have that many
mouse and keyboard events anymore
because you only need to correct.
So the task basically changes from a control setting
where you really did all the heavy load yourself,
to supervising the machine setting.
The question that we are concerned with in this work
is how does this affect
the design of the translation environments?
Should they still look the same they look nowadays?
Which interactions would actually be best used for
which task in post-editing?
So we want to figure out on a modality level,
like touch and pit for example.
Which modality do professional translators
think is appropriate for which post-editing task?
But since these are not professionals in interaction design
we want to regard common modalities
that they might not be aware of
that we currently use in the HCI literature.
Then we also want to know what they would envision
an ideal translation environment to look like
and in the end we had this side question
on cognitive load measurements
because post-editing was shown to be a very demanding task
and there's a lot of research going on,
on measuring cognitive load,
so we were concerned with can we somehow adapt to that
and what would be good approaches here?
There is some literature on different modalities
for post-editing, but actually not much.
Speech was used, but only as a substitute for the keyboard
so as another input for dictating the text
instead of typing it.
Eye-tracking was also used,
but more on the logging side analysis on the data,
but not as an explicit input modality.
Mobile post-editing, on a mobile phone was also explored
where you could re-order words with a touch
and also dictate with speech.
And pens have also been used, however only indirectly
in a separate area for writing that was then again
just translated to typing events
that were put at the cruisal position.
So what we want to figure out is what in a systematic way
which post-editing tasks might be
best supported by which modality.
So for this, we are doing an elicitation study.
For those of you that are not aware of elicitation studies,
the idea is to avoid technical thinking
and really making participants assume that everything
could be perfectly recognized to not show them a prototype,
but to really elicit the ideas.
So you present them with referents,
or things that you want to achieve
like there's a word missing, you want to edit
and the participants should propose
actions to achieve these referents.
This was done in many, many studies and was shown
to lead to interfaces that have
a higher increase immediate usage
than highly inter-rated interfaces.
So we thought that might be a good first step
to explore multi-modal interactions for post-editing.
Of course, the referents are quite important here
so we needed to find a good set of referents first.
For this we looked at the literature on post-editing tasks
and MT error classifications and came up with a set
that we thought might well capture
the interaction perspective.
So first of all, stuff could be missing
on the machine translation output,
so you need a way to add that.
Re-ordering errors often occur,
so sometimes a word is in the wrong position
or even a whole subphrase is in the wrong position.
You need to replace single words
when the wrong lexical choice was made
by the machine translation.
You would like another word there or replace part,
especially the verb form is often done wrong.
And of course, you need to delete single words
or groups of words.
So with that, we went into a study.
We first did an initial questionnaire
with some demographics,
but we also asked for complexity ratings
on how difficult the participants
would envision these referents to be.
Then we went through referent by referent
through the study and first did
what we call unbiased normal elicitation
where they only were asked to propose a single modality
and we didn't bias them in any way.
We didn't tell them what would be feasible,
what options would be there.
We really just wanted to know, what are your ideas,
to gather their ideas and to make them think broadly.
And of course, we asked them to rate each proposal
because we forced them to come up with something
so they might not be convinced
that it's actually a good idea.
Afterwards, since it was shown that such elicitation
as we do first yield to some legacy bias,
especially if you don't do it with interaction experts,
but with translators, as in this case.
So we did a biased multi-elicitation, as we call it.
So we first biased them by introducing
a common set of modalities
and then asked them to propose
as many things as they would like and to rate those.
We wanted to get some more common ground
and get the feedback on the major modalities.
Then we quickly talked about the CAT set up,
how many screens would you like,
how should they be arranged,
what would you like to see?
Since this of course, is also impacted
by the modality set you choose.
And afterwords we talked about such possible
cognitive load adaptations on if they think
that might help them and what adaptations
might be well accepted and which won't.
Participants were 13 professional translators,
eight in-house translators, and five freelance translators
covering a broad range of ages and experience
in both translating, in general, but also in CAT usage.
On average, they translated three language pairs
and had used more than four CAT tools.
Their CAT knowledge was rated as either very good or good.
However, the post-editing knowledge
was less good which is common
because even though post-editing is faster
and yields to better translations,
many still don't do it since they've don't it
another way for 30 years, so it's still not that
well established within the industry.
However, they need to switch
and now they get this pressure,
so that's why it's important to look into it.
Regarding the complexity ratings,
initially without really talking about everything in detail,
we found that re-ordering was seen as most difficult,
which is interesting because it's something that's
intuitively difficult to do with a mouse and keyboard
and especially for re-ordering, for example,
touch might come in handy.
That seems to be in their mental models.
In this first unbiased elicitation,
we got 91 proposals, one per participant and referent
and most were actually dealing with speech or touch.
So either those alone or some combination
with another modality.
We see that the ratings were quite high, actually,
so they really like their proposals
and also those subject to feedback was very good on that.
Why speech and touch were probably
due to them small terms that they use every day
so that's why they probably thought about it.
We also see that mouse and keyboard
were proposed quite often
so some participants were simply unable to come up
with another idea than what they do every day.
If we cluster this into tasks,
we see that for the addition tasks, mostly speech or touch
combined with speech were proposed for re-ordering.
It was mostly touch and they were
really enthusiastic about that.
For replacement, they wanted to use speech
for allegiency of the two.
We also see if the code ream and rate
which is a measure in elicitation studies
was really high for the replacing tasks
which suggests that these two can actually be seen
as a single interaction.
So maybe we would have needed
to split that up into two things.
In general, we found a medium agreement
of quite some multi-modal suggestions,
but as already said we see a strong legacy bias
not only in the fact that mouse and keyboard
were still proposed even though we told them
try to come up with something cool.
But especially in the fact that all the things they proposed
were of the form, select something
and then do something with it
which you do with mouse and keyboard
which would not be necessary in other modalities.
Then we did the second study
to get rid of some of this legacy bias.
So we proposed some com modalities to them
which were these that you see on the slide,
mouse-keyboard, eye tracking, digital pen or stylus,
speech input, touch, gestures, or any combination of those.
And then we taught them to again, we'll talk
about every referent and tell us how you would do it.
Now they were able to propose multiple things
so we got on average two proposals per participant
and referent which we classed that into 18 modalities.
Now the main difference was that they
really, really liked the pen.
They were really enthusiastic about having pen interfaces
which makes sense since translators
often need to proofread the work of others.
So they're quite used to having a text and checking it
for correction which you usually do with a pen.
And still they liked touch and speech quite a lot
and also combinations of those.
Mouse and keyboard was not proposed anymore
which shows us legacy bias was reduced quite a bit.
No one proposed eyes or gestures or if they did propose it,
they rated it very low, showing us that these gestures
are probably not a good idea for this task.
If you want to figure out how they proposed to interact
with the pen, look at the paper.
There's a lot of information on that in there.
In general, we can cluster it for tasks
that require no text production.
It was mainly proposed to use the pen or touch
while for text production tasks, touch was not proposed,
but only pen, speech, or combinations like this,
like pointing at something and speaking replace with.
So we see a lot lower legacy bias here
as I already said, and we see quite a vast amount
of multi-modal suggestions,
especially considering that literature.
In terms of interface,
the results were not that surprising I would say.
They wanted a single screen, relatively large,
tiltable, that you could operate
in seated and standing position.
And they had some interesting ideas,
but those were not common,
like having hardware buttons to interact with,
to confirm segments, or interacting with your feet
for deletions and stuff like that because they sit all day
and had some ideas like that,
but they were not really agreed on.
In terms of interface setup,
it was a bit more interesting.
We had quite some discussions on how you should arrange
the source and target on the screen.
They were mainly agreeing on having the current segment
a lot larger than the other one,
so you could better interact with a pen or touch with it.
They wanted a lot of browser functionalities
so they don't need to switch windows a lot,
so seeing corpurs or something
directly within the interface
and they didn't really agree on where to arrange what
so you should probably make movable support tools
that you can arrange yourself.
They also had some nice fancy ideas,
like words that you look up should be projected ambiently
or seen 3-D visualizations of the technical machine
that the text is currently about.
But again, these were just some small proposals.
So then we talked about cognitive load adaptations.
Why is that important?
Because post-editing is very demanding.
You need to not only consider the source and translator,
but you need to look at the source,
the surrounding context, the target audience.
If all these support tools like translation memories,
get multiple machine translations, so there's a lot of stuff
you need to consider at the same time
infusing a lot of load on your working memory.
So that's why a lot of research is going on,
on trying to measure cognitive load.
So we wanted to find out how could we adapt to that
if we are able to measure it?
How can we make sure that they are more motivated
or are less fatigued after working for a while?
So we quickly talked about several adaptation proposals
that they and we came up with
and what they really liked was to see additional resources
when the cognitive load rises or to get
an estimate of cognitive load on unseen text
which is especially interesting
since they are currently just paid on words,
no matter if it was difficult or not for them.
They thought that suggesting breaks or getting the segments
in a different order was okay-ish, but not too great.
And they didn't like the idea
to make it more simple when the load rises.
In general, they found it very exciting
and offering room for lots of improvement,
but they also found it frightening
and feeling very manipulative.
So to summarize, we argued that the post-editing tasks
strongly defers from the translation task
and therefore investigate different modalities.
We found especially pen, speech, and touch
to be important and would like to investigate those further.
Eye tracking and gestures we didn't see as too important
and we've got some ideas on how the interface
could look like and how it could react
to cognitive load adaptations.
But of course, this is all just an elicitation study,
so currently we're implementing that
and seeing how this could be used in the future.
Thank you.
(audience clapping)
- All right, any questions for Nico?
Okay, I have a question.
So my question is,
your elicitation study
and your future work.
You talk about lots of interaction schemes,
like touch, the pen based stylus, and this 3-D stuff.
Have you ever thought about using the human knowledge
to intervene the machine translation technology?
- What do you mean by intervene the translation technology?
- To intervene, like the machine translation
has some automatic algorithms, right?
- Yes.
- Have you ever thought about using the human intervene
the process of developing the algorithm?
- We do this, actually.
The machine translation outputs the so called beam letters
that you use to generate the final translation
and currently what you mostly do is
get the best translation out of that.
What we are also looking into is
should I maybe suggest, three proposals at the same time,
should I allow them to visualize what this beam letters
of the three on possible translations,
but at the same time, this was also shown to lead to
an even higher cognitive load
so it might help on the one hand,
but on the other hand you need to consider even more,
so we're also looking into that.
But for now what most post-editors do is
take one suggestion and edit that
and if it's a bad suggestion, say give me a new one.
- [Cammy] Cammy Vanya, University of Edinburgh,
interesting talk.
I'm interested in the fact that they are very interested
in the pen input, especially given that their whole world
and existence is because the machine translation output
is probabilistic and causes errors and one of the biggest
problems we have with pen input is
it's also probabilistic and causes errors.
So I'm curious if you got any feedback from them
about whether they had high faith in the pen input
actually producing the correct output on the screen.
- They really liked the pen.
They were really enthusiastic about it.
As we said, in elicitation studies you make them assume
that it would be perfectly recognized.
For example, for pens, they were also quite convinced
that that is possible, which was interesting
because as you say, that's not always great.
For eyes, for example, for eye tracking
they were not that confident.
They said it would require high concentration
and so on to really make sure there are not that many
recognition errors, but for pens themselves,
they thought this would work well.
- [Cammy] Thank you.
- So we'll see when we implement.
- Let's thank Nico again.
(audience clapping)