A lot of the voice and tech space
interfaces available to an end user today are monolingual.
And a lot of them only support English.
In multilingual populations though, a significant portion
of naturalistic interactions and conversations
often mix more than one language.
When talking to a Virtual Agent users
need to take a few trials to learn
to modulate their own speech and language
to match an assistant's capabilities.
But what if we didn't always have to do that?
What if we could talk to our agents freely mixing languages,
or technically what is called code-mixing, like they often
do with other people?
Generating such good mixed language
computationally isn't as hard as it used to be,
but it's still a challenging problem.
That's because there isn't one recipe for good code-mixing.
We've noticed a wide range of differing attitudes
towards code-mixing, both across societies and individuals.
And that got us interested in characterizing a good policy
for deciding when to code-mix and how much to code-mix.
So we designed two policies and tested
them to see if they improve user experience.
The first policy is simple, always code-mix.
This makes the presence of mixing very salient
and gives us a conservative estimate of whether users
do want their bots to code-mix.
The second policy, called the nudge policy,
tries to be more adaptive and initiates code-mixing.
But it then converges to the degree of code-mixing
that the user reciprocates, including
the case of no mixing.
We divide the participants into two groups for each
of the code-mixing policies.
In each group the participant first
chats with an English bot and then with a code-mixing bot.
They provide ratings on humanness
and conversational ability of both the systems.
We also ask some open-ended questions,
at the end of the study, to understand their evaluations
of the interactions.
To achieve seamless, real-time, and adequate mixing,
we designed human-in-the-loop prototypes.
Where individual messages between the user
and the English bot were translated by human wizards,
based on instructions from the mixing policies and the nudge
algorithm.
The same process was followed for the baseline
as well, to ensure a fair comparison.
After the study, a lot of the users
expressed feeling more comfortable,
noting that the conversations flow better,
and the bot being friendlier when it code-mixed.
But that was far from a universal sentiment.
Participants also reported feeling disconcerted
because of the code-mixing.
The two code-mixing policies scored
an average of one point more on a seven point scale.
The more proficient a participant was in Hindi,
the more likely they were to rate code-mixing better.
Comparing the two code-mixing policies, while the nudge
policy accommodates to the user better,
we did not observe significant differences
in the user experience.
One explanation for both being, more or less,
equal is that users who would rather not code-mix
were a minority in our sample.
As our participants had a high proficiency in Hindi
and were generally favorable to code-mixing.
So what are the key insights that our results suggest?
Code-mixing works in improving user experience.
Always code-mixing is a high-reward, but also
high-risk, strategy and assumes a favorable attitude
towards mixing from the users.
This may or may not be a valid assumption,
even in some highly multilingual user demographics.
Nudging works across user preferences
and is more suited to scenarios with new user demographics.
It seems optimal to adopt a nudge
policy for the first few turns, and move to a static policy
later on.
The nudge policy and the rest of the framework
can also be extended to other dimensions
of conversational style, like formality.
To learn more about our work, you can find the paper
at bit.ly/CodeMixingBots or contact the authors.