Hello and welcome!
My name is Bolat Tleubayev and today I would like to present you a demonstration of a CoWriting
Kazakh: Learning a New Script with a Robot project. This work was done by
Bolat Tleubayev, Zhanel Zhexenova, Thibault Asselborn, Wafa Johal, Pierre Dillenbourg, and Anara Sandygulova
This interdisciplinary project aims to assess and manage the associated risks with transition
of Kazakh language from Cyrillic to Latin in Kazakhstan in order to address the following challenges
a) teaching and motivating children to learn a new script and its associated handwriting,
and b) training and providing support for all demographic groups, especially senior
generation.
We present the system demonstration that proposes to assist and motivate children to
learn a new script using a humanoid robot and a tablet with stylus.
The system consists of three main components: handwriting recognition, learning by teaching and user adaptation
Now let us consider the first component - Handwriting Recognition
The handwriting recognition was trained on over a 120,000 samples of the Cyrillic-MNIST
dataset , which we collected for the first time for the Cyrillic-based languages.
Now let us consider the second component of the system - Learning by Teaching
A humanoid robot plays a unique social role based on the original CoWriter project’s
LbT paradigm.
The robot asks for help to learn Kazakh language but since it cannot read Cyrillic script,
a child becomes robot’s teacher who is committed to try her best to write the words in Latin
so that the robot can read and learn Kazakh.
Such approach allows to keep a child engaged in
the learning process that we utilized in the experiment with 67 children
Now let use see LbT activity in action
Now you can see robot issuing instructions to the user
A user inserts a word in Cyrillic script
A system then recognizes and converts the Cyrillic script to a new Latin Kazakh Script
The third component is user adaptation
In order to dynamically adapt to each child’s individual handwriting style and to provide
an alternative to current robot font, trajectories of human handwriting
are collected in real time are used in Learning by Demonstration and Reinforcement
Learning approaches.
In this video you can see a user inserting a word
a system then is trying to adapt its font size respectively
when a user inserts a word in a smaller font
the system adapts its font too
This work was supported by the Nazarbayev University Collaborative Research Program
grant, "CoWriting Kazakh: Learning a New Script with a Robot"
This research was also funded by the Swiss State Secretariat for Education, Research
and Innovation as a parts of the CIS Region Seeding Grants,
and the NCCR Robotics.
From this slide you can see the references that were used in this work