JINGTAO WANG: Hi, everyone.
My name is Jingtao Wang, and I'm a researcher in Google.
Before joining Google full time last year,
I was a professor at the University of Pittsburgh.
I have two primary research directions.
The first one is on-device machine learning.
The second one is the application
of machine learning in the context of education
and human learning.
Today, I'd like to share with you some ongoing projects
I'm working on as an intersection of those two
directions.
As we all know, human learning is
one of the most challenging and rewarding tasks.
Just imagine what kind of opportunities
you may have if you learn a new language,
or you learn a new musical instrument,
or you knew some emerging technologies
such as TensorFlow.
Meanwhile, learning is also equally challenging.
According to some surveys, more than 10 million people
signed up on MOOC courses such as Coursera.
But less than 7% of them could complete those online courses,
which means that it's non-trivial to learn.
At the same time, we have the requirement
to learn not just in classrooms.
We also need to learn in all kinds of informal environments.
Today, I'd like to show you some examples
about how to use machine learning to enable learning
on your wrist, in your palm, and on your walk,
during your fragmented time.
The first project I'd like to show you is called SmartRSVP.
The idea is how can we convert your smartwatch
into an intelligent learning device.
The first challenge to address will be the small display.
We know a typical smartwatch only has a 1.5 inch display.
It will only display two or three words a time.
It barely affords to read a sentence
on such a small screen.
How can we enable efficient learning?
Well, it depends on the task.
What if, for example, we break the task into bit-sized pieces,
such as we enable learning of flash cards on your wrist,
set up timely reminders and prompts
or checks on whether you understand certain content
in the flash card.
Another thing we like to address is,
how can we read long sentences on such a small watch?
The solution is a technology called RSVP.
The idea is displaying one word at a time
in a sequential manner so each word can be larger.
And it can be delayed faster, and you
can read each and every word without moving your eye gaze.
Here, we introduce two technologies
to facilitate such kind of speed reading on smartwatches.
The first one is we detect your visual attention.
We only display the next word when
you're looking at your screen.
We will pause the display once you are looking around
somewhere else.
The second one is we monitor your heart rate in real time
to know your attention.
If you have full attention, we display
the text at a faster speed.
If you do not have full attention,
we display it at a lower speed.
And according to the user study, it does much matter
when compared with a version without such kind of attention
tracking--
visual attention tracking and focus tracking.
The other project I will share with you
is about how to improve interaction
in large classrooms.
We know that large classrooms are pretty
popular in [INAUDIBLE] studies.
It's not uncommon to have a large class
with over 200 students.
How can we improve the interaction
in such an environment?
We designed a intelligent mobile app called CourseMIRROR.
It has three features.
The first feature is it will send you
a reminder automatically after each lecture,
asking you to reflect on the content
and remind you to ask some questions, if you have any,
about things you'd like to know more or clarify
with the instructor.
Furthermore, we will have a natural language dialogue
system to facilitate you to ask deeper, more specific questions
other than general questions that nothing
really confusing or gravity is confusing.
Gravity is confusing, but it is not a good question
because it's not specific enough.
We can give you feedback on how you can improve your question
to ask a question in a deeper, more specific way.
The third one is we want national language summarization
approach to aggregate and cluster,
for example, student responses, based
on importance and diversity.
So the instructors do not need to go
through hundreds of questions.
They only need to go through one dozen representative questions.
And this system has been deployed in classrooms
in the US and overseas.
More than 400 students have used this system
and give positive rating on this system.
And the next project is a project
I have been working on for over five years.
It's AttentiveLearner.
It's about making MOOC, which means Massive Open Online
Courses, learning more efficiently
on your mobile devices.
It has two components.
It uses the front camera to track the facial expression
of learners.
And it also uses back camera to track the physiological signal,
which, in this case, is heart rate variability of learners
when they are taking a class.
So we know how good the student is learning when
they are watching the videos.
And when we observe something special, such as learners
get distracted for either external events or struggle,
we have some intervention techniques,
such has adaptive remainder, adaptive reviewing,
to improve the learning outcome.
Details about this project can be
found at attentivelearner.com.
And here, I'd like to introduce the last and the most exciting
project.
It's a language learning project that
explores the intersection between AI
and [? ours ?] and culture.
This picture is a picture showing a language
called oracle bone script.
It's the origin of Chinese language
used by people 3,000 years ago.
And this is the only language that
has a [? distance ?] today.
Most other Asian languages died, and new language appears.
This language is the origin of today's Chinese language.
And we lack, surprisingly, from this table,
you can see the evolution of Chinese language
from the original oracle bone script
to today's simplified Chinese language.
You can see the original language
is highly hieroglyphic.
It looks like everyday objects.
And gradually, representation becomes more abstract.
Knowing our origin, knowing the hieroglyphic representation
of the language, can help us to have a deeper understanding
about today's language.
So we created an interactive exhibition.
And here is a higher level idea.
We ask the users to experience the process
of creating new characters based on everyday objects.
And we are also able to animate the character
you created in a natural scene.
So every object in the natural scene is a character.
It illustrates the hieroglyphic nature of characters.
It can help us to have a better understanding
about today's language.
And here are some of the technical details.
We used a multilayer convolutional neural network
to calculate the similarity between the character you
created and the image prompt shown on the screen.
At the same time, we also use a stroke correspondence algorithm
to know, for example, what's the corresponding meaning
of each part of the character you created and then animate
it.
Now, we try to build a connection between every object
and the meaning, the representation
of the Asian language.
By knowing such kind of connection,
it can help us to understand the language a little bit better.
And this project is launched in Google Developers Days, China,
2019, on September 10.
That was more than 20 days ago in Shanghai, China.
More than 1,000 participants had a live experience
aboard this demonstration.
And this demonstration includes two parts.
The first part, two large touchscreens.
On those touchscreens, you try to imitate
the ancestor of Chinese people.
Try to create oracle bone script based
on image prompt in a hieroglyphic manner.
And after a successful creation of a character,
that character will animate and compose intact entire scenes
on a larger screen.
It can build us a connection between everyday objects
and the corresponding meaning of those objects in oracle bone
script.
All right, so this is a broad overview
about some of my ongoing projects.
I want to emphasize that machine learning can definitely
help us to learn more efficiently,
learn in a more engaging way.
Learning does not necessarily happen always in classrooms.
It can happen on your wrist, in your palm, and on your way--
on your walk in an engaging manner.
Thank you.
[APPLAUSE]