[MUSIC PLAYING]
ERICA HANSON: Next up, we're going
to welcome the team from Germany.
They worked on a project called FreeSpeak.
And they're going to be sharing more about--
essentially, it's a product that helps people
with their public speaking skills
and is using machine learning.
So it's super cool.
And I wish I had used this before I spoke to all of you
today to make sure that my body language is working out.
So really excited to see this project.
And we also have special guest Laurence
from the TensorFlow team, who's going to be asking them
a couple questions.
So let's take a look at the project.
[VIDEO PLAYBACK]
- Giving a good presentation is really hard.
[MUSIC PLAYING]
Now imagine this.
You are currently in a selection interview
for a really big company.
You have two minutes in which you have to present yourself,
two minutes that will decide if you
will get approved to the company of your dreams or rejected.
One presentation that will have a big impact in the future.
When we looked around at our local university,
we noticed something.
Many tutors or people giving presentations
have one thing in common--
they know a lot and can provide lots of knowledge,
but oftentimes the audience just falls asleep.
We have asked experts in the field
of [INAUDIBLE] communication.
And what we have learned is that many people focus
on improving their content, but they
forget one of the most important aspects of presentation, body
language and emotions.
Now, a presentation without proper body language
and emotions is like foods without proper seasoning.
The problem is that our current education system just
doesn't teach us these important soft skills.
Getting professional soft skill training is really rare to find
and really expensive.
We wanted to change this.
[CLICK]
We created the software that records the presentation,
analyzes it, and gives personalized, individual
feedback and tips to improve your presentation.
Using Google Speech API and Google Cloud sentiment
analysis, we can analyze data about talking speed,
the recognized emotions in your voice,
or the content of your speech.
With TensorFlow, we can use machine learning
to analyze the body language and facial emotional expressions
in the video.
In the future, we could add individual training exercises
or analyze more aspects of a good presentation.
And with today's technical advancements,
the possibilities are endless.
[END PLAYBACK]
ERICA HANSON: Hello.
Hi, team FreeSpeak, dialing in all the way from Germany.
Hi, how are you doing?
ALMO GUNADYA SUTEDJO: Great.
MARIA POSPELOVA: Great.
WEI WEN QING: Great.
MARIA POSPELOVA: Thank you.
ERICA HANSON: Good.
WEI WEN QING: Thank you.
ERICA HANSON: And we're joined by Laurence as well,
from Google.
Laurence, good to see you.
How you doing?
LAURENCE MORONEY: Good to see you too.
I'm doing great.
And I'm really glad that I'm towards the end of the day.
So it's like I've gotten to see all these projects all day,
and they're just absolutely amazing.
It's so inspirational to see.
ERICA HANSON: Definitely.
Yeah, I'm so inspired.
So yeah, we'd love for you to introduce yourself.
And I know you took a look at the project.
So I'd love to hear more.
LAURENCE MORONEY: Yeah.
So yeah, I'm Laurence.
I work in the Google Brain, Google research
into machine intelligence team.
I lead AI advocacy there.
So I work very closely with TensorFlow,
and with researchers in TensorFlow,
and what people are doing around the world with TensorFlow.
And so I'm just really delighted to be here
to see a couple of these projects that
have used our technology to just build amazing stuff.
I don't have a cool sweater behind me like Todd though,
so I'm just going to have to have a boring background.
ERICA HANSON: That's fine.
You have your jacket, so all good.
LAURENCE MORONEY: Yeah, I got my jacket, so yeah.
ERICA HANSON: Nice.
Great, so I'll pass it over to you.
LAURENCE MORONEY: Yeah, so team FreeSpeak, I
do a lot of presentations myself.
And I'd wished I'd had something like this when
I was learning how to do them.
So just congratulations on the amazing work.
And--
MARIA POSPELOVA: Thank you.
LAURENCE MORONEY: --I have so many questions
that I'd like to ask, but we've only got time for a couple.
So I'm going to just dive right in.
And one of the things that really impressed me
about this project was the fact that you used multiple machine
learning models, right?
You have the object detection, you have emotional sentiment,
you have text sentiment.
How was it to try and orchestrate
all of these together to really build a solution?
MARIA POSPELOVA: So yeah, I would
like to answer that question.
Yeah, you're right, we have many different models.
And with models, I mean that we have many different features.
And like you already mentioned, our features
are, for example, hand detection, or facial expression
recognition, voice emotion recognition, and so on.
And each of our features use a different model.
And of course, tying all of them up in one single program
was a really big challenge for us, even
the hardest one during our project.
And to tie everything up into one program,
we used multithreading.
And we have chosen multithreading
because some of our features, such as hand detection or face
expression recognition, or speaking speed measurement,
and so on, had to run in parallel.
And of course, multithreading is the best choice.
But multithreading can get very complicated,
especially if there are many features that
have to work in parallel.
So our program sometimes crashed.
And we are sure that it crashed because of multithreading,
because we tested each of our features
in many different ways, independently from one another.
And we also adjusted the code [INAUDIBLE] program.
And those features are working perfectly.
Also, for some of our features, we used pre-trained models,
such as text sentiment noses or speech recognition
from Google Cloud.
So of course, those features were
working without any problems.
So to sum up, our features are working without any problems
or major mistakes.
So the biggest challenge was tying them up,
like I already said.
But even if multithreading was really hard for us,
we had a great opportunity to learn more about it,
and also machine learning.
And we had a lot of fun with it.
And we got nice experience.
So in the end, we could manage to fix all the issues
by trying out some workarounds and also simplifying
our program.
LAURENCE MORONEY: Well, I'm really
glad you had fun with it.
I'm happy to hear that part.
And to me, I really think the future of great AI apps
are where you have specialized models,
and you're building something that
orchestrates them together, instead of trying to build
a great big unified model.
And that was something that really stood out
in your project, how you did that.
So that was really, really cool.
But one more question I have on it
is that one of the things that you're doing
is you're measuring people's hand movements and sentiment
in their face.
And how did that work from a training perspective,
with different people's shapes and sizes?
People look different to each other, right?
How was it for training models?
ALMO GUNADYA SUTEDJO: Yes, that's a great question.
And so for our machine learning models,
We used publicly available open source data sets.
Because in our opinion, one of the hardest parts of machine
learning was getting good data.
Because good data is really a key to high accuracy
of the model.
And because the time of this challenge was limited,
we wanted the best possible data in the shortest amount of time.
And this is why we looked for already existing open source
projects and the data sets which they used.
So the benefit of this approach was
that we found large data sets that already contained
a huge variety of hands, faces, or the voices of many people.
So for example, one data set which we used
was the [INAUDIBLE] data set for voice emotion recognition.
And it already contained over 7,000 audio files
spoken by, I think, 24 different voice actors
that mimic different emotions.
And another example is the Oxford data set, which
is used for hand detection.
And it even contains over 10,000 different hand instances.
And they already had the data separated into training
and validation sets.
So this is why we didn't have to tackle
the problems of different shapes and sizes
as much, because we already found
large data sets with a huge variety of instances.
But still, of course, there were some problems
that we had to deal with.
For example, positioning the camera or the lighting
can, indeed, have an impact on the accuracy of our model.
This is something that we're still working on,
and this is why we are also trying
to make our models more robust.
LAURENCE MORONEY: Excellent, excellent work.
Erica, I could talk about this project all day.
But I know we have to take questions from other folks.
So maybe we can take one from the YouTube now.
ERICA HANSON: Sounds good.
Yeah, let's see if there's a question from the live chat.
Did you have any performance issues?
MARIA POSPELOVA: I would say yes.
Because starting up the project, the program
could take, like, one minute.
Because everything should be prepared,
every model should be prepared, and so on.
So yeah, that was also one of our biggest issues
to deal with.
ALMO GUNADYA SUTEDJO: Yes, and another problem
was maybe about going [INAUDIBLE] on mobile device,
for example.
Because it does indeed take a lot of computing power.
So we might have to host them on different servers.
But currently, when we are running them
on our local computers, there is also, for example,
a difference between running it on CPU or GPU [INAUDIBLE]..
Surprisingly, for example, if you're running on GPU,
even makes the startup time, or the program,
a little bit slower.
Because it has to allocate resources.
And it takes quite a few minutes to load up.
MARIA POSPELOVA: So it's different on our computers.
Like, the project on my computer starts much faster than,
for example, the project on Almo's computer.
LAURENCE MORONEY: We know who has the nice computer.
ERICA HANSON: Very well, thank you
so much for sharing with us.
We do have to wrap up, unfortunately.
I also want to hear more, because we
could talk about this project for a long time.
But great work on this.
Thank you so much for joining us.
And also, thank you to Laurence for being here with us.
And he'll be sticking around for the next project.
But great work, FreeSpeak team, awesome.
LAURENCE MORONEY: Thank you so much.
MARIA POSPELOVA: Thank you.
WEI WEN QING: Thank you.
LAURENCE MORONEY: An amazing project.
ALMO GUNADYA SUTEDJO: Thank you.
LAURENCE MORONEY: Great job.
[MUSIC PLAYING]