Hi everyone.
Today I'll tell you how to grow your own Babel Fish.
So it turns out that the metaphor is not really that original, especially if you've
seen Sam stock from IBM Watson, but I'll briefly describe
what the Babel Fish is.
Babel Fish is unfortunately a fictional animal from Hitchhiker's Guide
to the Galaxy by Douglas Adams, and according to the author
it is a small yellow leech-like, and probably the
oddest thing in the universe.
It feeds on brainwave energy and long story short, it is a universal translator.
The main problem is you actually have to stick one
into your ear to make it work, but once you do it, you'll
understand anything that is said to you in any language.
If you haven't read the book, I really recommend
you do.
Okay, so moving on.
About two years ago I become interested in WebRTC.
Thomas will give a talk later today, so I'll describe it really briefly.
It is an API that allows you to create peer-to-peer connections
between devices, not necessarily just browsers, and you
can basically send anything you want through such
a connection: audio, video, or data.
About a year ago I get a talk about interesting applications
of WebRTC which was basically showing what stuff you
can build with it, and I was looking for some applications
to show, and the thing is that for data, there
are lots of really interesting applications.
I mean, WebRTC doesn't -- I mean the protocol is not based
on TCP, but rather UDP, so we can do very low working,
you can do file sharing because you have a direct connection
between two devices, you don't have to send a file like
for example in Dropbox.
Maybe some of you have even heard my application called shadrock(?) which
is like airdrop, which ironically doesn't work on
safari, but you can actually do crazy stuff with the data
translator, like implement protocol in JavaScript, which
works in browser, and that's exactly what WebRTC does.
But in the case of audio and video, things are boring.
We have Skype for actually quite some time, and, you
know, WebRTC does exactly the same but it is not really
that interesting, right, so I didn't have anything to
show and I decided to write my own application with a
small technical twist, and that's what I'd like to show
you today.
Okay.
So it is time for the demo.
So just sit back, relax and enjoy.
demo played) Okay, so
that's it.
[applause].
Thank you.
So as you can see it is basically a universal translator as well, just like Babel
Fish.
The thing is that on the left-hand side of your
video, yeah, left, you can select the language you're speaking.
On the right-hand side you can see what the language
the other person speaks, in and there is one picture
that I haven't really shown, is that if the speech
recognition is completely messed up you can type the message
here and it will also be translated and sent to
the other person.
So it looks like kind of cool, right, but it is
actually nothing new.
This is a screenshot from a Skype translator by Microsoft.
It was released in December last year and it does exact the same thing.
It does it even better because in my application you
have to press the button to turn the speech recognition
on.
I later describe why it is necessary.
In this application you can just speak all the time,
and it will all the time recognise what you're saying.
There are some advantages over Skype.
The most obvious one is that it works in the browser.
The Skype application is obviously an application that
works only on windows.
It uses open standards for WebRTC, Skype is some proprietary protocols, but to be fair,
Microsoft is currently working on a version of Skype that
will work in the browser it will use in the next version
of WebRTC, I'm not sure about Skype translator,
but at least a normal version of Skype should work
in the browser.
But the most important thing I want to say is
that really, the first working version of it took one
day to write.
And it really shows how powerful today's browsers are and how powerful the APIs are.
And it is really not that hard to write something like
that.
So hopefully you're all wondering how it works.
So I'll go over it step by step, warning there will be
some code ahead, but not much fortunately so the first
thing is obviously send audio and video, and it is
done via WebRTC, so it is unfortunately not the simplest
and easiest API, so I am using a library to make
it easier.
The library at the time is called simple WebRTC, and
using it looks like this.
So that's like less than ten lines of code and that's all you actually
need to make your own Skype, or very basic version of Skype
application.
So I'll go it quickly.
First, you need to instantiate it, then you just pass IDs of
DOM elements where you want to display your video, and
video for other people.
You wait for ready to call event, which is when your browser gets access to your camera
and microphone, and then you have to call to join
the room, which will connect you to all the other peers
in the same room.
That's actually the most complicated part if
you'll use WebRTC directly, and that's just one call in
simple WebRTC library.
Yeah, that's it.
We're actually down to the audio and video part.
So, moving on to a bit more interesting part, that is speech recognition.
It is done using web speech recognition API, and using it looks
very similar to the code that we've just seen.
Again, you have to instantiate an object, you wait for a result
event which will be triggered whenever the speech recognition
actually has some results, and finally you go to the
start that will ask for your microphone and start speech
recognition.
Okay, so the next step is that speech recognition gives us a transcript of what
you just said, so we need to send it to Google Translate
to actually translate it.
It is just done using plain code.
So no closed here.
Once you have the original transcript and it is translation, you can send it to the
other person.
And the cool thing is that there is WebRTC data transfer
for that, so you don't need any servers, so normally
you'd send it using Ajax or web sockets or something
like that, but in the case of WebRTC you have direct
connection with the other person.
We don't need any server for that.
So, using the data channels, this is really easy.
You have just one to send directly to all, you specific the channel name, payload name
which is basically what you're sending, and the data
itself.
Yeah, that's it.
So the last step is, so we already have the original end translator transcript
on the other side, and now it is time to read it out loud
and we use again API for that.
It is called synth API, it looks like that.
You have this really strange object, I just have to instantiate, the language, the message
itself, and then you have this, and then you have
this speak method where you have to just pass this object
to, and that's actually, as you can see, hopefully
the individual steps of it are really simple,
in the real application there is a bit more code, for
example, for call handling for handling language switches
and so on, but yeah, that's actually the core of it.
There is not really that much code in the application.
If anyone is interested in the actual code, I'll give the
link to the code at the end of the talk.
The second part, I'd like to give you a very brief introduction into
web speech API, I just actually wanted to mention two
things.
So the first thing is that web speech API recognition
has two modes, the first one is one shot, and
the second one is continuous.
In the one shot mode, in the speech recognition, the algorhythmic will stop when
you want, which is great for giving commands and that's
the mode used in my application.
The other mode, which is more interesting, is the continuous mode and basically
it will even if you make a pause, it will just
continue to work, at least in theory.
So this one is great for dictating stuff, or for my application, with
me at least, where users -- two users have a conversation.
Unfortunately it doesn't really work that well.
And I'll tell why really soon.
The second thing is that according to the spec, the API itself is like
agnostic of the underlying speech recognition implementation,
so it basically doesn't implement -- it doesn't
specify how the speech recognition should work.
And you can have speech recognition on server, or on the client.
So in Chrome, it uses Google services, so it is
on the server, but it could be done locally.
So while the speech API is really awesome, it has some issues.
So if you'd like to experiment with it, I'll just, you know,
briefly describe what the issues are, so at least
you know what to expect.
So the first issue is that the speech recognition will stop if you talk for too
long, I'm not sure what the exact limit is, but if you talk
for a minute or over a minute, it will stop.
You'll get abort event, and yeah, that's actually -- the second
issue is actually quite the opposite.
It will stop if you don't talk for as long.
I'm talking about the continuous mode which should handle pauses, but if you will
make long enough pause, it will just stop.
Again, you'll get abort event, and there is a work around these
issues.
I mean, you can wait for the abort event and
just call start again, but unfortunately, in some cases,
after you call start, it will abort again and it will
end up with this, which makes the application unstable
and actually that's the reason why my application doesn't
use the continuous mode.
So the most important issue, I think, is that web
speech API doesn't work with WebRTC media streams, so
the thing is the specification was created probably
before WebRTC application, so that's why it doesn't
mention anything about media streams, and it means that
the web speech API can only recognise input from your
own mic, nothing else, really, and that causes a lot of
issues.
So the first thing is that, for example, in my
application, which is WebRTC, and web speech recognition
API, it will ask you twice for your mic, which is really
confusing for users.
So once it will ask for WebRTC, the second time with the web speech RPI, but
the most important thing is, like I said, it can only
recognise what you're saying.
You can't get remote audio stream from another person and just pass it into
web speech recognition object, right.
So if you could do it, you could basically just write an extension or
plug into any web application and perform the whole recognition
of whatever any other person has said on your
computer, which would be really cool.
Another thing is I'm not sure if you're aware, but using WebRTC you
can actually make phone calls or make or receive phone
calls.
You have to use gateway service, but it is possible,
and if you could pass like remote audio to speech
recognition, you could actually, for example, make an application
where you just call somebody, take audio from them and
just pass it for speech recognition locally.
So it will be pretty good for, for example, people with
hearing loss, which could basically create a transcript
on the phone call on the fly.
So yeah, unfortunately it doesn't work, but like I said, this web speech
spec doesn't really mention anything about it,
but actually it would be really easy to extend the specification
without breaking it down.
That's it.
You just have to pass audio stream to this speech recognition
object and it could just work.
So that's actually what Firefox is doing right now.
They're extending, it has been at least, extending this API a bit without breaking
it actually, I just was told that it should work
in Firefox 2.5, so hopefully it will sync soon.
And like the most obvious reason why this API is not really
more popular, it works only in Chrome.
And the issue is that it is not even like all chrome, desktop and mobile.
The issue is that for example there is a small bug in
mobile version of Chrome which doesn't allow you
to use speech recognition if you're WebRTC at the same time.
So unfortunately my app won't work on android.
But what can we do about these issues?
I mean, the first is we can ask them nicely to update
the spec, ask Google to fix the implementation, or we could
ask another browser mender to actually implement
the API.
And while the speech recognition itself is really
complicated problem, the technology is not really an
issue here.
Personal assistance or virtual assistance is really popular lately, and it turns other
that every browser actually has some implementation of
it.
So Google obviously has Google Now, Apple has
Siri, Firefox actually doesn't have anything, but they are
using a library called Pocket Sphinx, which does speech
recognition locally on the device.
But there is also another way.
We don't really need to wait for a browser vendors, so you probably remember this one.
Right?
So all the time I'm talking about making implementation
of the API, but it is actually possible to create
a virtual library that will have exactly the same API
as web speech API and just would use some third-party
service to perform the speech recognition, or do it
locally.
So again, like I said, the speech recognition
is like really complex and hard problem, but again,
as it turns out, there are plenty of companies that are
actually happy to write.
So Facebook has recently announced Facebook M assistant, yet another -- they
also obviously have technology for speech recognition.
There is another assistant called hound by SoundHound.
I'm not sure if you have heard about it but it looks
promising, BaiDu has announced one recently, there are
some other companies that do speech recognition, obviously,
and there is also we've just seen a demo, there
is IBM Watson which also has API performing speech
recognition.
Finally, there is an interesting project called Pocket
Sphinx JS which actually uses the library that is used
by Firefox and it compiles it using scripts into
JavaScript and this actually provides the fully
functional speech recognition library in your browser
and actually in any browser that supports WebRTC.
It doesn't have to support speech recognition.
So that's actually it.
I kind of hope that you saw that, you know, the whole implementation wasn't
really that hard.
Each step was really easy.
You can see we have today's browsers are really powerful
and I strongly encourage you to maybe instead of writing
yet another framework just try to experiment with them.
Also I kind of hope that once Firefox actually releases
a version that of their browser that has a web speech
implementation API, I mean web speech API implementation, it will just become more popular.
My name is size a Szymon and I work with these
companies.
That's all.
Thank you.