RAHUL: Hi.
Thank you for attending.
My name is Rahul.
I do iOS and Android at Pristine.
And Aaron Alaniz is with me as well,
and he does Android and Glass at Pristine.
How do I click Next?
AARON: Sorry about that.
RAHUL: OK.
So EyeSight is our HIPAA-compliant communications
solution.
It's powered by WebRTC.
And we try to keep the UI and user experience
with the physicians in mind.
So what does that mean?
That means minimal call clicks and a simple call flow.
And so why did we choose WebRTC?
There's cross-platform native support between platforms,
including Android and wearables, and iOS and browsers.
Now I'm going to dive into why and how
building a WebRTC client for Android or iOS
is different than the browser.
So here's a familiar diagram you guys have all probably seen.
The difference here is that we replace the browser
with libjingle PeerConnection, the JAR and the SO.
Now this is pretty much where the difference comes in.
I'm going to go into a little bit more detail here.
So what exactly are we building when
we go native for Android and iOS?
We need to compile PeerConnection for the target
platform, either iOS or Android.
There's two parts, a mobile layer and a native layer.
Now, the native layer consists of stuff like libvpx, libopus,
voice and audio engine, and bandwidth estimators.
And the mobile side, we have setting up
the camera for the platform, connecting
the interface and protocols.
And one thing I wanted to mention as well
is AppRTCDemo was very, very important for our development.
We needed to get AppRTCDemo working correctly.
And so we need to understand AppRTCDemo
before we can make our own WebRTC solution.
So as I was mentioning earlier, that we
had to make those libjingle files,
these platform-specific APIs are not
available for Android and iOS.
For example, there's no Maven or CocoaPods repo
where a normal Android or iOS user would just
expect to be there and then just pull it down immediately.
This means we need to build the libraries rather than expecting
the platform to support it.
And every revision change is a pretty expensive update.
We have to retest everything.
The call might not work after 90 minutes.
The audio might drop out.
There just could be several problems.
So we basically have to go through a long testing
process for each revision.
So we're very careful when we update revisions.
So here's the WebRTC build process that we follow through.
So if you're targeting an Android device or an iOS
device, for Android you'll need to install the Ubuntu build
packages, and for iOS you'll need
to install the Apple Xcode tools.
That's a prerequisite for every machine.
And then after that, we'll need to clone and pull
depot tools, which is our git and SVN client.
WebRTC has so many small libraries
that it's pulling from that we'll need a git and SVN
client together to work to pull all these products in sync.
Once we get the depot tools set up, we'll need to sync WebRTC
by setting the target platform to either iOS and Mac
or Android and Unix.
And then at that point, we have an option.
We can use a custom gclient file to add
in changes like camera or H.264.
And then we prepare WebRTC by running gclient runhooks.
At that point, we have all the make files there in place,
and we can run ninja AppRTCDemo, which
will build all the native code for AppRTCDemo.
This will ensure that we don't try
to compile code that's not built for the right target, which
will almost guarantee a failure.
And then you can publish this archive
into either an Archiva repo or CocoaPods
for an Android or iOS developer to pull down.
Or you can just use our build scripts that we open source.
They're really simple to use.
All you have to do is just pull down
our GitHub repo and just source the build script file.
And you can do build_apprtc for Android and dance for iPhone.
For the iOS version as well, I also
add an Xcode project that runs AppRTCDemo with an Xcode.
So you guys don't have to worry about all the complicated stuff
underneath the hood.
So I'm going to hand it off to Aaron,
who's going to explain a little bit more of the EyeSight Core.
AARON: Thanks, Rahul.
So we spent a lot of time familiarizing ourselves
with the WebRTC repository, the build process,
and then the artifacts that are sort of the byproduct of that
build process.
And we didn't want to walk away empty-handed.
So we set out to build what we call EyeSight Core, which
is a native WebRTC wrapper library that allows us to build
our client applications on top of this.
My talk's going to be mainly Android-centric,
but a lot of these principles we used
building out the iOS application as well.
So we set out to build the core with a few goals in mind.
And the main goal being, how do we solve this general problem?
How do we leverage the native WebRTC libraries to build out
a video streaming solution that'll
work cross-platform natively for all devices?
And as Rahul mentioned, we spent a lot
of time working with the AppRTCDemo, which in our case
is an actual Android AppRTCDemo and the iOS AppRTCDemo,
unlike the web version that's available to a lot of you guys.
And we poured over this code, and we broke it down
into three major components.
So we have to have a signaling module that
handles what we call the WebRTC dance to get a call up
and running.
And then we needed some sort of wrappers
to interact with the native libraries.
And then in addition to that, we have
to build out a client-specific UX for each platform.
So here's a bird's-eye view of the core in action.
So it's composed of two components.
So you have the signaling component,
which is interacting with our signaling server
and relaying stuff like the ICE candidates, the offers,
the answers, and sending them across the line.
And then we have our PeerConnection wrapper
that mediates the communication with our application
and with the native APIs.
And collectively, these two work together
to push out very client-consumable events
for each application.
So the iOS app doesn't really care
about receiving an ICE candidate.
It just really wants to know about is there a call incoming?
Is my call connected?
Did they hang up?
Did the call drop?
So we wanted to abstract all of the WebRTC dance
into very client-consumable events that
can be built off of.
So dive a little bit further into what's inside the core.
And as I mentioned, the libjingle
wrapper that we have mediates that communication
between our application and the native libraries.
In addition to that, since we're trying
to be very cross-platform with our application,
we have to configure these STP packets
for a different context of a call.
So you might have an iOS device calling a Glass device,
or an Android device calling a Glass device,
or a web browser calling a Glass device.
We handle all the STP mangling inside
of this portion of the core.
In addition to that, probably the most important feature
of our wrapper is that it encapsulates the interaction
with the native libraries in a way that's
friendly with the Android life cycle
so that users can opt into our app,
and they feel like they're inside of an Android app.
They feel like they're inside of an iOS app.
It's not just some video streaming application.
And the signaling wrapper is much, much simpler.
We also implement Socket.IO.
I know a lot of people have been talking
about how they use WebSockets.
So for Android, we use Android Async.
It's a very good, very fast Socket.IO implementation
for Android.
And on iOS, we use AZSocketIO.
And this is much simpler.
It's just a message dispatcher.
It doesn't compose messages.
It just gets a message from our PeerConnection wrapper
and sends it across the line.
And I want to just show a brief example of what
I look at on a day-to-day basis compared
to what the standard WebRTC API looks like in the JavaScript.
And as you can see, it's not really much different.
Clearly, don't think that you can build something
off this right here.
It's just a short example.
But I just wanted to show the similarities between the two.
OK, so there are a handful of WebRTC solutions out there.
And I really wanted to speak to our work on Glass
because I feel like that's what really sets us apart
as a technology company.
So before that, though, we have to talk about some
of the constraints that come with developing
on a platform like Google Glass, or any other wearable
device for that matter.
And some of the obvious things are these hardware bottlenecks
that you don't really have to think about whenever-- well,
you have to think about a little bit,
but not definitely as much for something
like a wearable device.
So these glasses, you know, physicians
are walking around their Wi-Fi networks.
And they're walking in and out of good and bad Wi-Fi spots.
I know the guy from that last team
made a really good comment about most of the time
it's just a problem with the Wi-Fi.
But they don't really care.
So we have to build around that, especially with Glass,
because the response time to Wi-Fi events
is definitely a little bit more delayed
than a traditional Android or iOS device.
In addition to that, the CPU is a real main concern of ours.
So we use VP8 to obviously encode and decode
frames inside our core library.
And we see great performance between an iOS
and a regular Android device because the CPU power
is there to do the VP8 encoding and decoding.
But we definitely see a bottleneck effect
whenever Glass is forced to do this sort of VP8 encoding
in software.
We are trying to work with Glass specifically
to try and get some H.264 support.
It's hard to guarantee any sort of great performance
improvements.
But we have been working with that as well.
In addition to all of that, battery is also
a huge concern of ours because like I said,
these devices are in constant movement.
They're walking from room to room.
In some cases, they're even in an ambulance.
So we have to build our app with the notion
that these applications need to last all day.
And in addition to all that-- UX limitations
for a wearable device.
It's completely different.
We have to really simplify the application for a wearable user
so that they don't really feel confused whenever
a call's coming in or they're in a call.
We really want them to feel like everything makes sense
while they're in a call.
So with that, I'm going to talk about some features
that we built out specifically for Glass.
Yes, I have thank our business dev guys
because this is really good.
So I'm sure we all, in some capacity,
are familiar with the AppRTCDemo on the web.
And the standard demo is showing your local video
in the top right-hand corner, and then the remote video
gets most of the screen real estate.
We didn't really have a compelling use case
for showing the person calling a Glass device's video.
In addition to that, we didn't really
want to waste any decoding time trying
to decode these frames coming across the line from a client
user.
So what we decided to do was just only
send video from Glass.
And if you're calling a Glass device,
you don't have to worry about sending your video.
And Glass is not even going to receive it if you try it.
So this allows us to preserve battery life on Glass
and build a UX that make sense.
And so, on the left there you can
see the early implementation of our app.
And it has a black screen for the Glass user.
And then of course, across the line on the other side,
you see the game of Operation.
Very quickly we found that Glass users
want to see what they're sending.
A good example of this is when a surgeon's
operating in the room, their vision
is not necessarily looking at what is probably
the most important part of the call.
They might be looking straight ahead,
or they might be looking down like this.
So we added a viewfinder so that they can line up
their line of vision with what's going across the line
during a call.
And to add on to that, since we don't really
have a use case for showing local video when you're calling
a wearable device, whenever you call a wearable device on any
of our applications, you get the full screen
of that wearable device.
So on the left there, we have a nice picture
of our capital in Texas.
So you can get a feel for the entire screen real estate
from a wearable device.
And just to juxtapose that, we have an iOS to Android call.
And since we are sending video both ways,
you definitely want to show your local video
so that they can see what they're
sending across the line.
OK, so with that, I want to do a quick demo here.
Hopefully it goes OK.
So Rahul's got his glasses on, and I've got my glasses on.
He's going to call from his iOS device,
and I'm going to call from my Glass device.
RAHUL: OK, Glass, start EyeSight.
AARON: And as you can see, you can share voices with Glass.
RAHUL: It's good for testing.
It's really good for testing.
AARON: All right.
So I'm calling.
And you should see a little prompt come up.
And I apologize.
The lag is pretty apparent whenever
you're doing Android Screen Monitor here.
So I'm answering the call.
And there you go.
You see-- oh, I think I lost the call here.
Anyway, on the left there, you see
Rahul showing off the viewfinder.
And if he actually swipes back on the D-pad,
it'll actually disappear.
And we can use this to save battery life in situations
where you don't really need to line up a call.
When you slide forward, it brings it back up.
I'm wondering what happened to mine.
Oops.
I'm sorry.
I just took a picture of everyone.
Wrong application.
AUDIENCE: We all signed disclaimers.
AARON: Yeah.
Oh, interesting.
Anyways, so that's really all I have.
I'm going to wrap it up here.
So if you're interested in what we're doing at Pristine,
you can check out our engineering blog
at tech.pristine.io.
We like to blog about once or twice a month.
And at this time, I'll open the floor for any questions.