[MUSIC PLAYING]
VIJAY RAVINDRAN: Virtual reality is an incredible medium
for teaching kids.
It's fun and engaging.
It's able to simulate real-life situations
and create a practice environment that's repeatable.
CHARACTER: Out the left, back--
VIJAY RAVINDRAN: As a digitally delivered therapy,
there's data that's simply not captured
in traditional therapy today.
Virtual reality can be used for therapy
by designing experiences that simulate the real world
or enable the practicing of skills
where practice can lead to better
performance in the real world.
[MUSIC PLAYING]
MAN: The New Reality, a look at the future
of consumer technology.
With your host, Poppy Crum.
[MUSIC PLAYING]
I'm Poppy Crum.
Welcome to the Arm New Reality series,
where we explore the future of consumer technology.
What happens to our brains when we interact with technology?
It's all about feedback.
My role at Stanford--
I actually teach a class I created that's neuroplasticity
in video gaming.
So it's a video game development class
that focuses on augmented reality,
closed-loop sensory neural technologies
because it's really about how that technology interacts
with us that's most important.
But that class also focuses on when
we engage with any technology, how does it change us?
Every technology we use for any period of time
transforms who we are.
Some of the games that my students have built in my class
have been focused on things like helping children with autism
learn to communicate or read emotions better.
10 years ago, how would you solve that problem?
How do you emulate emotions?
Transition to where we are today and what we can do.
They can build games where AI can read their emotions.
They can get feedback on what they're doing well
and on what dimension they can do better so that they can
learn signatures or ways of ascribing
that feedback to improvements in social communication.
You're getting data and insight that you didn't have before.
Having new channels of communicating and channels
of visualizing that data to us in real time
lets us do things with less error, more precision, more
flexibility, more capability, and insight
than we did before--
so many opportunities for how we engage, and how we learn,
and how we augment ourselves as humans.
POPPY CRUM: Floreo is a company harnessing
the power of 3D gaming and mobile VR in behavioral therapy
and education.
[CHIMES]
[MONKEY SCREECHING]
[CHIMES]
[RUSTLING]
VIJAY RAVINDRAN: Floreo is a virtual-reality technology
company that uses virtual reality
to deliver clinically designed therapy for kids
with developmental delays to help them develop
social and behavioral skills.
Floreo's inspired by my son.
His first experience with VR combined with his autism
gave the idea that virtual reality
could be a new interventional therapy medium
for kids with developmental delays.
For children with autism, they likely
are entered into different various therapy
programs that involve building social and behavioral skills.
For kids with autism, they often need help
in building those skills out.
They need more practice time.
And so the therapists, and special educators,
and, ultimately, parents, end up spending more time articulating
steps that otherwise would not need
to be articulated to a typically developing child.
One of the first things I learned building VR experiences
with my son, who was then wearing a headset,
was that I couldn't see what he was seeing.
He was making expressions.
He was looking around, and I had no idea what he was doing.
One of the first features in the Floreo system that I designed,
and that our team built, was to stream
a live view of what the child is seeing in virtual reality
to a tablet.
And from that tablet, we enable someone to be able to observe,
someone who's able to provide live coaching as they're
going through the lessons, and in cases, to puppeteer some
of the characters to create the social dynamics that
lead to practicing a skill.
I've been working on Floreo for over five years now.
This company started as a basement project.
I was self-funding, working with a therapist, a pediatrician,
and a software engineer to implement
an initial vision of what therapy could
look like in virtual reality.
CHARACTER: Hey.
VIJAY RAVINDRAN: In virtual reality,
you can create animated scenes with engaging characters that
can, essentially, train children how to develop better eye
contact, how to follow people's faces better,
how to read emotions.
CHARACTER: Stop looking at me.
VIJAY RAVINDRAN: You can simulate situations
that are difficult to do in the real world
or in a therapist's office.
[MUSIC PLAYING]
The work that Floreo does and the customers they
support really straddle health care and education.
Manhattan Children's Center has been one of the best customers
that we've worked with.
And they've also done a really great job involving
the parents along the way, and so they've
been able to also collect feedback from the parents
to help shape what Floreo is today.
[MUSIC PLAYING]
VIJAY RAVINDRAN: Imagine learning how to cross a street.
I can tell you how to cross a street in an office.
You can practice in the real world,
but the consequences of doing it wrong the first time
are severe.
In virtual reality, we can create a street,
we can create cars and traffic.
We can help train the individual to understand the walk
sign, judge risky behavior from cars,
and, ultimately, be able to visually practice
that skill that has, really, no margin
for error in the real world.
Floreo has been designed to be as affordable and accessible
as possible.
And so we combine smartphones, which are virtual reality
capable, with inexpensive plastic headsets with lenses
that range between $10 and $20, and then
use a tablet to be able to power that coaching experience.
So the Floreo system leverages the GPU
in the smartphone to render the virtual-reality scenes
in real time.
The Floreo system is built on top of the Unity 3D gaming
engine.
So we use Unity extensively in rendering our scenes,
as well as to use its networking capabilities to link
the smartphone and tablet.
You know, our idea only works because mobile smartphones
are virtual reality capable.
It's always been central to the Floreo mission
to enable the system to enable access
to reach rural parts of the country that
don't have in-person therapy centers where, simply, therapy
services are unavailable.
What we're able to do using smartphone-based virtual
reality, combined with having a coaching application that
can run as a telehealth application over the internet,
is enabling access broadly.
And so we have customers today using
LTE to get therapy service into the homes who might not even
have broadband, enabling professional therapy services
to be rendered, where, otherwise, the family
might be under an impossible logistical challenge.
The world of immersive technologies and experiences
is rapidly changing.
When I look at what's available today, technologically, versus
five years ago when I started Floreo,
it's really moving fast.
Virtual reality is an area where we've
developed quite a bit of skill in understanding how therapy
can be built and delivered.
And we see similar opportunities in the future
in augmented reality, and as systems become more lightweight
and batteries become longer-lasting,
that these can go from just being therapies
to real-time assistants that are actually helping
people navigate the real world.
The ultimate outcome of a system like Floreo
is that the child is able to translate
what they learn in virtual reality to the real world--
put it into practice.
[MUSIC PLAYING]
KETAN SHAH: So mobile technology has really
brought a new level of capabilities in the XR space.
To get these new platform experiences,
obviously, it requires a tremendous amount
of processing, low latency, quick graphics to the end user.
So Arm brings different types of experiences and enablement
of technology for VR.
Some of the things that we work on
is, how do we provide the highest-performance CPU
processing to meet the different types of workload requirements
there might be?
We also actually spend a considerable amount of time
enabling different types of graphics features.
Those could be things like foveated rendering, which
actually only render out the types of images
that you need where your eyes are looking
on the particular VR headset.
Arm is also enabling functionality through AI.
The developer can actually build their solution such
that it provides a deeper and richer experience.
Whether it's tracking your eyes or whether it's
doing other types of machine learning,
it can transform that overall experience,
and the content then gets rendered appropriately.
XR is a nascent market.
It's an area that Arm is spending
a lot of time looking at, how to enable computer vision
at the low power.
One of the things about low latency
is the ability to be able to do things at the edge.
The time it takes to send information from the cloud
down to the device can be hundreds of milliseconds.
What Arm does is really provides many of the VR functionality
right there on the edge so content
can actually reside on the device itself
where it gets rendered in the milliseconds versus it
being hundreds of milliseconds.
User engagement has been shown as better
when there's high-performance, low-power requirements
on the device, such that it doesn't heat up,
and actually, the content is rendered as quickly
as possible, where right now, we're
doing 1080p, or 2k per eye, we expect to see,
in the future, 4k-per-eye type of devices,
which means quite a bit of graphics
that need to be rendered.
One of the areas that Arm is investing in
is a technology called ray tracing.
This is about providing greater shadows
and details to the actual end user as content gets rendered.
You may say, why is this important?
Because at the end, people want to have that very
rich and immersive experience.
And we expect more of that to occur as time progresses.
[MUSIC PLAYING]
Thanks for joining us.
We'll see you in the next episode.
[MUSIC PLAYING]