>> How about making drones even more intelligent by bringing
the AI intelligence develops and
trained into Cloud onto the drones themselves.
That's exactly what Neal did combining Azure Percept and a drone.
That's today on The IoT Show.
[MUSIC]
Hi, everyone. This is the IoT Show.
I'm Olivier your host and today we'll talk drones.
Who doesn't like a good drone?
For that we have Neal
invited to the show today. Neal, how are you?
>> Doing well. Thank you for having me.
I'm really thrilled to talk about
drone technology and the Intelligent Edge today.
>> I hear and we can see behind you,
you've put together an Azure Percept decay with a drone.
Before we get to the point and people I'm
sure already excited about learning more about this,
more than just a little experiment,
tell us about yourself and how you came to
mixing and matching these technologies.
>> Sure. I'm a Microsoft principal Cloud Solution Architect
on the AI Ranger team.
I'm also certified commercial drone pilot
and to those drone pilots out there known as
an FAA part 107 certification for small unmanned aircraft systems.
But we'll just call it drone today.
>> Well, for most of the audience I'm guessing this is
going to be what's going to trigger their interests anyways.
First before I need to ask which one is your job?
Are you a Technical Specialist for Microsoft or
drone pilots and or both or is it just for fun?
What's going on here?
>> My day job Cloud Solution
Architect working on Intelligent Edge solutions and I've done
a lot of side work
incredibly enthusiastic and passionate about drone technology.
Autonomous drones, custom made drones,
how to move drones into different spaces
commercially as well as for a hobby.
I took my day job and my passion put them together,
and we have an opportunity to talk about
some intelligent solutions with the Azure Development Kit.
>> We very often have the imagination going on.
What could be done with drone?
We have vision videos of things that could be done.
I remember actually a Build Keynote
with Sam George like three years ago,
where we had a drone flying over
a set of pipes and detecting an anomaly down there.
Interestingly at that time that's not faked with almost,
the drone was just there to capture
the image and then analytics was happening somewhere else.
But here you combine the Azure Percept device
which has capabilities to do AI,
basically onboard where the camera,
microphones and so on,
with a drone to send out that intelligent device
up on top of what
your roof or someone's roof to do some inspection.
Tell us a bit about this project
and then we'll walk through to video
when you give us a go and
we'll talk about this montage of yours here.
>> Excellent. I was inspired by
that inspection use case that we saw on stage.
What I've done here is created a roof inspection use case.
This one happens to be for
a residential roofs but it can apply to commercial roofs,
it can commercially apply to other properties.
But what I did is I really looked
forward to creating a use case to
not only inspect residential roofs
which is accessible to a very large audience,
but applicable to another audience that might want to
take up a solution like this for commercial groups.
Now, a lot of fun doing it
>> I bet combining your work was a passion as very often.
This's one of the things that's working in IT would get
to do and very often.
Let's roll that video and what we're
watching it tells us what we're looking at right now.
>> Sure. In this video what
we're looking at is several different components.
We're looking at a map to
the right which contains a series of waypoints noted by
the green pins and we're looking at
some instrumentation on the left
which is coming from the drone itself.
Now there's two components,
there's the drone and the Intelligent Edge,
the Percept on the drone itself.
This route that I've created over this property
is programmed and uploaded to the drone flight computer itself.
When I take off and set a routes and tell it to go,
it autonomously flies over the roof without any need for control
via a controller in my hand or another pilot doing this.
It's really enables the drone to go across
the roof without any real input.
We can concentrate on the results
that are coming from the Percept.
Now as we move this forward,
we'll see a little icon.
It's a drone and we've reached
the roof where we're actually looking through
a web stream coming from the Percepts
and we're noting areas on this roof.
I have programmed a custom vision model that look for scrapes,
lifting of shingles and discoloration.
>> You basically send out your drone over the roof,
capture images and train the model to detect
the anomalies it would say or
the potential problems on the roof, right?
>> Right, exactly.
>> Here we're looking at the web stream,
which is one of the features of Azure Percept,
allowing you to eventually have a live view of what the address
Percept [inaudible] is seeing and how it's
analyzing the image to detect these problems,
these errors based on the model.
You would not send that web stream
up to the web from the drone.
>> No. What we're looking at is
web stream coming from the drone itself.
On the Edge we are using inference
coming from the drone real-time via the camera
that's actually picking up images or video or
recording the roof condition itself.
We're not recording or grabbing
video and sending it up to Azure for inference.
It's all happening on the Edge,
it's all happening in the Percept or on
the Percept on the drone in real time.
We're able to look at that web stream
and see what's happening in real time.
>> That's pretty interesting because
someone will assume that analyzing images,
detecting anomalies on an image is something
that is having terms of compute.
They would need some beefy graphics code
on a big machine server or not,
or use the Cloud for that.
But here you're embedding that muddle thanks to
Azure Percepts and the technology is
underneath Azure IoT Edge and so forth,
in order to run that on the drone itself.
Tell me a bit about why
would someone want to run AI directly on the drone?
What are the use cases we can see for that?
>> Typically in an inspection use case you
might have a drone that goes up and captures video.
Then that video is downloaded from the drone.
It might sit in an office in a backlog,
might have to have an engineer
interpret the results of that video,
and then a delay in results or a delay in our report.
What we're looking at now is again real-time inspection,
real-time identification of damage and defects,
and that can be shown based
on this video recording and provide it to
a client much faster
than waiting for a recording to be inspected,
waiting for a recording to be viewed or interpreted,
and then report compiled and then sent to the customer.
I think we've cut down the workflow significantly.
Also you have the opportunity to show
additional detail that maybe the inspector or
the individual piloting the drone and watching the video
later just didn't catch just based on how the model is trained.
>> Yeah, totally and I can imagine also the operator,
the inspector was receiving the messages about,
hey, there's a problem here,
and eventually wants to do a second pass or wants to
take control of the drone to go look closer
exactly at that spot that has been detected as faulty,
and do that because they have the data immediately
after that drone went through it's routes
and was able to give you that information locally, right?
>> Exactly. There's a couple of other pieces of
information that can help with this report and
really inform and help
the operator understand what they're looking at.
Since the Percept connects to the IoT Hub,
we're receiving telemetry from the model of
identification of damage or defects,
we are bounding boxes in this example.
We can correlate damage and defects to location,
we can potentially GPS tag those and we can have
better evidence of where
those damage and defect items are found on the roof.
Now I think that again increases speed of workflow
to going from initial looking at the roof to providing a report.
In addition we can improve the model,
we can iterate on training the model and provide
additional information about a different defect.
Let's say the spectra of wanting to look
for shingles that were completely removed or gone.
You can iterate on that model you can retrain the model and
improve it to also identify missing shingles.
You can today look for a problem set of
items and then tomorrow after retraining
this model which happens very rapidly via Azure Custom Vision,
you can enhance what you're
looking for or enhance the results that you're looking for.
An example could be you're doing
residential roofs in a neighborhood,
tomorrow you might be in a disaster zone or you might be in
an area where you're inspecting for damage based on a hail storm.
>> Now you're talking to the drone pilots,
the professional ones,
the ones who are making money out of that
because they're doing these different jobs.
How hard is it for someone
was a drone pilot typically and not like
you savvy in Cloud technology or embedded technologies?
How hard would that be for them to push a new model and
update the software on the drone itself?
>> Very easy. I went from zero to prototype within
a couple days and what I
used is Azure Custom Vision which is a no code solution.
What I did is I took photos,
uploaded those photos to Azure Custom Vision,
I tagged the areas,
the scrapes, the lifting of shingles, the discoloration.
I tagged those all within the web interface.
I saved that model and I trained it,
and after that the Percept studio
in Azure itself handled all of the deployment.
I didn't have to download a model onto an SD card,
I didn't have to connect via USB cable,
I didn't have to do some of
the other types of activities that takes to
update companion computers or
other types of technology that are now used in drones.
It's very simple.
>> Cool. It seems indeed pretty
simple to push your models to the drone,
in that case to the Azure Percept device.
That's opens up for tons of new scenarios.
Here we saw a real-life scenario,
which is inspection of the rooftops,
but there are other ones that your customers
that you're talking to you daily are interested in.
Can you mention some of them?
>> Sure. We're working with a couple of
customers in the agriculture space.
We're looking at inspecting
crops for damage especially after storms.
Also in the agriculture space,
looking for areas where
irrigation may not be 100 percent correct,
may have puddles of water.
You might have other areas that are not getting enough water,
so using an elevated platform like a drone
to go and look at the entire crop as opposed to using
an insurance adjuster to walk the crops or or drives through
various areas has saved a lot of time in that area.
Also we're talking to other one of the company
that runs theme parks
across the country and actually across the world.
One of the things that we're talking about is
ride maintenance or predictive maintenance.
Now the current use case is an individual
climbs very high spaces rollercoasters things like that,
other rides and inspect some pretty long intervals,
30 days, maybe more.
Now we've explored the idea of
sending a drone up using
the Percepts to inspect the tracks or even
using a route like what we saw for
residential inspection or residential roof inspection where we
can autonomously have the drone
traverse the tracks and look for Hotspots,
damage, rust various other things.
We're really exploring a lot of different spaces right now.
A couple of other areas where we're thinking about,
pipeline inspection, we saw that
example on stage a couple of years ago.
We also are looking at cell tower inspection.
The reason why a lot of these use cases have come
about is because there's a safety aspect
even either an individual needs to
climb a ladder to a very high place,
there's fall hazards,
there's other hazards with EMF or other electrical emissions,
radio towers might be damaging to human beings or
just microwave and other areas
or is it drone it might make more sense
to send a craft up like that and take a look around
as opposed to having individuals go up very high.
>> You were saying the scenario of the theme park
inspecting the rides is something
that's an individual will go once every 30 day,
because it's physically hard,
there's like lots of places to go,
you have to set up, just stopped everything or do it
at night and then it takes time to get up there.
The drone is going to be like up there.
Then you can inspect more often as well.
You can have an inspection daily versus once every 30 days.
The other aspect of the scenarios you're describing that triggered
some reaction where and when you're in
a remote place you don't have Internet connectivity,
there's no network connectivity.
This is where we're bringing it back to the reason of having
the intelligence running directly on the device itself,
that you can go in remote places and have
the inspection happen whether
it's what we're saying could be power lines,
it could be rubs in a gas extraction area or sites.
This things will require
recording the images and then do all the analytics,
afterwards it was delays and so on,
while here inspector goes,
runs the drone,
goes through the whole inspection and then tells,
"Hey, you have to fix that now.
Don't go back and come back next week,
do it now," or so forth.
That's the scenarios that
a reaction that your examples triggers for me.
>> Yeah, great example.
Just expand a little bit on the ride example.
Safety lockouts, you have to make sure the ride is not running.
One of the reasons why it's every 30 days,
because that's the window for
those safety lockouts and it's also not prime time.
If you fly a drone
under an autonomous path collecting all of that data,
number one, it can go faster than individual walking the track's.
Number two, not as many,
I shouldn't say not as many safety lockouts,
but not the need for such an
elongated session to lock the ride out.
Since the drone is faster,
it might take 45 minutes as
opposed to an entire day to take the ride down.
>> That makes plenty of sense and I think there's
plenty of other scenarios we can definitely
apply this requirements or advantages in that case.
That's great Neal, combining
Azure Precepts Edge Intelligence with a drone,
we've seen it, but here it's becoming real.
I appreciate your time and coming to the show today.
You did put together a blog post
that goes into details into your projects.
We'll bring up a URL down there
aka.ms/iotshow/AzurePerceptDrone.
Neal thanks for your time once again.
You're welcome back to The IoT shows.
Everyone appreciate this project.
>> Absolutely we are working in many new things.
I really enjoyed my time and hope
to bring some new innovations forward, and hope to see you soon.
>> You will for sure.
My new innovation with IoT and Edge Intelligence and drones,
I'm in, no problem.
Everyone thanks for tuning in and
watching The IoT show. See you soon. Bye.
[MUSIC]