[Music]
my name is nick facey i'm the senior
program manager at unity accelerate
solutions
my name is uh jack sue i'm a senior
manager at boeing vancouver so jack uh
you know we got to do an incredible
project together working on ar for
inspection on large objects you wanted
to give us a high level overview from
your perspective uh what the project was
what technologies were used what
problems we ran into uh augmented
reality for aircraft inspection was a
project around looking at the emerging
technology of augmented reality to make
the aircraft inspection process more
efficient for the mechanic and engineer
aircraft inspection is something that
happens many many times through the life
of an aircraft and so our thinking was
that if we can
use
paint
augmented
sort of dot indicating locations of
defects dents and buckles on top of the
aircraft
that would allow the mechanic to see
existing damage much more easily and
also allow them to identify new damage
uh much quicker
and so we were taking a machine learning
approach to anchoring that 3d virtual
model on top of the aircraft that they
see
through the ar device
and what we found in the course of the
project is that to train a machine
learning model required a lot more data
that you could gather a lot more images
than you can gather by just going and
taking photographs of the aircraft and
instead we came up with using synthetic
data to generate those images for
training of that machine learning model
how were they currently doing the job
what was the process of the existing
inspections they say an aircraft is at a
gate and uh there's a sort of a cursory
inspection that occurs at a gate so a
mechanic may walk around with some paper
which he can note and annotate where a
new defect might have occurred there may
also be a listing of existing defects so
all defects uh
bird strikes lightning strikes those
types of things what's called dense and
buccal sometimes are recorded in a
database and so the inspector may print
out all the existing defects now
depending on you know the type of
inspection uh like an inspection that
gave is very cursory as i said and so
the mechanic may spend a lot of time
being much more much more carefully kind
of walking around the aircraft looking
at it they may actually get on cranes
and this type of thing to look at the
top of the aircraft or damage there
underneath the aircraft in more modern
airlines that have using remodeling
technology like ipads they
they could run on an application boeing
actually has an application called turn
time that basically supports this
process and so again it would be
typically with paper or maybe with a an
ipad application that they would carry
around we started with you know the real
world pictures uh of the plane that we
were taking our you know the project
team was taking
uh and then we realized we were gonna
run into a problem of not enough data to
train the machine learning with yeah
that's right we were taking a machine
learning model approach to solving that
anchoring problem we went actually out
to an airfield and took thousands of
images of an aircraft and also used a
particular algorithm to calculate the
position of the camera relative to the
aircraft we took those thousands of
images we fed them into the machine
learning algorithm
and got to learn you know how to
calculate position based on an image
that was coming from the camera but when
we actually went out to the field again
after we trained the model and and
tested it it didn't do a very good job
um and after a lot of analysis by our
partners at simon fraser university they
determined that the reason it wasn't
doing a very good job was that there
just wasn't enough data we didn't need
um
thousands of images we needed hundreds
of thousands of images a lot of
variation in viewpoint and also um the
algorithm is also using the sky and also
the background
and the markings on the aircraft to do
that pose estimation and so we need a
variation in those as well variation in
the markings of the aircraft in the
background in the types of you know uh
tarmac
that was the plane was sitting on
whether it's indoors or outdoors all of
those things need to be varied in terms
of the image that was used to train the
algorithm and
we just couldn't do that um in with
field data like we we couldn't create
that variation and we couldn't create
enough images very time consuming
a process to actually collect those
images and determine the uh the position
the xyz position of the camera uh to
really make it an operational process
and in terms of the next steps towards
the solution because we didn't stop
there so uh what do we do next so after
realizing that that's what we need to do
is we need to collect a lot more images
and a lot more images with uh variation
we thought hey we should use synthetic
data because
synthetic data is where you you create
these images using a gaming engine like
unity's gaming engine like creating
realistic synthetic images which
basically act um
as a proxy for real images you collect
in the field and feed that into the
machine learning algorithm because
you're doing uh you're creating
synthetic images in a virtual world it
means you can create
millions of images um and very quickly
compared to going out to the field and
actually taking pictures and so that's
why we uh we came to you guys to talk
about if you had the ability to do that
at unity and it turns out luckily that
you did that's where we engaged the
perception team uh and we started to
iterate on the exact items that you
talked about yeah it was it was it was
great actually because we found out to
you nick that um there's a team at unity
that that has been working on exactly
this problem creating synthetic images
using the graphics engine
and uh taking those images and feeding
the machine learning model and so what
your team was able to do was create
within unity and the perception toolkit
the ability to randomize the delivery or
the markings on the aircraft to you
actually created environments to place
the virtual aircraft into like inside a
hangar
different types of hangers
uh in outdoors in in cloudy conditions
in sunny conditions with uh different
tarmacs different ground covering and
different lighting conditions so sun at
a particular position a different type
of light inside the hangar and also at
hundreds of thousands of different
viewpoints that the camera could be
above the aircraft below the aircraft
where on the side of the aircraft left
side right side behind the tail and all
sorts of randomized kinds of environment
conditions and and markings of the
aircraft and take millions of pictures
of the environment
and then what we did is we took those
synthetic images and we fed them into
the machine learning algorithm um to see
what the impact is and the results were
very promising
so do you think it'd be safe to say like
the synthetic images is going to make
the possibility of ar for inspection of
these large aircraft like actually more
realities moving that bar forward
absolutely i don't think it's possible
without the use of synthetic images i
think that uh synthetics really has made
it a possibility to do this type of work
i think that without synthetics it would
not have been possible that was one of
our big lessons learned from the project
aside from time one of the big
advantages of synthetic data is rather
drastic reduction in cost to generate
all the images
yeah yeah absolutely it's uh
it's it's a huge uh decrease in cost i
mean um
i mean it's spent uh sfu and uh and
boeing we and some of our guys went out
to the field and you know it took them
something like seven or eight
hours to collect something like 3 000
images of the real aircraft once the
programming is done with unity in the
perception toolkit you know it would
take like three hours to generate a
hundred thousand images and potentially
more and have better quality in some
ways right and with more accuracy and so
there's really no comparison in terms of
costs so aside from inspection is there
somewhere else you think you can see uh
synthetic data going as a problem solver
yeah absolutely i think when it wherever
there is a requirement for data to to
drive machine learning there's a role
for synthetics and
that happens in in many areas one
example is in aviation one of the other
things that that different groups in
boeing have been trying to look at is
using machine learning to recognize
a defect and so for example
uh if you had an ipad camera or even a
drone that's flying around an aircraft
and sort of doing surveys of the
aircraft you want to be able to
automatically detect oh there's
something here there is a dent or a
buckle here at this location so you
you're giving it a bunch of images of
the aircraft and you're getting the
algorithm to recognize this is the
lightning strike this is the bird strike
and in order to do that work once again
with a machine learning algorithm you
have to feed it many many images of bird
strikes or lightning strikes in order
for the algorithm to recognize that this
is what they're seeing and so this is
where i see another opportunity that's
that's similar if we move away from
synthetic data um
what about kind of the real-time 3d
engine you mentioned the realism being
important you know what other use cases
do you think you can see in the
aerospace industry there airspace is
getting uh especially in our area and
the things we're involved in boeing
you know 3d models are becoming more and
more important and so people are
starting to really realize the value of
being able to see planes and components
and parts in 3d i mean it's it's being
used today
i'm sure as people know in the design of
aircraft in lots of areas in design
right in the design process but even
within the um the post-sale market the
the services market afterwards
3d models just enable so many things
allows the the end user to visualize
parts
to find location of parts in the future
we envision it to see underneath the
skin of of the aircraft the ability to
stream a 3d sort of graphics in a very a
performance efficient way
it's something that uh we're very
interested in in
talking about and and thinking about and
developing applications for what was it
like to work with the unity team the
accelerated solutions team the
perception team unity as a whole it was
uh it was fantastic i have to say it's
one of the uh the best partnerships i've
been involved in in my career um i
thought it was just fantastic how
uh you know about three quarters of the
way through the project uh two thirds of
the way through the project we decided
hey we gotta take a look at uh synthetic
data
and um and that ball was kind of picked
up by by you nick and the unity team and
say hey i think we can help with that i
found that you guys are just super
responsive
you put a team on this really quickly
you knew what we wanted to do
and uh you just got to work and and i
was uh
really pleased with um with how quickly
your team was able to build the
environments to brainstorm with us on
the best way to get this done and so i
thought it was a great great team effort
but i was uh really really pleased what
unity uh brought to the table awesome
jack and that was absolutely mutual on
our side awesome to work with boeing
awesome to work with sfu even though we
were all three very different companies
it felt like we were very aligned on
wanting to solve this problem together
[Music]
you