- Coming up on How We Built It,
we're joined again by GE Aviation,
this time to look at the steps that they're taking
to modernize their on-premises records management system
as they move incrementally to Azure
to make it easier for their airline customers
to respond to new requirements and regulations.
And we'll look at their implementation
of Azure's Custom Vision to improve the variety
and also the accuracy of document searches.
So I have on the show today Nate Hicks,
who's a Senior Product Manager
for GE Aviation's Digital Group.
Welcome to Microsoft Mechanics.
- Hi, and thanks for having me.
It's great to be with you.
- And thanks so much for joining us from Ohio today.
So especially in these times,
as flights are getting back up and running
and airlines are under increased pressure
to show that they can safely host passengers.
Whether that's checking the air filtration system
on planes due to public health requirements with COVID-19,
or also meeting industry standard
general maintenance requirements.
Really, being able to keep good records
is more important than ever.
- It sure is.
It can be a huge task across an airline's entire fleet
to record and then easily retrieve evidence
of maintenance activity
or compliance to an Airworthiness Directive.
A lot of what gets archived, things like parts receipts,
vendor airworthiness certificates
or maintenance records are still paper-based.
And these documents can contain both handwriting
as well as printed text.
And over the lifetime of an aircraft,
we're literally talking about pallets of paper.
So being able to convert documents
that are in paper format to digital format,
as well as process and store any documents
that were never printed in the first place
with scale, speed, and accuracy is more important than ever.
- So, how does the solution now work today?
- Right, so today, most of our customers
use our AirVault legacy Records Management product,
which GE acquired a few years ago.
And the best way for you to get an understanding
is for me to actually show you the user experience.
So when users first log in,
they will arrive at the landing page.
Documents get uploaded in a variety of different ways,
and they go through some processing steps,
and then they arrive in a customer's audit queue.
Then the documents get audited
in an application called Web Index.
From here, I can select a batch of scanned documents.
And when it opens, you can see that our system
has taken this document badge
containing four unique documents,
as indicated by the green bars at the top,
totaling 68 pages.
Customers can then manually populate the index fields
on the left or they can automate this index tagging
through integration with their maintenance system.
And then when they're done
they can publish this to the appropriate records repository.
On top of this legacy Records Management product,
we built an application called Asset Transfer System,
which provides both a project management
and a records interface.
In the project module, users can see projects
to which they're assigned as stakeholders.
They can create projects using industry standard checklists
for a wide variety of project types
to include things like aircraft transitions, annual reviews,
or even engine transitions, and much more.
The project module is enabled with the workflow
in a third-party audit tool,
where you can invite different parties
from different organizations to collaborate
on meeting specific document requirements.
And this really enables remote collaboration and review.
For example, here we are rejecting a document
and then creating a workflow issue.
Users can see and then interact with the aggregated list
of open issues for a project,
and they can get an at-a-glance view
of project completion status.
In the records module, users can search, view,
and transfer records at an asset level
using industry standard Spec 2,500 data formats.
So to summarize, our product portfolio
offers a wide range of product capabilities
that encompass everything
from digitizing paper records to searching, collecting,
and then transferring those records to third-party agencies.
- Okay, so given that a lot
of your infrastructure currently is on-premises,
how are you looking then to transform
what you have in place today?
- Right, so as you saw,
we have our legacy application.
And then we have modernized certain components of it
for things like Asset Transfer System.
Our ultimate goal is to rewrite everything
using Azure Kubernetes Services
so that we can have key microservices
that are easier to maintain and update the code base.
- What incremental steps then have you taken
to leverage Azure in a hybrid state then in the meantime?
- Well until now, we focused most of our efforts
on enhancing the operational core,
which is being able to automate the digitizing of records
where we index them and then we make them searchable.
At the end of the day, this is a production system
that needs to be able to process
about 200,000 files per day during peak times,
and we can't afford downtime.
So, we've been really thoughtful about
how we shift core services to the cloud
in a way that does not impact
our legacy application functionality.
To start with, we figured out a way to take advantage
of Azure Optical Character Recognition, or OCR,
which is part of the Azure Custom Vision,
by introducing a hybrid integration layer
that creates a bridge
between our legacy Open Source OCR system,
our database hosted in our private data center,
and Azure OCR.
So this way we can improve things like our OCR capability
and expand on what we had with the legacy solution,
by adding new capabilities like handwriting recognition.
- So how does this then all work?
- Sure, so the first step is the document scanning.
These scanned files get uploaded as image files.
And if they're PDFs,
we convert the pages to either a PNG image or a TIFF file,
depending on whether they're color or black and white.
These then go through a pre-processing pipeline
that is configured for each customer.
The whole goal of all these steps
is to make these documents ready for OCR
and then match up index values and metadata
with images to make them searchable.
For example, we auto-rotate the images
if they're upside down.
We also have a barcode reader
that will read all the barcodes on the page
and use that for index values,
and another tool that identifies whether
or not the images is a form.
So, indexing is then applied,
and from there, they enter a queue
passed to Azure via an API.
This API connects to Azure Cognitive Services
for the full text extraction.
At this point, the index values
and metadata are attached to the image
and passed back to our legacy AirVault System
via our integration translation layer, and stored on a SAN,
and the extracted text and metadata goes to a SQL database.
Once that is done, the documents are fully discoverable
and optimized for search in ATS or AirVault.
In this way, we are able to leverage Azure OCR capabilities
without requiring a system rewrite.
And in the future, of course,
our plan is to natively upload directly to Azure.
- Okay, so how long did it take you then
to get everything up and running and integrated with Azure?
- Right, so once we had the integration layer built,
it really took a couple of days
to get the Azure OCR Service itself configured and running.
- What kind of performance are you seeing
versus what you had before?
- Yeah, it's really impressive.
So, we're talking about 35% faster processing speeds,
and about 60% more accuracy in terms of the data
that it can extract.
And just to show you how much more accurate,
I'd like to show you a comparison
of pre and post Azure OCR.
Let me show you a comparison here for the same sample test
that we had with the same 10-page document.
We're here in SQL Server Management Studio.
On the left is the output from our legacy OCR system,
and on the right is the output of Azure OCR.
Each row here represents a full page of a text extraction.
And you'll notice a few things right away.
On the right, you'll see that Azure OCR
has uniformly captured the tabular data,
whereas this is more inconsistent on the left.
If we look into the document details,
we can see the impact of those inconsistencies.
So this is an example of an Engine Accessory Report.
It's basically an on/off log for parts coming into the shop
versus parts going out.
And it's a pretty standard format for maintenance,
repair, and overhaul facilities.
Pay attention to the Nomination column
as we compare results.
Our legacy OCR Engine has had some problems historically
with accurately capturing highly tabularized data,
especially with small printed text
within small fields on the table.
You'll see that in rows 230 to 280,
it's not accurately capturing the part nomenclature.
Also, the part numbers and serial numbers
are not always clearly captured and searchable.
And this is really important
when searching for serialized parts.
For example, if there's a part quality finding
and you need to find out
on which aircrafts it was installed.
If you look at row 260, the serial number is EGS023.
You'll see that the OCR read this as #68023.
Now let's compare those results
with Azure OCR that we've implemented.
Here, you can see the values that were missing
in rows 230 to 280 are clearly represented and searchable,
and the words are separated correctly.
Also our part numbers
and serial numbers are correctly captured.
If we look at the same row 260,
the serial number EGS023 has been read accurately.
- And that's quite a difference.
But, how do things work then with handwriting?
- Yeah, so I mentioned earlier
that our legacy solution really couldn't do handwriting
with any sort of accuracy.
So now I want to show you
the improvement that we have with Azure OCR.
I've opened our legacy AirVault web search application.
This is where our customers come in
to search their documents.
Here, they can search for a specific word
contained in a document.
In this case, I'll search for the word 'seat.'
This will take a minute, but to the human eye,
in some cases 'seat' looks like 'scent,'
but the machine learning is able to derive context
from the words around it,
for example, 'copilot seat.'
And just to show that this isn't a fluke,
I'm going to take a second word like 'insulation.'
You'll see it's returned highlighted results.
Again, the penmanship here is pretty hard to decipher,
but Azure OCR is able to find everywhere
'insulation' shows up on this dataset.
What's interesting too, is that beyond the handwriting,
it also works really well on printed text.
So if you zoom in on what's pretty distorted printed text,
and is 90 degrees out of rotation,
it has still managed to pick up the word 'insulation.'
And what's even more impressive is
that this is sandwiched between German language texts.
Now this is our test repository,
but in our production environment,
we have literally hundreds of millions
of documents in our system.
So this more advanced Azure OCR means
that we have a richer auditability
and searchability on our record sets.
- So, what else then do you have on your team
then in plan for Azure?
- Right, so we're currently also testing out
Azure Forms Recognizer.
Here in our process map,
you can see some of the processing steps we use
tailored to each customer.
We work with the customer
to identify specific document types.
Our current process is that we leverage Forms OCR,
which is a template-based solution.
So we take a series of sample documents from the customer,
and we'll mark up the specific document attributes
like Task Card, that you can see here highlighted in yellow.
This helps us to then identify the document type.
And we then annotate the specific key value data pairs
that we want to extract into index fields,
as you can see here in the blue shaded zones.
If you look at the work order,
the extracted text does not match
the handwritten value on the right, 17382.
Similarly, if you see the task card number,
the extracted text is missing a few characters.
Using Azure technology, I can upload a form in the same way,
but it goes through a simplified process map,
and has higher accuracy
with extracting printed and handwritten text.
So now, if we look at the results again,
the work order number was correctly transcribed,
and the task card number is completely captured.
Additionally, the Azure Forms Recognizer solution
enables a more efficient mechanism
for training the document classification.
And you can see here how the document type
was correctly identified as a task card.
And, as we streamline the process
for scanning and indexing our documents,
we'll be able to use Azure Search across our entire body
of both structured and unstructured data
for more powerful search results and data utilization.
- I know that GE Aviation's Digital Group
is also experimenting with Azure Synapse Analytics
for predictive maintenance,
so how will this help in your scenario in this case?
- Right, so ultimately,
our goal is going to be to move our system
and all of its data into Azure.
And when we do that, Azure Synapse will help us
to build analytical models
that allow us to view the records data
across our various parts.
- Thanks so much for joining us today, Nate.
And also, if you're interested in learning more
about GE's Records Management System,
check out AirVault.com.
And to learn more about Azure OCR, Cognitive Services,
Azure Search and Azure Synapse Analytics, visit Azure.com.
And if you haven't already,
be sure to subscribe to Microsoft Mechanics
for the latest tech updates across Microsoft.
Thanks for watching. We'll see you next time.
(upbeat music)