- Coming up on How We Built It, we kick off a new series
to learn from early adopters of Azure Synapse,
starting with global aviation tech company, GE Aviation,
who are evaluating Azure Synapse to drive the development
of predictive maintenance analytics
at scale to help airlines, and the industry overall,
get ahead of issues and optimize flight safety
and operational efficiency.
So, today, I'm joined by Luke Bowman,
a Senior Product Manager at GE Aviation's Digital Group,
joining us today from Austin, Texas.
Welcome to Microsoft Mechanics.
- Thanks for having me on the show.
- And by the way, if you're watching and new
to Azure Synapse,
it's Microsoft's limitless analytics platform
that really brings together enterprise data warehousing
and big data processing into a single service,
removing the traditional constraints for analyzing data
of all shapes and sizes, and you can learn more by watching
our introduction to Azure Synapse
at aka.ms/mechanicssynapse.
Now, GE Aviation, if you didn't know,
specializes in designing leading-edge tech
and digital solutions for the aviation industry.
So everything from specialized military engines
to the world's most powerful engine ever produced
for a commercial aircraft.
So, Luke, your team just evaluated Azure Synapse
really as part of the next-generation data
and analytics platform for GE Aviation's Digital Group.
What's behind this?
- Yeah, so, for the past 20 years, beyond building the tech,
GE Aviation's Digital Group specializes
in providing analytics for airlines
and others in the industry with a focus on safety analytics.
As you can see here, we have a data enabling layer
called Data Actuator, which we built off
of our flight data processing platform called EMS.
EMS currently is used by dozens
of the world's largest airlines to decode
and transform flight data recorded by sensors on planes
and provide analytics for safety, efficiency,
and maintenance purposes.
So as part of our architecture, we have a server
for ingesting data, and for each flight
we're capturing the raw time-series data
for the entire flight
from as many as 350,000 data parameters on the plane.
We combine the data with other sources at an airline
such as aircraft type, flight plans,
and other operational data,
as well as sources like forecasted and actual weather,
runway and airport data,
and all this is stored in Azure blog storage in SQL.
So after we ingest the data,
it runs through our compute layer,
which is called the distributed processing system,
where we run analytics over the dataset
using a library of over 10,000 predefined aviation analytics
that we've created over the past 20 years or so
for the detection of abnormal events
through all phases of flight.
And we can scale this processing horizontally as needed.
Each customer then has their own remote desktop client
to access the analytics, and they can also access APIs
to visualize the events in dashboards
or create other visualizations to make data-driven decisions
around their operations.
And as we migrate to Azure, we're running our instances
of EMS for customers in Azure IaaS.
- So, how do things change, then,
with Azure Synapse being utilized here?
- [Luke] Yeah, so now with Synapse,
we're able to pull our entire ecosystem of data from EMS
via a rest API into Synapse as a central location
to build analytics on that data at scale.
We can use our Spark clusters on Azure
as our big data engine for compute and analytics,
and then we can explore data in the lake
without being restricted by predefined schemas
and query directly over Parquet files in the lake
using our existing skills in T-SQL.
Then from there, we can output the resulting datasets
into our data warehouse and visualize the results
in Power BI, all from a single managed environment.
- So, can you walk through then
the process that you took with Azure Synapse?
- Yeah, so our first proof of concept
took a set of analytic models
developed to produce condition indicators
for an aircraft's pneumatic system
and ran these models against a sample set of flight records.
Here, you can see the Synapse workspace.
First, we created a pipeline in Synapse
in the Orchestrate tab to load the flight records
that have been decoded by EMS.
Using a standard copy data connector,
I'm able to retrieve all of the flight data,
which is transformed into a Parquet file
and stored in Azure Data Lake storage.
- Great, so now, the data, I think it's in,
it's ready to explore, but where do you go from here?
- Right, so once the data's in the Parquet format
and loaded in the data lake,
it's easy to run our Python analytics on it.
In the Develop tab,
we pointed to our existing pneumatic analytics
against the data,
and we provided some additional environment information
necessary to run the analytic, and then we submit the job
to the Spark pool for processing.
- [Jeremy] All right, so, I can see the Spark pool
and the Spark job here, but what's that doing exactly?
- Yeah, so the Spark job generates
about a dozen or so condition indicators.
You can think of these as floating point aggregate results,
where we've measured various aspects
of the time series flight data.
This model in particular is looking for parameters
relevant to the pneumatic system, which controls airflow
to and from the engines for other equipment in the airplane.
These results are output as a separate Parquet file
and persisted back to the Azure Data Lake storage
so that we can now run analytics on top of those results
as well, and then we have a third notebook written in Scala
to load the results into the SQL data pool,
This then does some reshaping of the data,
and it splits out the data into one fact table
per condition indicator.
We create a couple of dimension tables
that help us describe the aircraft associated
with each of these condition indicators,
like the fleet of the aircraft and the aircraft ID,
and now, when we execute the code,
it writes directly to our SQL pool,
and query and visualize the data using Power BI.
Power BI allows us to do faster
and more performant dashboard reporting.
- Now, you're able to actually visualize
and kinda predict potential maintenance issues, then,
with the aircraft's pneumatic system.
- Yeah, so before, we never had any way to do
this end-to-end process without multiple steps
in different environments.
Here, we're able to query and visualize the data
using Power BI.
The top graph here shows aggregated results
for one of those condition indicators.
So, analysts can then hone in on all of these spikes
and do more drill-down analysis as to why the spikes
are occurring and what corrective maintenance activity
may need to be taken.
Everywhere you see a spike
indicates a particular high-pressure valve
not being fully closed during the cruise phase of flight,
which may potentially need to be replaced.
- So, you touched on earlier the capabilities
also related to your data schema and the things that you did
to develop that in Synapse, but how does that change things?
- Yeah, so we can now use Azure Synapse to build new models
in the predictive maintenance space,
but also in other spaces like safety and efficiency,
where we have products.
We're opening up Azure Synapse to our data scientists
so that they can directly work on the data that we have.
We plan to open up this tool set to our team
of data scientists over the next couple of months.
- This is really great.
Thank you so much, Luke, for joining us
and really sharing your Azure Synapse evaluation
with us today, and if you liked hearing from GE Aviation,
please check out the rest of our How We Built It series
at aka.ms/azuresynapseseries
to learn from our early adopters.
Thanks for watching.
(upbeat music)