- Hello.
My name is Justin Wilson and I'm a Senior Data Scientist and
Mitre Transportation Innovation Center.
Today. I'll be presenting
our presentation on
extending a big data platform for transportation
to space.
The core responsibility
of our department is to operate a big data analytics
platform.
We call this platform,
the transportation data platform or TDP for short.
My co authors are Marco Adelfio and Van Hara,
together, we make up the subset of our team currently
working on
extending TDP for space.
I'll start with an introduction to TDP and why Mitre
decided to develop this big data analytics platform.
Then I'll provide an overview of what is TDP,
including a brief description about the people process and
technology used to develop and support it.
And finally, I will highlight some extensions
that we've made to the platform so far for space and
potential collaborations in the future.
Please feel free to contact us directly if you'd like to
learn more. Marco is a Principal Data Scientist for TDP,
and Dan is the group lead in product owner of the platform.
Contact information from the last slide of the presentation.
Okay, so let's get started with why MITRE decided to develop
TDP.
The diagram to the left is an example analysis process
before or without TDP.
So imagine that you're a Data Scientist or Data Analyst
tasked to do an analysis with aviation and aerospace data.
First, you have to locate the data within your organization
or purchase it If it's not publicly available,
this can be costly from time lost or duplicate data
purchases.
Once you have access to the data you may need to prune or
filter it down to a manageable size. Transportation systems
can produce large complex datasets,
which is where big data capabilities are needed,
which can scale up computing resources as needed. Next data
preparation in the form of scrubbing fusion with other data
sets and generating derived data or metrics is needed.
Finally, after these processing steps,
you can start your analysis.
With tight deadlines
this often leads to limited time to coordinate or share best
practices, which may lead to duplicate efforts or
inconsistent results.
This can also lead to data silos or data islands that are
managed independently.
Each having its own technology stack requires skills and
expertise, In summary without coordination or reuse, teams
may be reinventing the wheel.
The figure to the right is the analytic process with or
after TDP.
We brought data islands together into a common,
big data analytics platform for consistency and reliability.
First, as you as the analyst, know that the data is
available in PDP, and if it's not,
you can ask our community of developers or analysts and we
can work together to incorporate it for next time and others
to use.
There's a consistent process for scrubbing fusion and
deriving metrics and it's an automated process
that's run daily in TDP
to prepare data for use.
The process is transparent.
Data and code is accessible and can be improved
and matured as a community.
Analyst start at the analysis step right away,
Everybody's starting at the same spot,
singing from the same sheet of music, same data,
same code libraries, same tools, and same documentation.
The figure to the right is an operational view of TDP.
In the center of the graphic we have a multi Petabyte Hadoop
Cluster or enterprise data warehouse.
We currently processes over 40 different data sources with
varying data formats daily in the platform.
For aviation TPS
process and archive of 10 years over 10 years of data
and for space a year has been processed so far with
the ability to access historical data or other data sets
based on business needs.
The bottom left of the figure shows the ability to access
data on premises or in the cloud using Jupyter
notebooks in our Python and other languages.
The bottom middle of the graphic shows about almost half of
MITRE projects for the federal aviation administration using
PDP.
And then the bottom right is our rest API and services
which enable dashboards to be driven by TDP data.
Some other highlights, it's evolved over the last decade,
mainly for FAA work, but we can benefit other domains
without the same upfront time and cost that was needed to
get TDP where it is today.
It's a collection of services.
So it's not just the hardware that was mentioned.
It's software code developed a MITRE,
scalable data science environments and access services.
It's built on an extensible architecture using open source
technologies, allowing the platform to continue to evolve.
And although the systems called the transportation data
platform, we've mainly focused on aviation until early
this year, but a lot of the software and architecture it is
domain agnostic.
TDP is the extended to support space, which we will now
discuss.
So in addition to engagements
to understand current needs
and capabilities, we started to prototype space and TDP.
The current state of our prototype for data includes
processing from space track.org for orbital main elements,
the unified data library for observations and conjunctions
and the international laser ranging service.
ILRS data.
The figure at the top right of the slide, shows our
automated workflow, their processes, the space data daily.
You can see the four data sources at the top.
We have also prototyped and orbit change detection method,
which shows platform's ability to generate derived data sets
from the raw data.
Derived data are depicted as ovals,
and then directed acyclic graph of that figure.
The bottom line figure shows an example of notebook that
allows analysts to quickly access and analyze data within
TDP.
We've been able to reuse many of the existing tools within
TDP, for example,
the Jupyter Hub environment figure in
the previous slide, as well as the data availability view
which we see in the top of this slide,
Data availability shows what date has been processed for
which dates and the location of the data.
It also shows the data versions as the platform evolves.
So we've been able to use the same tools
current TDP analysts are using for aviation
and we use them for space.
We've also extended our tools to support the space data such
as our flight viewer in the bottom right of the slide,
which now shows brown traces and associated data for each
space object.
We've also added to the TDP code base by adding dependencies
to internal/external code for integration with others
methods.
Finally,
we continue to engage within and outside of minor to the
develop other drive metrics,
to support aviation and space use cases.
Everything that we've covered in the previous slides are now
available for space.
To name a few of the initiatives that
we support now or in the future include discussions with
preservation of space domain,
and to surface the space for retrospective historical
analysis and planning, miters enterprise for space analysis,
known as Mesa,
for data integration and simulation and
office of space commerce, project teams,
possibly for data repository and cloud services.
These are all still exploratory at the moment.
We're also looking to connect with other FFR DC's,
The UDL data science working group and SAC T observers
partners, and finally participants this ascend conference.
This concludes our presentation on extending miters
transportation data platform, TDP for space.
Please read our technical paper for more details.
On behalf of Marco van and myself,
we'd like to extend our thanks for the opportunity to share
our work and look forward to hearing from you.
Thank you very much.