[MUSIC PLAYING]
Hello, everyone.
My name is Youssef.
I'm a Forward Deployed Engineer at Palantir
and the Tech Lead of our Foundry for Builders program.
In this video series, we're going
to give you a detailed overview of the Foundry platform's data
integration capabilities and showcase
just how Foundry can accelerate your company's time to value.
But first, what is Foundry?
Well, Foundry is your data operating system encapsulated
into a single platform.
It provides infrastructure that spans
across data integration, the flexible analytics,
visualizations, model building, and dynamic applications,
all within one unified platform.
Where you're interested in building operational workflows
on top of your data, Foundry can provide you
with the needed tooling to clean and transform
your data into a semantic data asset, which
we call the Ontology, that you can build applications
on top of.
One of the inherent benefits of Foundry
is that this flow isn't just linear and read only,
but a record of all your interactions
and decisions you take on top of this data
is persistent inside the platform,
allowing you to enrich your users' operational decisions
into your data assets.
Where your need is perhaps not just on the operational side,
Foundry also allows you to run automated workflows
on your data, such as perhaps a payment settlement or order
processing engine.
The underlying infrastructure is the same.
And therein lies the value of Foundry.
Having a fragmented data stack means
wasting hundreds of valuable engineering hours
on maintaining your data operations infrastructure
instead of building your company's core value
proposition.
With collaboration built in as a core tenet of the platform,
Foundry lets you reuse the work we've already
done to unlock new applications and insights
with minimal effort.
We'll see examples of this in the series,
but this idea is really powerful,
as it allows you to compound the value of your data assets,
taking the time required to produce
a new application from weeks to just a couple of hours.
Nothing else compares to how seamlessly Foundry scales
your data operations for you.
While there's a lot to cover in terms
of Foundry's capabilities, this series
will focus on Foundry's data integration suite of tools.
Using notional data, we'll be exploring
how we can connect our various data systems to Foundry
and transform our data into ontological data sets
ready to unlock workflows and use cases.
With that, let's get right into it.
See you in the next video.
[MUSIC PLAYING]
Well, as behind some of the most useful workflows
or applications that we've seen is not just the snazzy visuals,
but actually the data integration that backs them.
Foundry offers a suite of tools purpose-built to take
your raw data feeds, wherever they
may be stored, and transform them
into value generating assets.
In this video, we'll be taking a look
at the foundations of connecting your data systems and building
pipelines in Foundry.
[MUSIC PLAYING]
First, let's clarify some concepts
as we take a look at a sample data pipeline
that we might build for a fictional supply chain
company using Foundry's data lineage tool.
The data lineage tool helps you navigate your data pipelines
by showing a visual representation.
What we see here from left to right
is the journey from raw data sources to your unified data
interface and applications.
Each node on the graph represents
a data set, which is Foundry's atomic container of data.
Foundry handles both structured and unstructured data formats.
Where we're dealing with structured data sets,
Foundry natively interprets the schema
and allows you to preview the data frame.
The areas between data sets, which are automatically
generated and maintained by the platform,
represent a data transformation between parent and child data
sets.
For example, say we wanted to consolidate
all the different date formats in a column.
We could do this using a transformation that
does just this cleaning step.
The result will be a new data set with the date column clean,
just as we wanted.
All the way to the right, we see the objects
that make up our ontology, such as plants, materials,
and customers that all represent real business objects
relevant to our operations.
Where you can further see is that these objects
back numerous applications and workflows that
do different things.
And therein lies the value of the ontology.
With the unified data asset, the marginal effort
of building a new workflow becomes minimal
since all the groundwork has already been laid out.
All the way to the left, we have a set of source systems
that we're pulling data that pertains to our company's
supply chain from.
Foundry's data connection suite offers a wide compatibility
to connect data from wherever it's located,
including JDBC sources, file systems, cloud-based storage
systems, REST APIs, and many more.
In this case, to build our plants
we have to consolidate data across several sources, namely
a Postgres database, an Azure Blob Store, and a public REST
API.
Foundry's data connection suite offers
an incredibly intuitive interface to sync data
into the platform.
For the Postgres table, I can simply
set up a new Postgres source and add in the host
name and credentials.
With the source created, I can set up a connection
to a specific table in the Postgres database
just using SQL to select the chunk of data
that I'm interested in.
Once saved, I can run the data sync and see it in Foundry.
We also have some flexibility in processing entries
we've already seen before.
In some cases, for example, it might not
be wise to sync all of the data every single time.
So what we can do is configure these things
to be in a pen type, where we only
pull in the new data since the last time we synced data
in Foundry, or update, which only connects new and modified
entries.
Best of all, despite originating from these different sources,
once data has been synced into Foundry,
we don't need to worry about any sort of specific quirks.
This is because data sets are inherently
source agnostic, which means that you can spend
your time building out value from your data
rather than working around compatibility.
[MUSIC PLAYING]
Now that we've connected Foundry with our source systems,
we'll begin creating some transformations to slowly build
up our pipelines.
See you in the next video.
[MUSIC PLAYING]