[MUSIC PLAYING]
In the last video, we saw how we can connect Foundry
with our data sources, which in our example
include a Postgres database, an Azure Blob store, as well
as a REST API, among others.
With our data now synced to Foundry,
we can start transforming it.
[MUSIC PLAYING]
If we click on any data set in our data lineage
view of our pipeline, we can see the code
that was used to apply this transformation by clicking
on the code on the bottom panel.
What we see is the PySpark code that was
used to create the data set.
Under the hood, Foundry leverages
a Spark on Kubernetes implementation
as the compute engine for batch computation,
and Flink on Kubernetes for streaming purposes,
providing a flexible compute foundation that can
be tailored to your use cases.
The platform generally supports any languages
that have Spark bindings, as we run Foundry's Compute Engine
using Spark on Kubernetes.
[MUSIC PLAYING]
In addition to the actual transformation logic, which
is captured here, there are a few additional decorators
that help you natively interact with your Foundry environment.
For example, the Transform Decorator
allows you to specify the input and output data
sets, abstracting away the underlying data cataloging
system that ensures you're always
running on the freshest data.
Similarly, the Configure Decorator
allows you to configure the Spark environment
on which a given transform will run,
since you'll sometimes need to tweak the environment
to handle larger workloads.
To see how you're able to alter all this code,
let's click on View in Repository, which takes us
to Foundry's code repositories tool, an integrated development
environment that allows you to build robust validated data
pipelines.
[MUSIC PLAYING]
To the left, we've got a dynamic sidebar that updates
to the view that you choose.
For instance, displaying your file structure,
repository-wide searching, and a package management UI.
In the main part of the application,
we have a code editor that comes complete with Linting and Code
Assist.
This IDE comes fully backed by a highly available Git service,
to enable you to commit, version,
and collaborate in your data pipelines.
This idea of versioning doesn't just end with the code
that you write.
Foundry also versions the data itself.
So when you go to create a branch off the production code,
like I've just done, to implement
some changes to the pipeline, Foundry automatically
creates a branch for that data, too.
You're now able to branch an entire pipeline from production
into a development branch, make changes, and validate
your modifications.
This is one of Foundry's key strengths.
Data branching removes the traditional trade-off
between speed of iteration and the stability
of your pipelines.
With data branching, you can move
as fast as you need to build value in your business,
but without the risk that usually
comes with normal staging to production promotion paradigms.
Best of all, Foundry takes care of all the underlying
coordination for you, again, all in the singular pursuit
of maximizing your focus on building out value
using your data.
Let's make some [? RR ?] changes here and commit them.
[MUSIC PLAYING]
Once we're happy with the modifications,
we can merge our changes back into the master branch
with a pull request process, just
as you would in a traditional software engineering scenario,
by clicking Proposed Changes.
And we'll go ahead and give a title and description
to our PR.
In the PR Creation Wizard, you can
give your PR a title and description
as well as add the relevant approvers to review changes.
[MUSIC PLAYING]
While the pull request process is largely similar to what
you might have faced in a regular software engineering
scenario, reviewing changes to data pipelines
requires additional tools to really help the approver make
sense of the proposed changes.
Foundry makes it incredibly intuitive
to understand the impact of the changes
by showing you the diff in the code files,
not unlike what you may have seen when merging code
to another branch before; an impact
analysis of the modifications, which helps you understand
the downstream effects of your modifications
on the overall pipeline; an overall pipeline review, which
provides a graphical representation of the pipeline,
highlighting data sets that have been changed;
a conversation tracker, where you can keep track of comments.
Once everything looks good, your reviewers
can approve and merge the PR, which
will merge your modifications back
into the production pipeline.
[MUSIC PLAYING]
Code repositories makes pipeline building
as seamless as possible, allowing
you to focus your efforts on building your data
assets by abstracting away all the overhead required
to maintain these pipelines.
In the next video, we'll take a look
at some of the other functionality
available in Foundry to complement your data
integration.
[MUSIC PLAYING]