LORIN PRICE: My name is Lorin Price.
I'm a Networking Specialist Customer Engineer
at Google Cloud.
And today I'm going to talk to you about the networking
concepts for Cloud Data Fusion that
are helpful to know for implementation, security,
and connectivity.
[INTRO MUSIC]
Cloud Data Fusion is Google Cloud's native
managed service for enterprise data integration and ingestion.
It provides a visual point-and-click interface
enabling a code free deployment of ETL and ELT data pipelines.
Cloud Data Fusion has a number of components that exist behind
the scenes that power the service,
and it creates a tenant project and VPC
to hold those resources.
The Data Fusion components that are surfaced to the user
are the Data Fusion instance and the Dataproc clusters.
Each Cloud Data Fusion instance contains a unique independent
Cloud Data Fusion deployment that
contains a set of services that handle pipeline lifecycle
management, orchestration, coordination,
and metadata management.
Cloud Data Fusion creates ephemeral execution
environments to run pipelines when you manually
run your pipelines or when pipelines
run through a time schedule or a pipeline state trigger.
Cloud Data Fusion uses Dataproc as an execution environment
in which you can choose to run pipelines as Spark or Spark
streaming programs.
Cloud Data Fusion provisions an ephemeral Dataproc cluster
in the user project at the beginning of a pipeline
run, executes the pipeline using Map Reduce or Spark
in the cluster, and then deletes the cluster after the pipeline
execution is complete.
Alternatively, if you manage your Dataproc clusters
in controlled environments, you can also
configure Cloud Data Fusion not to provision ephemeral clusters
and run pipelines against existing data clusters instead.
The Cloud Data Fusion instance can
be thought of as the design environment
and the Dataproc cluster as the execution environment.
There are two types of Cloud Data Fusion instances
based on an access model, a public instance,
which is the default deployment, and a private instance.
Let's dig into the access patterns
from different scenarios.
We can start with the public IP deployment pattern.
In a public instance, network communication
happens over the open internet regardless
of the source location, which is not recommended
for critical environments.
The Cloud Data Fusion instance for the design environment
will have a public IP.
However, the data cluster used for the execution environment
can be deployed with either public IP or private IPs.
Let's move on to the private IP deployment pattern.
Cloud Data Fusion private IP has different setup steps
depending on whether you are using a standard VPC
or a shared RPC network.
If you're using a standard VPC, you
can create the instance using either the Cloud
Console or a cURL command.
However, if you are using a shared VPC,
you must use a cURL command.
Check out the documentation that's
linked in the description for more information.
When private IP instances are created,
a separate tenant project is created
for each user project in which Cloud Data Fusion
instances are provisioned.
Additionally, Cloud Data Fusion private IP
requires peering in connection to communicate privately
with sources located in the user VPC
and is required for the design environment capabilities.
This peering connection must be created manually.
If you need to access a public source using a private IP Cloud
Data Fusion instance when designing pipelines,
you must use version 6.4 or above.
In the design environment, Cloud Data Fusion traffic
will traverse a VPC peer from the tenon VPC to the user VPC.
Traffic will then be directed to an internal load
balancer fronting a managed instance group of gateway VMs.
Traffic will egress through the gateway VMs
through a Cloud NAT configured for the subnet
into the public source on the internet.
The Dataproc cluster used for the execution environment
should be configured to egress through the Cloud NAT
as well to reach the public source.
For sources that are hosted within the user VPC,
Cloud Data Fusion will connect over a VPC peer.
Firewalls must be configured in the source VPC
to allow the source to communicate with the Cloud Data
Fusion instance, as well as the Dataproc execution environment.
If you are in a peered hub and spoke architecture model
and you are trying to make a connection from a VPC that
is not directly peered to the Cloud Data Fusion tenant VPC,
you must use a proxy due to the nontransitive
routing nature of Google's VPCs.
This will only be needed for the design environment.
Be aware you can also replace a peering hub
and spoke architecture with a VPN hub
and spoke model so that the Cloud Data Fusion 10 and BPC
route can be advertised over the VPN.
Nontransitive peering also comes into play
when trying to connect to a Google Cloud source that
is deployed using another tenant project,
for example, Cloud SQL, Cloud Data Fusion and Cloud SQL
are both hosted in tenant projects,
but these projects are different.
In order for Cloud Data Fusion to connect to a Cloud SQL
source, a proxy VM must be deployed
in the user VPC to proxy traffic to the Cloud SQL instance
when using the design environment.
Since the Dataproc clusters used in the execution environment
are hosted in the user VPC, they will
be able to connect to Cloud SQL without the use of the proxy.
When accessing sources in an on prem environment with Cloud
Data Fusion private IP, a hybrid connection
must exist between the user VPC and on prem environment,
either a HA VPN or interconnect can be used.
Both the IP side arrange for the tenant VPC
where the Cloud Data Fusion instance is hosted as well
as the IP side range of the Dataproc cluster
in the user VPC must be advertised
over the hybrid connection.
And on prem firewalls must be configured to accept ingress
from these ranges.
If the VLAN or VPN lands in another peered VPC, then
you'll need to use a proxy to connect the source to the Cloud
Data Fusion instance.
When accessing data sources, both public and private
instances make outgoing calls to Google Cloud APIs
using private Google access.
Let's briefly cover some security topics.
For Cloud Data Fusion versions below 6.5,
you can only manage access at an instance level.
If you have access to an instance,
you have access to all pipelines and metadata in that instance.
For version 6.5 and above, you also
have the ability to use role-based access control
to establish name space level isolation within a single Cloud
Data Fusion instance.
This feature is currently in preview.
Pipeline access to data is provided by granting access
to the service account, which can be a custom service
account that you specify.
You must make sure that the service account has proper IAM
permissions in place to access any Google Cloud hosted data
sources.
For a pipeline execution, you control ingress and egress
by setting the appropriate firewall
rules on the user VPC on which the pipeline is being executed.
Before we wrap up, let's summarize the three most
important takeaways from today.
Cloud Data Fusion exists as a design environment
that runs within a tenant project in VPC
and an execution environment that runs as a data cluster
within the user VPC.
Cloud Data Fusion can be launched
with a public IP or private IP, and VPC peering must manually
be configured between the tenant VPC and user
VPC for private IP.
Firewall rules must be configured to allow connections
to and from both the Cloud Data Fusion instant [? sider ?]
and the Dataproc [? sider. ?] Thanks for watching,
and look out for another video on the concepts of networking
for Google Cloud's managed services.
[MUSIC PLAYING]