MARK MIRCHANDANI: As companies start
to collect large amounts of data from an ever-increasing number
of locations, building a data pipeline
can be quite the challenge.
Join us as we talk to Two Sigma about how they architected
a data pipeline to fit their many demands
on this episode of Stack Chat.
Thanks for joining us.
Tell us a little bit more about Two Sigma
and why your data pipeline was so complex to build.
TONY WALKER: Sure.
So at Two Sigma, we make use of data science and technology
to uncover value.
Essentially, we can buy massive amounts of data
along with computing power and financial expertise
to develop sophisticated training models.
We need to ingest and process data at an ever-increasing rate
whilst protecting it from infiltration,
either accidental or malicious.
MARK MIRCHANDANI: So can you show us
how you were able to keep up with the demands of so
much data?
TONY WALKER: Sure.
Let's take a look at a diagram.
Our first attempt at this was to have a GCP project which
was connected to the internet.
This allowed us to arrange for vendor data
to be ingested from various sources, such as S3 or FTP.
This enabled us to have an environment where
our researchers were able to explore datasets and discover
new features.
From there, our engineers could build data pipelines
using native services like BigQuery, Google Cloud Storage,
and Dataproc or Dataflow.
This enabled us to rapidly derive insights
from the data in a more scalable manner
when compared to our on-premise environment.
MARK MIRCHANDANI: You mentioned earlier
that you needed to protect the data.
Can you tell us more about how you dealt
with that in this environment?
TONY WALKER: As you can imagine, the security
of our intellectual property, both internal and external,
is of paramount importance.
As a result, any operations that deal with the movement of data
in or out of the company's network
must be as secure as possible.
So the most significant concern with this design
was the fact that the project was connected to the internet.
Although this made it easy to ingest data,
it also forced us to isolate our engineers
within the environment.
We essentially allowed them to use native services, but only
from instances within the project,
essentially creating a DMZ-like setup.
This was a pain because it required us to maintain
two separate IM identities--
one with full privileges, and one with heavy restrictions
that could be used safely on site.
MARK MIRCHANDANI: You mentioned handling the import of data
from lots of locations.
How did you accomplish this?
And did you see any impact from dealing
with the size of this data?
TONY WALKER: So dealing with multiple sources was relatively
easy.
In some cases, we had automated jobs,
which would collect data from sources such as FTP.
In many other cases, we were able to leverage the storage
transfer service for interacting with vendors
which share the data via S3.
This was great, as it removed the need for us
to build anything bespoke for the job.
MARK MIRCHANDANI: Going back to security, how did you
ensure that only the right people could access your data?
TONY WALKER: The key change we made
is that the product is now part of a secure perimeter.
This is a concept from the secure VPC product.
Essentially, it allows us to have fine-grained control
over who has access to data stored within BigQuery and GCS.
The other key difference is that we use private interconnect
between Two Sigma and Google.
With that in place, we could enable engineers
and researchers to safely access the data using native tools
directly from within Two Sigma, without the risk
it could be shared outside.
The other bonus with private interconnect
was that we had dedicated bandwidth for data transfers.
Ultimately, the new architecture was a game changer
for our data engineering team.
They could now operate independently
without impacting anyone else within the company.
MARK MIRCHANDANI: Now, this has been running well for a while.
And you've learned a lot along the way.
If you could go back and do it again,
what would you do differently?
TONY WALKER: Well, one thing we've rather glossed over
so far is the way in which we ingest data
in the new environment.
The secure VPC product makes it very easy
to lock down where our data can be accessed from,
but it also introduces a new level of complexity
when dealing with getting data from external sources.
This also required us to think carefully
about how secure API traffic is routed between Two
Sigma and these projects.
In order to do this, we implemented a secure VPC
concept called bridging.
This allowed us to use a single product
as the gateway for all other secure traffic, which
has eased management, but presents us
with a new bottleneck, which we have concerns
with in the long-run in terms of scalability.
This is something we'll look to address
as we on-board more teams.
So by architecting carefully, we were
able to handle large volumes of unpredictable external data,
combine those with our internal data sets, and stay secure.
MARK MIRCHANDANI: Thanks so much for giving us
some insight into how you built your data pipeline to handle
all of these challenges.
TONY WALKER: My pleasure.
MARK MIRCHANDANI: If you'd like to get some hands-on experience
and try setting up a VPC perimeter for yourself,
check out the quick guide in the description.
Thanks for watching.
And make sure to subscribe to the Google Cloud Platform
channel for more great content.
We'll see you next time on Stack Chat.
[MUSIC PLAYING]