[MUSIC PLAYING]
CHAD W. JENNINGS: All right.
Well, welcome everybody to this month's Fireside Chat.
Debanjan and I are joined by three colleagues from Broadcom.
We have Padmanabh Dabke, the senior director for analytics,
Zander Lichstein, the technical director for the data lake
projects, and Sanjay Gurnani, architect for the data lake
products.
And so we're going to go through a pretty significant data lake
migration that they have done in the last year,
and we've prepped a bunch of really good stories
for everybody.
So looking forward to it.
Debanjan, take it away, please
DEBANJAN SAHA: Thank you, Chad.
And Padmanabh, Zander, and Sanjay,
thank you for joining us today.
We have a lot of questions for you.
This is one of the most exciting projects we have seen.
So of course, people are really, really excited.
So why don't we first start introducing Broadcom
and the STAR data lake project?
We are also really interested in the use cases that you have,
so let's start with that.
Padmanabh, you want to take that?
PADMANABH DABKE: Yeah, absolutely.
So first of all, thanks for having us.
As maybe we have already mentioned,
we're all part of a group called Security Technology
and Response, also known as STAR,
within Broadcom security endpoint division.
As the name suggests, STAR provides the core technology
behind our threat protection.
We also have a team of experts that's
part of the response who analyze and respond
to emerging threats.
So as part of this effort, we migrated
around 80 STAR applications in the last 12 months.
And I would broadly classify them
into two different segments.
So first, we have traditional file protection applications
that process a large number of malware samples every day
and publish security defs to our endpoint agents.
And second-- and that's perhaps more relevant
to today's discussion--
is our data analytics stack.
We have a data pipeline that receives telemetry
from millions of devices around the world,
and we store it in a data lake on GCP.
And on top of that, we have built a suite of applications.
And they range from machine learning
applications and models that are directly tied to our protection
stack, file and URL reputation systems,
to threat intelligence applications that can detect
advanced targeted attacks.
So that's the summary of what we have on GCP today.
CHAD W. JENNINGS: So that's quite a lot of infrastructure.
What was the motivation for you to move to GCP?
PADMANABH DABKE: So it was driven
by a number of different factors.
There were certainly business drivers, including
cost reduction, for example.
But more importantly, it gave us the opportunity
to cleanly implement a lot of different architectural
transformations.
For example, we containerized a lot of our workloads.
We set up our environment completely
using an infrastructure as code.
One of my top priorities was to decentralize our data analytics
platform.
So we went from a very big, centrally-managed Hadoop
cluster to running most of our applications
on smaller, ephemeral Dataproc clusters.
And with all of this, it's still early.
But I think we are beginning to see
the benefits in terms of cost savings
and simplicity of operation.
DEBANJAN SAHA: We are really interested in your data
architecture and use cases.
But before we go there, there are a lot of telemetry data
that you collect.
Where does this telemetry data come from?
I mean, what are your data sources?
ZANDER LICHSTEIN: I can take that.
So semantics products secure millions
of customers around the world through software installed
on desktops, mobile devices, email servers, network
gateways, cloud storage.
The list goes on.
Providing that protection requires
what you can think of as a giant feedback loop.
As we detect and block threats in the field,
those systems send back telemetry and samples
the kinds of threats, where they came from,
what they tried to do on those machines, et cetera.
And we analyze those samples and sift
through that telemetry to decide what's bad, what's good,
what needs to be blocked, what websites are safe or unsafe,
et cetera, and convert those opinions
into new protection, which is then pushed
back out to our customers.
And the cycle repeats.
In the early days, this was all done
by people, experts mailing floppy disks around, even.
So these days, the number of threats and the amount of data
is so overwhelming that we have to use machine learning
and automation to handle the vast majority of that analysis.
This allows our people to focus on handling just the newest
and most dangerous threats.
And then these technologies are introduced in the field,
and the cycle repeats.
So all of this activity generates billions and billions
of interesting events per day.
And we have dozens of teams and hundreds
of systems, which, together, provide
this protection, detection, exoneration, intelligence.
And all of that requires just handling
a massive amount of data.
DEBANJAN SAHA: That is really, really fascinating.
Let's talk a little bit about the architecture.
Maybe you can tell us a little bit about what
is your data lake architecture.
And you already talked about Dataproc.
What products are you using to build that architecture?
SANJAY GURNANI: Data lake is a decentralized, cloud-based
architecture, where we provide our data lake users with shared
data and cloud storage.
We provide the metadata services via a shared hive meta store.
We provide them orchestration services via Cloud Composer.
And then we provide authorization
via IAM and Apache Ranger.
So users use their own ephemeral or persistent clusters,
so their compute is decoupled from the core data lake.
And they can scale as needed.
And the same goes for their storage.
For the shared metadata and our orchestration services,
we scale them according to our use.
From the product use perspective,
we use Apache Spark quite a bit for our data ingestion
pipelines, as well as our analytics workloads.
And then we use most of the tools from Adobe [INAUDIBLE],,
whether it be Hadoop, Hive, Ranger, Zookeeper.
So Dataproc was an obvious choice for us to use there.
And then we use a Composer for orchestration services.
And then we have a few use cases where
we use Cloud SQL and Bigtable.
For our containerized workloads, we use GKE.
Some folks do use Cloud Scheduler and Cloud Functions.
And then for storing our secrets, we use Secret Manager.
DEBANJAN SAHA: A lot of tools.
So I totally understand that you have your choices.
So why did you decide to use Google Cloud Analytics?
And what value did that provide to you?
ZANDER LICHSTEIN: Our legacy platform actually evolved
from an on-prem solution.
So as Padmanabh mentioned, it was
built as a single, massive, relatively inflexible,
multi-tenant system.
And this, in some sense, simplified
the job of administering the platform
but introduced a number of obvious things.
It even encouraged bad habits from app teams.
Accountability was hard.
Changes and upgrades were very, very painful.
And ultimately, performance suffered.
And so moving from this monolithic implementation
allowed the data lake team to deliver a platform
focused entirely on enabling app teams to do their jobs.
This gave our engineers much more flexibility
in how they develop their systems,
while providing perfect cost accountability,
allowing app-specific performance optimizations,
and completely eliminating resource
contention between teams.
Having this distributed control allows teams to do more,
allows them to make their own decisions.
And it's also proven to be much more cost effective,
which means we can deliver more value.
DEBANJAN SAHA: I'm sure you had heard this
when we have talked to you.
But our fundamental or basic tenets
for building our platform is making it open and flexible,
intelligent and secure.
Do these resonate with you and your teams?
And how did that help you in choosing
Google Cloud and analytics platform specifically?
SANJAY GURNANI: Absolutely, so we do value open a lot.
So we want portability across cloud vendors,
and we don't want vendor lock-in.
So that is very important for us.
From the flexibility perspective,
I would say [INAUDIBLE],, is one from the design
perspective, another from product perspective.
So design, as Zander mentioned we moved from a big,
shared cluster to a decentralized model.
So that definitely gives us more flexibility.
So teams that are the users of data lake,
they're going to spin up their own clusters
and be self-managing.
So it gives them the flexibility to scale their needs as needed,
and they are not logged in by this when all the upgrades can
happen on a shared cluster.
And then everybody has to agree along
before we can make any changes there.
From the product perspective, another example I can give you
is, for example, Composer as an example.
So Composer provides a rich set of operator libraries
that allows us to submit jobs, whether it
be a data proc or to a magic [INAUDIBLE] cluster,
for example.
So those are definitely good.
It has helped us increase our adoption of our data lake.
And we have given the users the tools
necessary to scale their needs as needed.
So they definitely like it.
ZANDER LICHSTEIN: And there are, of course, pros and cons
of that flexibility.
On the plus side, teams are managing
their own infrastructure.
And this ephemeral, serverless architecture
allows you to do that easily.
On the other hand, as you say, Google prioritizes flexibility.
And that can be difficult for a security-minded company.
So we had to make sure we had the right guardrails in place
to make sure our systems remained secure and efficient.
DEBANJAN SAHA: So the next topic I had, you guys
did a huge migration.
So big congratulations on that.
Now, it's not an easy thing to migrate from on-prem to cloud.
So what was your experience?
I mean, was it hard?
Did it provide the right support, et cetera,
to make it easier?
SANJAY GURNANI: Some things were easy.
Some things were a little tricky.
So from the easy perspective, yeah, most of the workloads
that run on Dataproc or Apache Spark jobs and Hadoop jobs,
they were easily portable on Dataproc.
So that was the easy part.
We didn't face a whole lot of issues,
even though we had to downgrade a version on GCP
for our use cases.
For example, other thing was we were using Apache Oozie
for most of our orchestration.
So we had to switch to Composer, but that's where the Google PSO
team was very helpful.
They worked with us, and then we did a PoC
with one of our reference implementations
and proved it to work on Composer
and shared with all our teams.
So that was very helpful.
From the challenging part, installation of Composer
was a little challenging for us in an enterprise environment.
So you have to involve multiple teams
and get the networks right, so it was really challenging.
Other challenging part was we still
had some multi-tenant use cases across Composer,
as well as Dataproc, which we shared with your team members.
And then I was in one of your roadmap meetings
a couple of weeks back.
So I see some of those are getting
addressed in your roadmap.
DEBANJAN SAHA: And thank you for that.
Now that you are in production, how are things going so far?
SANJAY GURNANI: We are a happy camper at the moment.
Most of our data lake users are pretty happy with the outcome.
They like the flexibility.
We are taking another pass at our migration
and going over all our processes,
with the intent to optimizing [INAUDIBLE]
So most of the users I mentioned are
self-sufficient and self-managed.
But it comes with the other responsibility.
You have to be a responsible citizen now.
You have to manage your costs yourselves.
So that is one thing you have to be careful about.
And the other thing is previously,
when we were on a big monolithic cluster,
you used to get about 20% to 25% calls
around [INAUDIBLE] and blocks and things like that.
With this new design, all those things have disappeared.
So our support room chats, we don't get those calls
in our chat rooms.
DEBANJAN SAHA: I'm glad to hear that.
CHAD W. JENNINGS: Yeah, that's a really fantastic result.
I wanted to take a step up a little bit,
and maybe ask Padmanabh here.
So part of Google Cloud's goal is not just
to be your cloud provider but also
your digital transformation and technology partner,
like, for the next 50 years.
So how did the partnership and codevelopment with Google
play into your decision-making?
And how do you think about that going forward?
PADMANABH DABKE: Yeah, I want to go back
to that question about was it hard or was it easy?
It was hard, I would say.
And we certainly needed a partner like Google
to pull it off in the timeframe that we did pull it off in.
We worked with Google right from the start.
STAR, again, like I have mentioned,
has a large number of systems that we had to pull.
It came in all kinds of shapes and sizes.
There were multiple owners.
There are 200-plus people working
in STAR that handle a variety of these systems,
so coordinating all of that was not easy.
We probably took advantage of every little feature
that Google has.
In particular, we mentioned security as one of the issues
where we needed a lot of understanding.
We needed to set it up correctly.
There was another partnership that
was close to my heart, which was setting up
infrastructure as code.
We really partnered with Google's professional services
in making that happen.
So overall, I feel like we benefited a lot.
We had to work with Google very closely.
Honestly, I hope that we gave Google
a number of items that might be on the bleeding edge that
would become part of Google's roadmap.
And I'm pretty sure Zander and Sanjay were neck-deep
into this, and they would have a lot more to say on this topic.
So I'll hand it over to them if they have any comments.
SANJAY GURNANI: On my side, I would
say we wouldn't have been able to do the migration
without Google's help.
Also, it would have taken us a little longer.
But I really appreciate the support
that we got from the Google PSO team.
So they worked just like an extension of our team,
working with us on a daily basis and participating
in our stand-up.
That came handy to just jump-starting
the migration for examples.
They could provide us with technical examples,
or facilitate the support from other product teams as needed.
So that was definitely [INAUDIBLE]..
ZANDER LICHSTEIN: Chad, let's put it this way.
Previously, it took almost an entire year
to coordinate just the version upgrade of the data lake.
And here, we moved all 80-odd systems
across the entire division to a whole new cloud
provider in about a year.
It was a massive, massive effort.
But ultimately, it's looking like a success.
CHAD W. JENNINGS: That's great to hear.
Both the tech is working for you, and that the people are.
Debanjan, you had a question.
DEBANJAN SAHA: No, that is fascinating.
And to your point, Sanjay and Zander and Padmanabh,
we want to work with you as your partner, as you know.
And whatever feedback you have in terms of roadmap,
we are going to work with you on that.
Now, one thing I wanted to also ask you,
but you mentioned a couple of times
about some of the database products.
What database products do you use along
with the analytics products?
And have you integrated your workflows
across the analytics services and the database services?
ZANDER LICHSTEIN: Well, I think most notable
is we had a few critical systems that
were based on HBase, which we were able to easily migrate
to Bigtable.
Performance has improved, and it's obviously much,
much easier to maintain.
DEBANJAN SAHA: Now, with that, I think we
are at the end of our session.
So Padmanabh, Sanjay, and Zander,
thank you for joining us today.
And big congratulations for putting out
one of the biggest migrations of data
lake from on-prem to cloud.
And looking forward to working with you in the future,
and work as partners in making data lake really very
successful for you, and possibly for others and your customers
Thank you.
CHAD W. JENNINGS: Thank you so much.
ZANDER LICHSTEIN: Thanks
PADMANABH DABKE: Thank you, everyone.
[MUSIC PLAYING]