- Hi, everybody, really glad to be here.
Hope, you're all doing well.
I'm here to talk about Hybrid Cloud Access Control
in a Self-Service Compute Environment here at Comcast.
I want to just real briefly mention
that Comcast takes very seriously
its commitment to our customers too protect their data.
I'm part of the Comcast, what we call Data Experience,
Big Data Group and big data in this case
means not only public cloud, but also On-prem data.
So we have a heterogeneous data set,
which offers some challenges and challenges are fun, right?
Our vision is that data is treated as an enterprise asset.
This is not a new idea, but it's an important one,
and our mission is to power Comcast enterprise
through self-service platforms, data discovery or lineage,
stewardship, governance, engineering services
all those important things that enable people
to really use the data in important ways.
And we know as many do, that powerful business insights
the most powerful insights come from models
that integrate data that spans silos,
insights for improving the customer experience
as well as, you know, business value.
So what this means for the business,
there are some examples,
basically this is based on the tons of telemetry data
that we capture from sensors in Comcast's network.
We capture things like latency traffic,
signal to noise ratio, downstream and upstream error rates
and other things that I don't even know what they mean
but this enables us to do things
that improve customer experience
like plan the network topology
to help, if there's a region that has a ton of traffic
we might change this policy to support that.
Minimizing truck rolls, truck rolls or what we call it
when the Comcast guy, cable guy comes or cable female
comes to your house and in times of, this Covid times
we really would like to minimize that even more.
And if we can analyze the data ahead of time,
we can perhaps make any adjustments
or suggest adjustments that the user can make
to minimize the need for people to come to their house.
We can monitor, predict problems
and remedy them often without, before the user even knows
because of this data
and this involves both the telemetry data
and integrating it with other kinds of data
across the enterprise,
and then optimizing network performance
for a region or for the whole household.
So now this is really important stuff
and it really helps the customers
and we're working to make this even more prevalent.
So, what makes your life hard?
This is a professional statement.
If you want to talk about personally
what makes your life hard, we can do that later,
but what makes your life hard as a data professional?
People usually say, I need to find the data.
So if I'm gonna be integrating data across silos,
I need to find it.
I know where it is in my silo, but maybe but,
and the way we do that is a metadata search and discovery
which we do through Elasticsearch.
Then once I find the data that might be of interest to me
I need to understand what it means.
So somebody, what someone calls an account ID
might not be the same account ID
that you are used to calling an account ID,
billing IDs or back office account IDs
need to know what it means in order to be able to join it,
to make sense as opposed to franken data,
monster data that isn't really appropriately joined.
We need to know who produced it,
did it come from a set-top box?
Did someone, did it come from a third party?
Who touched it while it was journeying through Comcast?
If it came in through Kafka or Kinesis
and someone aggregated it
and then maybe somebody else enriched it with other data,
and then it landed in a data lake.
The user of the data in the data lake
wants to know where these, where the data came from,
who and who added what piece.
And you could see this as both,
the publisher looks at the data in the data lake and says,
you know, this looks screwy, what's wrong with this?
Who messed up my data?
You could also say, you know,
or they could say, wow, this is enriched really great.
I wanna thank that person, right?
And also someone who's just using the data
wants to know who to ask questions.
You know, what did you enrich this with?
Where that data come from, that kind of thing.
So, and all that really is helpful
when you're doing this integration.
We do, that's data governance and lineage
which we do in Apache Atlas.
That's our metadata and lineage repository.
And once you found data and understood it,
you have to be able to access it.
And we do that through at Apache Ranger
and its extension that's provided by Privacera.
Once you have access to it,
you need to be able to integrate it
and analyze it across the enterprise.
So finally, now we get to the good stuff
to be able to actually get our hands on the data.
And we do that with self-service compute using Databricks
and Databricks is a really powerful tool for that.
And finally, we find that we do really need ACID compliance
for important operations.
And we do that with Delta Lake.
So I'll, I can talk about this in more detail
as this topic goes on or in the question session.
I'm an architect, so I have to have dots and line diagrams.
So this is a high level view of our hybrid cloud solution.
So income passed on our data centers.
we have a Hadoop data lake that involves Hadoop Ranger
and Apache Atlas working together.
We are, as many companies are kind of phasing that out,
but not immediately, it takes, you know, it takes awhile.
We have a Teradata enterprise data warehouse.
Similarly, we are thinking to move that
and this not necessarily to the cloud entirely
but maybe to another On-prem source
like the Object Store.
We use MinIO and basically that gives us,
makes this Object Store look like S3.
So when the spark jobs that we write to use on S3
also can run on our On-prem data store
and that's a big plus of course.
And for that, we also, we have a Ranger Dataservice
that helps with access control there.
Up in the cloud, we use AWS
though Azure also has a big footprint in Comcast.
And Databricks Compute is kind of the center here.
We use it to access Kinesis.
Redshift is only, we're just starting with that.
We use Delta Lake and S3 Object Store
and we have a Ranger plugin
that the Databricks folks worked carefully with Privacera
to create so that our self-service Databricks environment
can have all the Init script and the configurations
that it needs to run the access control
that Privacera provides.
We also use Presto, and for our federated query capability,
it also has a Ranger Plugin and all the tags
that are applied to metadata on which policies are built
are housed in Apache Atlas
and Ranger and Atlas synced together,
and that's how Ranger knows what policies
to apply to what data.
And in the question session,
if you want to dig deeper into any of this,
I'd be very happy to do it.
So this is very exciting to me.
We're just rolling this out and it's so elegant
and I didn't create it, so I can say that.
So Ranger and Atlas together
provide a declarative, policy-based access control.
And as I've said, Privacera extends Ranger
which originally only worked in Hadoop
to AWS through plugins and proxies.
And one of the key ones that we use
for us is Databricks on all three of these environments.
And basically what I like about this
is we really have one Ranger to rule them all
and Atlas is its li'l buddy
because he provides or she provides the tags
that really power access control.
So here's again a diagram.
And we have a portal that we built
for our self-service applications
and the user tags, the metadata with tags, like this is PII,
this is a video domain, that kinda stuff,
that goes into Atlas,
the tags in the metadata associations
are synced with Ranger, the policies based on that.
So who gets the CPII, who gets to see video domain data?
Those are synced and cached in the range of plugins
and then when a user calls an application,
whether it's a cloud application in Databricks
or even an On-prem application,
the query that the application asks Ranger,
does this user have the access
to do what they're asking to do on this data?
If the answer is yes,
and it's very fast, 'cause these are plugins,
if the answer is yes, they get access,
if no, then they get an error message
and we can also do masking and show the data,
if someone has access to many columns, but not all columns
I would say a glue table.
We can mask out the ones that they don't have access to.
And still give them what data they are allowed to see.
Recently we've really needed ACID compliance.
So traditionally, big data lakes
are write once read many,
you don't, you know, we have things streaming in
from set top boxes in the cable world.
You know, you're, those aren't transactional,
that's not transactional data.
That's what we're used to.
But now increasingly we are finding
that we need to delete specific records
from our parquet files or whatever.
We can do this in Spark, but it isn't terribly performed,
it certainly can be done,
but it turns out Delta Lake does it much better.
Their deletes are much more performant
and you get to view snapshots of past data lake states,
which is really pretty awesome.
So we're really moving toward, I love this word,
a lake house, being able to do write once read many
and ACID all in one place.
And that is largely thanks to Databricks.
So this is me.
Please reach out to me an email if you wish.
And I'll be happy to answer questions
in the live session if you have any.
So thank you very much for listening.