[MUSIC PLAYING]
And so I think like the traditional kind of analytics
or data architectures come from a long lineage of saying
the primary problem is data consolidation, right?
Data is coming from different sources.
They put it together in a place where I can then access it
for reporting or for BI.
So explain just in detail, what does a relational database do?
Why is it built for certain things?
And why will break down in going from read from read to write?
And then we'll talk about how it breaks down
in the disjointed world.
Yeah, so relational databases, and specifically data
warehouses and what we think of as MPP or massively
parallel processing databases, are
designed for effectively serving data at low latency
to hundreds or thousands of users
where the primary access pattern is
"I'm trying to read this data."
So how data is locked, how data is kept somewhat consistent
is within the paradigm of saying,
the primary use case is I'm trying to visualize this data
or pull it into another place to then look at it
or work with it in a read-only way.
And now let's just start with the very basic functionality
of, you have to write-- meaning write
function means you would actually take insights and put
them in the data that are not the ones in the database.
Where does that break down?
And here, we're going to bracket,
where does it break down in a multi-jurisdictional
perspective?
Where does it break down from an access control,
meaning data protection?
Where does it break down in low latency, so in a classified-
We're going to leave those all out for now, all of which
we built products for, which is why we're powering
a lot of the Western efforts in intel and defense,
but just very, very simple, moving read to write,
why does it break down?
Yeah, so if you think about how data
is intended to get into these systems,
it's through standard interfaces where
there are existing connectors or existing
interfaces from other systems.
So it's system-to-system writing, right?
The key difference is when you have
users trying to write data.
Typically that happens through updates or inserts
that are coming from end users.
There's very limited attribution as to what the user is doing
or how they're actually doing that insert or update.
There's also very limited capability
to segment out priority as to what
users rights or updates should take preference over others.
It's very much designed to be a bus of in and out for, I'd say,
pretty coarse types of rights.
You think about user rights the same way
you think about system rights.
So that introduces a lot of parallel in terms of, well,
how could I potentially overwrite--
Maybe a different way of asking the question would be,
if you didn't have a standardized system to do this
and you allowed people to write to the core code base or data,
what would happen?
And why are they preventing people from doing this?
What is the technical challenge?
Yeah, literally the technical challenge
is data that's coming in from source systems
into these databases are getting stored
in directories or tables.
Sure.
--Can then be modified in the exact same modification schemes
by end users.
You can delete data.
You could overwrite data.
You have no ability to understand why data was
modified to a certain extent.
And so it's just a lot of orchestration.
And a lack of versioning, a lack of lineage,
means that you can't really allow people in a multiplayer--
Most people don't know what versioning is.
What is versioning?
So versioning is actually understanding over time,
how did data change?
And not just how did it change, like,
today's data versus yesterday's data,
but who changed it and from the results of what code or what
process changed it?
That's kind of the key bit that's difficult.
You can, of course, have a log or a transaction
log of data changes.
Another assumption people have is
could just make some kind of app and write it
on top of your relational database,
and all of a sudden you would be able to do versioning.
Why is that wrong?
Yeah, so the devil's in the details there, right?
So if you have exactly one app or a set
of apps where you've built in the processes into those apps
or into some middleware where you can actually orchestrate
and say this app under these conditions
will write into these tables and take preference
over simultaneous potential rights from another app,
then this can all work at scale.
But you have to basically put the burden
of that sort of orchestration or reconciliation work
somewhere in the system, where you could technically
theoretically extend the database
to do some of that but it becomes a pretty
intractable problem if you think about dozens of applications,
hundreds of users, all doing rights
under different paradigms.
Again, here we're bracketing some
of the very important issues that
are in the national security context,
like how would you deal with peering
between people who are both allies
but not sharing all the data?
How would you do this under with high latency?
How would you do this from space to mud,
as we've said, somewhat in our jargon?
How would you how would you reduce
the aperture of the data?
So we'll bracket those issues for another video, all of which
we probably should do.
OK, so now, we have a basic nascent understanding
of why it's hard to write to a database.
How do you solve this problem and how do you
move from read to operational?
And then as a last part of this video, what
we want to get to is, in the past,
we've been viewed as antagonistic to IT programs.
And this is a process, but it's my view
that there's been a major shift there where
IT programs across the commercial space
have done an enormous job in building read
extensions onto their database, and now have
begun to understand that they're going
to have to do write functionality
into operations, and in fact very much poised
to buy our functionality.
In the same way they were opposed,
they've already built functionality
that we built 10 years ago into their enterprise.
So first, how do you solve go from read to write?
What is the solution that we've built
in the core of our architecture?
And then how do you go from write to operational?
So I think the key thing that we did
is we built a layer that sits on top of the relational data
stores.
We assume there's actually going to be many of them.
The NoSQL data stores, the IoT data stores,
the geospatial data stores called this ontology,
this ontological layer, which actually provides
an abstraction point where you're actually
able to not just present all of the data
in a semantic and kinetic view to end users, but also say,
this is going to be the mechanism through which you
accept rights with all of the relevant metadata required
to actually reconcile those rights with the underlying
data.
And so the ontology provides, in a sense, not just
an operating layer
Why is it hard to build an ontology like this?
Yeah.
I mean, first of all, why is it very, very hard?
So the term ontology is actually not new, right?
In the industry, people think ontology has
something to do with taxonomy.
Ontology is a little older than the industry.
But let's go back to that.
Even in the industry, right?
Even in the industry, people sometimes
think it's just a data model file,
it's just just a taxonomy.
And the way that we've actually thought about the ontology
is it has to encompass both data but also actions
and process as well.
So I think the key thing is, is you
want to understand not just what is the way in which you need
to present data in a governed, semantic way to users-- so
the factories, the trucks, the transactions, the customers,
the interconnections-- but also, what are the legal actions?
What are the processes that can be taken with those objects
by different user groups?
So I can actually modify this work order.
I can actually flag this customer transaction.
And as those different actions cascade,
I'm capturing their consequences in the ontology,
but also then pushing them back to underlying systems.
And so it gives you sort of this ability
to simultaneously model both objects
and also processes together.
Maybe a little bit of background here.
When we entered the IC space, what we discovered was--
it actually sounds like the same problem,
but it is actually very different in the IC space--
that you would have to be able to index against data
sets for both legal and practical reasons.
So you would actually be working with the indexing layer, not
the underlying data.
And we realized that the actual databases were not
set up to abstract out of the databases
where the abstraction is actually the reality.
But you have to disambiguate the abstraction,
because there are going to be thousands
of abstractions inside an enterprise that
have to work together.
And then we brought that to Foundry.
OK, so assume you have the ontological layer,
and it allows you to work with abstractions.
Why does that allow you to write?
And then you've already begun to answer it.
Why does it allow you to operate against the data,
or the data becomes operational, independent of what
you're doing?
So the key thing, I think, is that the ontology layer holds
both the objects and the relations,
but also it holds the actions as well.
And by way of doing that, it not just captures how actions can
cascade across other parts of the ontology, but also says,
what are the mechanisms through which we could
capture those rights, reconcile them, make sure they actually
are approved, and then push them to other systems?
And so you actually have a buffer where--
By the way, you're adding in how an access control model works
against the abstraction layer.
You should probably talk about that first.
Yeah, I think that's critical to why the rights can work.
So the fact is, there is no write happening
in the ontology layer, either from a system coming
underneath or from an end user coming back,
that isn't going through the access control model.
So this is a combination of group-based access controls,
which could be integrated with the existing enterprise
systems, classification-based access controls, and also
purpose-based access controls.
And by the way, without going into my macro thoughts
once again, obviously, if you were building something
for a disjointed world where jurisdictional sovereignty was
more and more important, you were sharing data
across jurisdictional sovereignty,
but only according to the rules where you're allowed to do it,
you would need a robust ontology and an access control model.
In any case, that's the reason we spent
a lot of time working on this.
OK, so we began in early parts of our business understanding
the limitations the way in which data was stored in software.
We moved from read to write to operational.
You see this both in our government
and in our commercial business and the enormous impact it's
having and of course even more in the future.
Why was this viewed as being antagonistic to IT?
By the way, in government forever, although our business
there is so big that that has changed.
And then there's a caricature of Palantir that--
best software in the world.
They do a lot of things.
We don't actually understand, but they
seem to be very, very valuable, and someday we
will understand them.
But is there going to be an absorption of this?
So you've seen it in among some of the largest
companies in the world, and obviously it's growing
and American business has doubled from 50 to 100 to 200.
And we see these enormous absorption
with a nascent sales force which is quite, I think,
anomalous 18 years into a business.
I don't know any other business that looks anything like this.
So just this acceleration, which you
don't expect from a business that's been around,
which is obviously product-driven.
But there is this thing-- well, is the friction coefficient
of the internal architecture of internal IT structure going
to be--
now, we went through the same thing in government and all
over the world.
And we've seen how this shifts from being antagonistic
to being people's best friends.
But what is it in the IT context that
has shifted in the last, say, year, and how have we
shifted also to accelerate that shift in how
we're providing Palantir?
Yeah, I think the fundamental source of friction
was the assumption that we're trying
to compete with the data integration and model
integration layers, right?
So the idea that, hey, I've invested
in my existing databases data lakes, data warehouses,
the Snowflakes, the Redshifts, the GCPs of the world,
and therefore I can't use Palantir, I can't use Foundry.
Whereas, from our perspective, it's like,
all the value is in extending, like you said, past data
storage into operations.
And so I think what we've been able to clarify,
and I think what we've tried to do an increasingly better job
of is telling the story of saying what we actually think
this ontology-driven approach is and how the actual software
works, which is highly modular, is you can bring your existing
data stores.
You can bring your existing analytics tools--
It's kind of ironic, because of course it's not competitive.
Because we built these capabilities
almost 10 years ago.
So we know where the road is going,
because we've already been supplying those kind of things.
But in any case, keep going.
I think what we had to clarify, though, was saying like,
OK, even though we provide solutions
for those layers of the stack, you can swap in your own
if you want.
And I think that's where a lot of our engineering effort
has gone, as well, is saying like, how can I
make it seamless to come to the table with my existing data
warehouse?
Your version is-- and, by the way, it's the true version.
It's not exactly the way I see it.
I think it's both true that we probably--
because we're primarily an engineering organization
and we're not particularly good at sales--
have done probably a pretty crappy job
of explaining to people how we're not competitive
and probably a great job of explaining to people
how we're smarter, which is whatever.
Probably not exactly what you're supposed to do.
But it's also true in my view that there's a learning process
that just has to take place.
A lot of the issues that IT organizations are dealing with
are exactly the issues that we had to discover
you're going to deal with.
I mean, if you just look at our progression,
first we built things that were like data warehouses
so that people could actually get something done, right?
So the nascent PG.
And then we saw well, of course, that
won't work with unless you have these other capabilities.
And then we went to build it into a robust product,
and then we saw if you wanted to work very quickly,
you'd have to be able to build a product that you could
sell into the architecture.
And then we began to understand, what
are the issues up the chain?
And so some of this is our fault, and some of it's just,
it takes five years of doing these things to actually
understand the issues that are coming down the road
Yeah, and I think that process has actually been underway
with the whole cloud transformation, too, right?
So people have moved to the cloud,
and, I think, successfully ported over analytics
workloads, right?
So instead of kind of the data stores of yore
and the MPP databases of yore, they're
now using data warehouses, data lakes and also analytics tools
to do the BI and dashboarding stuff.
But now the logical question is, in the market today, awesome,
I've done that stuff.
Now how do I actually get this stuff into operations?
Like what is the way to do that?
Also, people are under a lot of pressure because of just
general macro events, and that accelerates their learning
process.
Yeah, so I think broadly, I think about it
as sort of the difference between knowing
you have a problem and being able to act upon it
or change plans or implementations that
might already be in place.
So as an example, in utilities, it's
like you might know that you're having
like net decreases in power or you
have issues that need to be addressed
through a public safety power shutoff process,
but then how do you actually modulate how you're actually
running the assets?
Or how do you decide which work around decommissioning
of old assets is prioritized first?
There's an existing plan in place,
which is a read-only view.
They'd be able to update that with new information not just
based on our hypotheses-- what people are doing in the field,
as well.
So there has to be an orchestration
of field-driven learnings that are actually
building the knowledge asset.
And that's what's super exciting that we see across supply
chain, telco, health care, as well,
is the decisions being made by end users
in different contexts is actually the most
important data oftentimes.
On supply chain, why is it difficult?
People tend to think, OK.
I mean, they watch some movie where people are like, oh,
why is this a difficult problem for them?
In the classical world, the ERP and the planning process
were all pretty integrated, right?
So if you think about where the forecast came from,
it was codified in a way that assumed
the underlying standard sources were
contributing to the forecast.
And then if there's disruptions happening both in the supply
and demand side all the time, you
say, well, I need to develop a better forecast.
Really what I need to do is develop a better
forecast almost every day, and so to do that requires
different data sources, requires basically understanding
which models to use under which circumstances
and getting constant feedback from people who are operators
using those forecasts.
And so it changes from saying here's
the forecast for the month or the week to saying,
what's the process for continuously assessing
what optimal looks, like with changing
objective function sometimes across factories or facilities,
too.
So it's just like a very different granularities
of problem and also speed of iteration.
[MUSIC PLAYING]