So we have as you know, we have an ongoing series.
Parvathy did one and where we talked about general Palantirian culture.
People often wonder how this motley crew of people ended up powering the most important institutions of the West, especially in the military.
Largely things we can't talk about, especially since we're outsiders.
And actually in USG for years and years didn't even have a salesforce.
But we've built some of the most important back end instruments of war.
These instruments, as you know, are powering some of the most interesting, important war efforts in the world now and have left led to disproportional success on the battlefield.
Probably the most important thing we've built.
You are in charge of next Nexus Peering in that context.
Before we even get to the what, why is it difficult to supply operational data on the on the war in battle?
Like what what is it about that that would require you to build a special component like Nexus Peering?
Why is it difficult?
And will you turn on TV and there's like whatever that TV show is, 48 or 24, you know, it's like it's just the data is all there.
They go in the battlefield, the data is all there.
And then it turns out for special operators, the data was never there.
Yeah, I mean, for so many reasons.
There's just so many reasons why it's hard.
I think at a very basic level.
You know, I think about that in terms of sort of geography, organization and network, right?
So special operators work across many networks, across many geographies, across many organizations.
And that, you know, gives you.
But why don't the organizations I mean, you know, lay people are like, well, they just get all the data from all of the organizations.
Right?
Why don't they just get all the data in real time?
What is?
Why not?
Yeah.
I mean, I think to the extent you want to collaborate, you have to have a common language.
And so if an organization wants to model data for themselves, they're going to model it in a way that makes sense for them.
You know, we call the ontology.
Each organization models the data differently.
And so if you wanted to if you were a special operator climbing over a mountain,
jumping after you jumped out of your helicopter,
sneaking into a building to take out bad people, you de facto get gobbledygook.
Exactly.
It's sort of like trying to translate Spanish to English.
Right.
You know, if you don't speak both languages, there are things that you actually can't express in either side.
And so there's only a sort of subset of all the data, all the data written in Spanish that you can really translate to English.
That's not a great example, but the more different the languages are, the more different the organizations are,
the less you can actually sort of have an individual end user collaborate with another user who's sort of speaking a foreign data language.
And some other issues.
Like, so you have security issues.
What are those issues like how?
Why you know, what is involved in, say, you have, you know, an organization.
Leaving aside even the complexity of multiple countries, each organization has its own classification methodology.
Why is that difficult to translate onto the battlefield and what are we doing to alleviate that difficulty?
Yeah, so there's really two forms of security.
You have physical security and sort of, you know, virtual security.
So from a physical security perspective, data is just stored in different databases on different hardware.
That hardware can't always talk to one another and there's not even a network connectivity
available between two physical servers.
So when we first built Nexus Peering, we were building it for two customers who,
you know, the medium of exchange was a DVD once a week, right?
There was no cable plugged in from customer A to customer B.
And so we had to design the whole system sort of end to end to work on a DVD.
Right, with extremely high latency.
And so most of the modalities of collaboration, you would imagine.
So chat, real time sort of typing and seeing that synchronously between two users,
you're getting that once a week.
And so you just have to design the whole system in a totally different way.
And latency is another like so we have classification model.
So how do you take different classification models and different ways of using the data?
So that's one problem is how do you view the data, what we call ontology
or like our our ontology allows you to translate their ontology.
Then you have classification models.
So every organization requires you to work at different levels of classification and on different missions.
And now you're getting to a different issue, which is the latency issue.
Latency issue, in my view, is the one that people understand the least.
Yeah.
So people intuitively understand the ontology issue, the classification issue,
and understand maybe their software is built to do that.
And but the latency issue is an issue, I think that I've never met anyone that understands it.
And on the battlefield software breaks.
Why is latency so so such a difficult issue for software?
So I think it's about not just the software, it's sort of the user has certain expectations of the system.
When they log in, they sort of have a mental model that they're operating within.
And the latency issue really breaks that mental model, right?
When you log into Gmail, you don't have to think like, where is my email?
It's just sort of replicated in certain ways.
But sort of it's obvious once you say it, we build analytics systems.
Those systems require huge volumes of data, and so that data just can't make it to the edge.
If you have a small device on the edge, a tactical node,
you're not logging into a server, you're logging into a mesh of servers.
And so what that translates into is there's both a latency issue of sort of how quickly can you move data between servers?
But there's also this mental model the user has to have, which is that
they're logging into a mesh and the data is going to be sort of available.
Different volumes of data are going to be available, different parts of the mesh.
And so one of the really hard things is sort of how do you get the right data to the right piece of hardware
such that the user can actually access it, who's on the tactical edge.
And how do you do that?
And what is the role of the vector clock?
What is a vector clock?
Yeah.
So if you think about a sort of basic system, how Palantir was for the first couple of years,
we really thought about data versioning provisioning in terms of logical clocks.
What what is data versioning?
Why is it important?
Right.
So data versioning is basically just, you know, data changes over time.
Palantir has grown.
Palantir has more offices.
Over time, the world changes.
And you want to reflect that in sort of your digital twin, in your data itself.
The clock is really like a watch.
It's basically saying, hey, this time, what did the data look like at this time?
What did the data look like?
And so, you know, if you have one clock and I can put my watch on the table,
we can both look at the watch and we can see what time was.
We can sort of roll back the clock.
But as you have two servers, those two servers have independent clocks.
They don't know what time it is.
And so the two clocks or three clocks or four clocks or 1,000.
A thousand clocks.
Clocks.
Exactly.
They know the relative time of one another, but they don't have any way to reference absolute time.
Right.
So you can think about it.
As I can tell you, you know, 2:50 on my watch is later than 1:50.
But I don't know what your watch says at 1:50.
And to figure that out, I have to ask you and you have to tell me.
And so there's this sort of bidirectional problem of, you know, figuring out what time.
Without which the server basically collapses.
Exactly.
And so this is super precise work, right?
Like when you mess this up, it's super corrupted.
The system just sort of falls apart.
And so you want to be able to very precisely revision sort of what happened when relative to other things.
And it just for historical clarity,
we had already we already understood versioning because versioning is a is a normal concept in software coding outside software, people don't understand it,
but the versioning allows you to also then work with thousands of developers and write the code with the trunk
while still being able to experiment on the branches.
And then we took that idea of versioning and built it into all of our software.
We had versioning, which versioning still won't work in this context because you would need something like a clock.
The lay version of this would be like debate two to timestamp every little shred of data so that you could version against it.
And otherwise the system collapses, which.
And then special operators don't come home.
Yeah.
Yeah, that's right.
And I think that with a normal clock, there's sort of a very well-defined relationship between something that happened before and happened after.
We call this a happens after relationship.
With a vector clock, that relationship sort of falls down because, you know, your clock can move forward.
My clock can move forward, and neither of us know how fast our clocks are moving.
And so there's this special relationship with vector clocks, which is sort of
much more ambiguous and with the traditional logical clock,
which is basically, you know, things can happen at the same time.
That's just impossible with the normal clock, right?
Because the time always increments when something happens.
But because things can happen at the same time, you actually build conflicts.
And so, you know, one of the core, you know, really hard parts of Nexus Peering from a technology perspective,
but then also from sort of a user perspective, is communicating conflicts.
Sort of helping the user understand when there was a conflict, how that conflict arose,
and then trying to sort of build the system such that it encodes as much intent about what the user meant to happen as possible.
And why is that important on the battlefield?
So, you know, I think the biggest reason is because.
You know, if you don't have that, you're going to smash conflicting edits in ways that just, you know, override one another.
And so if you lose that intent, if you lose sort of like, well, why did I change the data?
Why did you change the data?
We're going to make these arbitrary conflict resolution choices that sort of kind of result in either, you know.
You win because you happened after me or I win because I came before you.
But really, like helping the user understand it.
And we have a pretty multimodal understanding of both intent and conflict resolution
being able to sort of express the policy that gets everyone on the same page.
Like, you know, ultimately all this is about how do we provide a consistent view to everyone.
Right.
Well, maybe another way of putting it is if you're in a small unit,
you better have a unique but consistent understanding of the world view before you go behind enemy lines.
And in a situation where getting caught largely means a slow death.
Right.
And so there you know, you have to have that an absolute consistent understanding of what it means.
What does X mean in regards to Y?
Right.
What is the concept of an operational transform?
What does it transform and why is that also important?
Yeah.
So, you know, you can sort of think of two ways of communicating about the changing of data.
Right.
I can send you snapshots, which is sort of the full corpus and I can send you the whole thing or I can send you diffs, right?
I can send you the edits, the changes I'm making.
Being able to send the changes, they're much smaller and they're much more intent driven.
So if I give you a snapshot of my data, you can compare it against your data, but you can't know what I did to it.
You can't know sort of, how did I change it?
Right.
It's sort of a classic example in a database is, you know,
if the value is three and I tell you the value is four, you know, maybe I added one
or maybe I added four and subtracted three.
Right.
And so if we're trying to make edits to that, that field,
knowing the exact edits that I'm making sort of at a diff level, at an atomic level, which is sort of a difference.
Yeah, yeah.
Just like an atomic unit of change.
We just have to make sure.
Yeah, most people watching this video are not writing code, but yeah.
And so O.T.
is sort of a system that was designed and, you know, theoretically implemented in the eighties but never really brought to practical fruition, operational transforms, that is.
And it's a way of basically encoding sort of the intent behind the editor with the mutation.
So I can encode as much information as I want and then sort of transform it against the state.
And so if you get my sort of operations, you can then transform them.
Well, the lay audience often believes data is like just a natural resource that flows out
in the processing of the natural resource from something that is maybe coming from the ground to something that's pre usable through transforms
to then is actually usable through management of the transforms is just kind of very much under the hood.
Yeah.
And I think historically, you know, I think a lot of what we've been doing that I'm really proud of that I think is really, really special, is a lot of these things were just back end things.
You know they're things that database people talked about that developers talked about.
But actually building a system that brings it to the user and actually makes them user facing things,
you know, that's the art of sort of connecting the back end with the front end, which is, I think a lot of what we've done.
And then on your experience kind of running this like without obviously going into details that we can't share publicly
but like at a high level, like what what is the impact of, of this in things you've seen?
Yeah.
I mean, I think the biggest impact is just dynamism,
like so many of these synchronization systems require you to know how you're going to collaborate before you turn them on
and you don't know how you're going to collaborate before you turn the systems on.
And so being able to connect the systems after they're already on, after they're already running,
after people or, you know, using them independently, collaborating independently,
being able then build that bridge.
Like to me, that's the thing that really in the way, you know, you can build software in a lot of different ways.
But I think the way we built Nexus Peering was really quite general purpose.
We didn't, you know, we had a couple of cases to work with, but we built a key that wasn't fit for a single lock.
You know, we built a key that actually can go in a bunch of locks.
And so you can actually collaborate in a super dynamic way,
a way that, you know, today you want to collaborate, tomorrow you don't you can turn it off.
As an example.
If you were, it might be particularly relevant in current or current conditions
if American adversaries were collaborating in a wartime situation because of the things you mentioned,
security modalities, the the vector clock problem,
and then the ability to manage operational transforms,
actually leads to a context where allies that may not want to share everything or organizations that may not want to share everything
can collaborate in a way that our adversaries otherwise couldn't because they they don't have these tools.
Exactly.
In a year ago, we may not have wanted to collaborate that way.
Right.
Like, I think the dynamism of being able to turn it on and then turn it off.
And modulate.
Exactly.
So moving from, you know, a partitioned world to an on partition world, back to a partition.
I think the modulation that you bring up,
it's like in a in in world conditions that are like where the macro waves are just changing the world.
You basically need a back end that will allow you to modulate according to them, in either direction.
And yeah.
Yeah.
I mean.
When by the way, just another thing I'm particularly proud of is we didn't we start building this in like 2008.
2008, yeah.
Before I joined, you know, it was the first team I worked on.
I worked on it for ten years.
Shout out to Bob McGrew.
Bob McGrew, John Carino, Kathryn Brainard.
You know, a ton of.
Yeah.
Great people.
Yeah, yeah, yep.
Just a great people and I think.
Yeah.
I just.
It's.
I think with really great technology, it doesn't know what it's going to do when you build it.
You know, you build it in a way.
You build it.
In a way that sort of generalizes to do things you don't even intend for it to do in the future.
One of the things I'm very proud of is this 2008, and we must have
first of all, we I remember working on the problem or talking to Bob about it.
We had like six months before we figured out how we could solve the problem with basically bringing in the vector clock.
Mm hmm.
And then when do you think this basically was a product?
Like, I would say, like, two or three years later.
Yeah, but then you've kind of.
Yeah, probably.
2012, 2013 was the the first version of it that really felt pretty, pretty rigorously ready to go.
Um, I think the big shift since then has been moving it from pretty asynchronous,
really relatively high latency to really real time.
That's where the operational transforms came in.
And yeah, so moving from, you know, collaborating on units of minutes or hours down to sub second.
While remaining that adaptability, so that if you don't have those sort of communication channels,
you can still use the technology.
Yeah, but then, you know, a 2008 piece of unknown technology that is powering the world in 2022,
that's very cool.
It's really cool.
And I think we've learned, you know, I think in a lot of ways, from a technical perspective,
there's sort of two different types of complexity that we're grappling with.
In 2008, it was really a systems level complexity like can you make it work?
Just at a very basic level.
Since then, it's been almost like an evolutionary complexity.
So a lot of these systems calcify, right?
Because what you've basically done is you you've sort of forced two systems to to be able to communicate.
And so they can't evolve as quickly.
I think, with something like Apollo.
So when I was working on this a few years ago, when we released Apollo,
it was just a huge step change in terms of sort of decalcifying the system,
making it so that you could actually release every week, which is a whole different type of complexity.
So the system is running, but you know, you're evolving it on both sides across a multi node mesh, you know.
Those pieces have to move synchronously.
Okay.
I think this is really good.
Cool.