- Facebook spent 25 person years to create Threads.
25 to 50 engineers spent seven months straight
working on it.
But what if I told you they could have made Threads
with one engineer with under 10,000 lines
of code at least 30 times faster and cheaper.
If they were using Rama by Red Planet Labs they could have.
This new data backend platform
just dropped last week and I think it's one
of the most important developments
in software engineering in decades.
The creator of Rama is the prior creator
of Apache Storm and literally wrote the book on big data.
His name is Nathan Mars
and I'm proud to say I was one
of the first investors back at Initialized.
When I first met him I realized if Rama had existed in 2008
my startup Posterous wouldn't have died.
And if you're watching, your startup might not die today
because you got to see this video.
Watch to the end to find out exactly
how a brilliant new abstraction
for data structures means you won't be burning runway
to recreate the wheel of data again and again.
Let's get into it.
(upbeat music)
How programmers write software that works
with data is still stuck in ideas created 40 years ago.
But if we build it for modern systems
with lots of disk, lots of RAM lots of CPU
and knowing it's going to be in a cluster
and we do it with the right abstractions built right
into the data system, you get something magical.
- It's a totally new approach to building backend system
and what Rama does is it integrates
and generalizes every aspect
of what it takes to build a backend system at scale
and it enables you to build backends
to applications in 100xless code.
- Most devs today use some mix
of MySQL, Postgres, Oracle, MongoDB, Redis, Snowflake,
and each of those systems is from the ground up
a new system to store, index, and retrieve data.
But with that is a whole new set of code and tools.
We're literally repeating ourselves over
and over again in that data stack.
Each new data stack is a new code base, new set of servers,
new DevOps needs and more surface area for bugs.
- And so what Rama does is it enables you
to build applications in fundamentally different ways.
Because it's such a general purpose system
you don't need to combine dozens of tools together.
You can just use this one tool and if we look
at like one piece of Rama to see like how is it possible
it could be so general purpose,
able to do what all these databases can do combined?
You can look at how Rama does indexing.
(dramatic music)
- My favorite thing
about incredible technical teams is they're able to back
up these strong words with a real feat of strength.
In Nathan's case, one of the key examples he shipped
was a full server implementation of Mastodon.
A Twitter like social network,
but one that actually scales now
with his code to Twitter scale.
- So we built a a scalable clone of Mastodon
which is basically the same as Twitter
when you compare it to the consumer product,
and the amount of cost it took Twitter
to build the consumer product was about a million lines
of code and 200 person years of time.
The cost for us to build essentially the same thing
on top of Rama was 10,000 lines
of code and only nine person months of time.
So it's 100x less code,
but actually more than 200x less person hours.
- Rama allows developers to do this in a fraction
of the time because the abstractions Nathan
and the team created enable auto-scaling
and extreme expressiveness.
- The main difference between what we built
and the open source Mastodon
is the fact that hours can scale.
The open source Mastodoon cannot scale anywhere
near what we can.
So like in our instance that we have live right now,
mastodonrevfile.com, we're running it with
a hundred million bots posting 3,500 statuses per second
at an average fan out of 403, which is many orders
of magnitude beyond what the open source Mastodon
can possibly do.
And we're less code than them.
Our backend is at least 44% less code than theirs.
- Less code to do the same thing
is the holy grail of software.
And that's why Rama is such a crazy leap forward.
(upbeat music)
What Nathan has figured out
is how to help software engineers
do the most basic things they try to do everywhere else
in their code, but haven't been able to do
in how they use data systems today.
Don't repeat yourself.
D-R-Y.
Rama is D-R-Y, dry, for how you store and retrieve data.
The programming model isn't new, but the way it integrates
the pieces together has never been done before.
- [Nathan] What makes Rama so powerful
is how much it integrates and generalizes
each of these pieces.
On the left are depots.
These are distributed, durable, and replicated logs of data.
If you're familiar with Apache Kafka,
depots are very similar except integrated
with the rest of Rama.
- Depots are append-only replicated logs.
They're the fastest way to stream data to disk.
- [Nathan] Next are ETLs,
extract, transform, load topologies.
These consume data from depot's in parallel
to produce indexes, which we call partition states.
Most of the time spent building Rama applications
is spent programming ETLs, which are programmed
with a pure Java API.
Next are partition states
which we usually refer to as PStates.
PStates are how data is indexed in Rama
and they're also durable and replicated.
They serve the same purpose that databases do,
but they're much, much more powerful.
The last concept in Rama is queries.
- When you think about scaling Twitter or Mastodon,
this is where complex scaling operations that might happen
at the app level can be brought down into the data layer
and scaled automatically like the de-normalized fan out
of feeds for posts going to followers.
- [Nathan] There are two forms of queries in Rama.
The first is ad hoc queries that can retrieve
or aggregate data from one partition of one PState.
These are called point queries.
The second are query topologies
which are predefined queries that can look
at any or all of your PStates
and any or all of the partitions of your PStates.
- Instead of trying to fit
all your data inside these simple,
but ultimately too unstructured forms of tables,
columns, and rows or even JSON blobs,
these concepts let you create data structures
that really mirror the underlying information
that you're trying to represent.
- [Nathan] PStates do in Rama what databases do
in traditional architectures
and you can understand PStates in contrast to databases.
When you look at databases
they all have fixed indexing models,
what they call data models.
There's key value, document,
column oriented, graph, and so on.
A better way to think
about indexing is in terms of data structures.
A data model is just a specific combination
of data structures.
Key value is just a map.
A document data model is a map of maps.
Column oriented is a map of sorted maps and so on.
Databases that support secondary indexing
just materialize additional maps
alongside the primary data structure.
A database like Redis is a little more flexible,
allowing your values to be sets or lists.
But ultimately all these databases
and their underlying data models support a very small number
of data structured combinations.
We're talking about just a handful
versus the infinite number that are possible.
Building backends has always been like this,
at least since I've been alive.
So it can be hard to see this as a limitation,
but this is like if in regular programming
you could only use a handful
of data structured combinations and nothing else.
- None of this is abstraction for abstraction's sake.
When people conceived of SQL databases
they didn't have a sense that people
in the future would have plentiful access
to data centers where the dominant way to scale
would not be bigger servers, but more servers.
Rama is built for this reality
and this abstraction allows you to keep the data structures
in one system instead of many.
- [Nathan] But this is the reality of backend development
from databases being so restrictive.
If you can't fit a use case onto your database's data model
then you need to deploy another database
with another data model.
Deploying and maintaining multiple databases is expensive.
If no database's data model fits your needs,
then you might need to make your own database from scratch
which is extremely expensive and unfortunately
not that uncommon.
Rama's PStates are completely different.
Every PState you create can be
whatever data structured combination you want
and you create as many PStates as you need
to support all your use cases.
- When Nathan says new databases, this isn't merely
just partitioning data and yet new database clusters.
Often in companies you're using
a whole new stack, MongoDB, Redis,
and back when I was running systems for my startup
at some point I realized I was just doing a crude version
of partitioning across new database-like runtimes.
And if you're doing that, you lose the ability
to join that data across systems to do useful things.
And not only that, you're actually compounding the problems
of scaling data systems.
Just because you've moved a table
or data to one microservice
to a separate stack doesn't mean the problem is solved.
It just means
that you need to shard that data further later.
And then you've got two problems
because you are tracking entire tech stacks, contracts,
support contacts, logins, auth, backup, and then all
of those other things that are necessary
when you add servers.
All of this in a modern data stack is a waste.
- Everything about Rama
is inherently distributed and scalable.
And so when you deploy a Rama cluster
you tell it how many partitions
and how many workers to deploy it on.
And then Rama has this great built-in monitoring
which basically tells you
when it's gonna need more resources.
And then it's literally just a one-liner
to terminal saying okay, I'm going to increase
this application currently running a cluster.
It's currently running with 20 workers,
let me increase that to 30.
And so the one liner, you press enter,
takes a couple minutes for Rama to execute the transition
and then that's it.
That's all it takes.
- So, that's Rama in a nutshell.
It truly is possible to change your abstraction
and get a massive boost.
At my startup Posterous we got stuck in MySQL scaling hell
and I never got out of it and our startup suffered for it.
Better software is possible and it's here today.
- So in terms of what's available right now.
So number one thing is we've released a build
of Rama that anyone can try out and use.
It lets you build full Rama applications
and run them within a simulated cluster environment.
So we've, on our documentation that's available
on our website,
you can find the details on how to access that.
And we're not gonna hide
behind the fact that Rama has a learning curve.
It definitely has a learning curve,
but the benefits are so extreme
that it is well worth your time to learn it.
And the best way to learn it is to start
with the documentation.
The documentation contains a six-part tutorial
that gently introduces the concepts and culminates
in building a complete social network end-to-end
and only 180 lines of code.
And the rest of the documentation does deep dives
on all the aspects of Rama.
So come check it out at redplanetlabs.com.
- Link in the description right now
to try Rama for yourself.
The startup that you save might be your own.
That's it for this week.
Thanks for watching all the way to the end
and I'll see you next time.
(light music)