♪♪
-Welcome to DigitalOcean Deploy.
My name is Paul Stack, @Stack72 on Twitter,
and my talk today is a talk called,
"Infrastructure as Software?"
So we all understand that there's lots of change
of companies moving to the cloud right now,
and trying to get the system
that they've developed correct to work in the cloud.
We...There are a number of variations
in how people actually deploy their systems
into the cloud, we can look at these.
So we have a V1 style system, and that's effectively...
They've picked up their entire dataset,
they've moved it to the cloud in its current state,
and you usually have an interior style architect here.
So you have web servers, you have application servers.
They're more than likely long-lived,
so because they're long lived
they're probably used in something
like a configuration management tool, maybe Puppet,
Chef, SaltStack, Ansible, one of these different things.
And it tends to be a little more static
than how you actually move.
It's, like, traditionally, you know,
you don't tend to use anything in the specific cloud itself,
but you're understanding
that there are a lot of different pieces in the cloud,
but you're just not, maybe, ready to use them right now.
And then you have people who have gone a little further,
a little more beyond that,
and they have a much more transitional-style architecture.
So when I say a transitional- style architecture,
I mean like, you know, they're starting to take advantage
of maybe a mixture of containers and VMs,
probably using like Docker or, you know, a hosted service
in the specific cloud that's there.
They could be bringing in, like,
a hosted database or using their own database,
and they definitely will be taking advantage of,
like, a monitoring SaaS-style tool.
So understand how their systems are working in production.
And then you have people that have gone
even further beyond that,
and this is what is starting to be called,
like, modern-style infrastructures in the cloud.
You've probably heard terms
like cloud-native or microservices
or all these different things,
and this style architecture, this V3 architecture,
is comprised of a lot of these different terms.
So you're probably taking advantage...
You're definitely taking advantage
of some compute-style services.
You're possibly taking advantage of some serverless,
so some functions, or some lambda or something like that,
depending on the cloud that you're working in,
and then more than likely,
you're taking advantage of a container hypervisor,
so something that's actually orchestrating
all of the different pieces
that's going on in there right now.
And at this point in time, it's quite popular
and fashionable to use Kubernetes
or HashiCorp Nomad or something like that.
And then, of course, you could be taking advantage
of other systems as well,
like machine learning or big data
or any of these different things in order to help you
actually understand what's going on in your system
and actually build more sophistication
and connected services between what's going on in here.
So traditionally, there are three ways
of managing the cloud.
We have people... And it's not...
You know, and it's okay to say
that they're still doing it today.
We have people who are still using the web console
to provision services in the cloud.
It's not the most beneficial way of building things.
It's kind of hard to scale.
It's difficult for users and resources,
and, of course, it's not really repeatable
and moving and stuff...
Repeatable in how you are actually building the things.
And you have other teams
who are going past the web console stage.
They're probably using, like, CLI
tools, scripts, templates,
some things that are actually starting to move
towards what we would class as infrastructure as code,
which is much more automatable.
You know, it can be versioned correctly,
and source control, it's reusable.
It's probably item potent,
but trying to roll things out is a little more difficult
because, you know, it's your own scripts around different things.
You need to understand changes in the upstream APIs,
changes in the system that's going on,
and it becomes a wee bit more difficult to understand
what's happening in here.
And then you definitely have a lot of people
in this era of engineering who are starting to use code,
and you're actually starting to take advantage
of writing your infrastructure
in terms of how you would write application codes and version
and testing, all these different things that pop through.
And different teams can have a variation of different ways
even within the same team,
but these are the main ways of automating the cloud.
Now, moving to the cloud is difficult.
Adoption of the cloud is tough because we have a lot of teams
who are built of legacy, sort of IT organizations,
and they still have a lot of silos in their organization.
So we have, like, developers and operators
working in completely different systems.
You have teams that have worked on a cloud in one environment
and a different cloud in the new environment,
and then, you know, it's difficult,
and it's error-prone for the tools,
and it's difficult for them to get started with.
And then as the current situation,
this era of COVID has made things even more difficult
because we have to increase the speed of our adoption
or movement up to the cloud much faster.
It's actually even tougher to get started in there
because the multitude of services available right now
actually means it's very difficult for people
to understand
how to get their system to run in the cloud.
It's not cloud native by default.
It's probably a Brownfield application and old application,
and it's slower and more difficult for people
to actually push it into the cloud.
So this is where cloud
engineering is starting to come in.
This is a new term that's starting to appear
and starting to be understood by a lot of different people,
and it's much because DevOps has transformed
a lot of IT organizations.
So, you know, you used to have centralized ops teams,
but of course they're passing the power back now
to security teams, to application teams.
We even have infrastructure teams, SRE teams.
We have all these different people
managing different parts of the system.
It used to be, and I'm sorry to have to say this,
that operations teams used to be the gatekeepers to the system,
and they used to be the people that said,
"No, you cannot release that," or, "It's not fit for purpose."
But it's now that the operation teams
are the enabler for the system,
so they're actually a lot of the teams
that are driving the change, riving the transformation
and driving what's happening in the system.
Things are changing as we go through,
and its very interesting to see how this is actually working.
Now, of course, that means that there are
different ways of building things.
Now, we're starting to build modern architectures,
so this is where we class cloud natives.
You know, it's more Greenfield applications.
It's more split out. It's more microservice-based,
and it's taking advantage of VMs, containers and serverless.
You know, that sort of empowers these high-modern teams,
which actually are able to continuously deliver systems
into he cloud, of course in a safe way
in combination of the operations team and the security team.
And, of course, if we can get this right,
we're able to be in this multi-cloud scenario
that a lot of people dream is the Nirvana,
where we have private/public hybrids
even across different cloud vendors
with built-in security and all these different things.
So it's very much a changing ecosystem.
And not only is it a changing ecosystem,
the goalposts are continually moving on the tooling
and everything that's there.
So just a very quick recap, what infrastructure as code is.
Infrastructure as code is bringing the best practices
to our infrastructure management.
It's providing a way to collaborate
on infrastructure changes,
and we track our changes from code commits and source code,
the whole way through to deployment.
It's important because it replaces manual error-prone
processes with automation, and that's, of course, the goal.
That's the key of what we're actually trying to do.
It's faster to get our code into production
because it's automated.
We have pipelines, and we can actually test things
as it's going through.
And the most important thing about infrastructure as code is,
it brings safety and predictability
to our infrastructure
because we're able to see and plan in advance,
what it's going to do,
and because it's item-potent, we will understand
that it's not just going to break things every single time.
Now Pulumi is a new Modern Infrastructure as Code tool
that allows us to start bringing programming languages,
or modern familiar languages that developers
and operation teams use into the workflow for the cloud.
And it works with any cloud.
We have a lot of providers.
It's open-source, and it's free, of course,
but it has a commercial SaaS aspect to it as well.
Now, it's very important to talk
about a few things for Pulumi first.
One, it is declarative by default,
so what you specify in your application
is actually what will be deployed,
but how it gets there is through imperative applications.
Now, even though we said that it uses programming languages,
you do not need to use any programming constructs
by default.
You will actually just use declarative syntax,
and you will layer a Pulumi SDK for a specific cloud.
But when you start to be able
to get more complexity in your system,
you can then start to refactor your code.
You can use conditionals and for loops and programming
constructs and what's happening.
You have the ability
that you can create multi-provider workflows
so that you can have the outputs of some resources
as the inputs of others
because Pulumi works on the idea of promises,
and it will resolve those promises at run-time.
And because of that, we're actually able to mix
and match different deployments.
In this example, you're storing some data in a storage bucket,
and you're using that data to be passed
into a Kubernetes deployment.
We very much build on top of the language ecosystem,
so that means that we can actually package
our infrastructure and code our infrastructure
using things like Pype, NuGet, NPM
and be able to create those specific packages
and share them around our colleagues and our teams.
And then some of the more interesting things about Pulumi
is that it actually enables this advanced orchestration,
these advanced strategies that we're really hoping.
So what we'd usually do for Kubernetes deploy
is that we would deploy a couple of instances of our application.
We would then go to our monitor or metrics tool,
and then we would come back to our application
and we would continue the run that...
The system of how it's actually going.
But with Pulumi, it actually can talk to any API or any SDK
that is currently in the code that you're writing.
And in this case, we can see that we deploy three replicas
of our application.
We then go off, and we check Prometheus.
When everything is okay with Prometheus, we continue
with the next sort of 10 replicas of our application,
and we can layer that through our workflow
as we actually need to go.
Now I said that Pulumi was starting to be able
to move us towards infrastructure
as software because of the fact
that it's Modern Infrastructure as Code with a need
to back up real teams and how real software works.
And one of the ways that we're actually able to do that
is by introducing test-driven infrastructure,
or test-driven development,
in our infrastructure development.
Now we have given you different ways
that you can adopt the testing pyramid,
so unit testing, integration testing and run-time testing
all can actually embed as part of your CI/CD workflow
as you will see very shortly in my demo.
And, of course, because of that, we can actually start
to really test the code that we're writing,
not just hope that when we deploy the applications
it will work as expected,
and we will have to then debug as we need to go through it.
One of the things that's important here to say
is when I say test driven,
I actually do mean we can be test driven
because we can mock the cloud.
We can mock the requests and the responses
coming back from the cloud vendor,
which means we can actually black-box develop our code.
And then when we're actually confident
that we're ready to start testing it,
we can then bring up our infrastructure
and test it as we need to,
rather than spending lots of time and money
on iterating within a specific cloud itself.
And then probably my favorite thing about Pulumi
is that we try and embed security practices
as part of what we're actually trying to do.
So for me, it's...You know, testing is one thing.
Development is another thing, but security is a huge thing
right now that we're not very good at in this industry.
We're trying to get better at it.
So we try and give that same sort of experience
that security is important right from the outset,
the same way as testability and CI/CD is.
So we give you ways in which you can actually implement
security management of your secrets
and your configuration as part of your Pulumi pipeline.
Now, of course, we can book...
You can use the Pulumi SaaS to do this,
or you can actually use systems like HashiCorp Vault, AWS
KMS, GCP KMS or Azure Key Vault in order to do this.
So as more and more security tools come out,
we'll integrate with more tools
in order to make, like, each team's scenarios work here.
So let's go and actually have a demo
of what we're going to do,
and in this demo, we're actually going to look at the testability
of what's actually going on in here.
Now as you can see from the results of the demo
that the CI/CD
workflow is actually embedded the whole way
through the Pulumi infrastructure test
and application that I showed you.
So we can see that straight away, we do a yarn install.
So this is a TypeScript application,
as I told you,
and that will allow us to install the latest dependencies.
Now of course if you need it to be against a local dependency,
you can include that as part of what you're doing,
but, you know, we actually have a yarn installed,
which will go off on NPM and install different things.
Now as part after that, we then Pulumi stack in it.
We run a Pulumi preview to make sure that everything looks like
it's going to be okay.
Then we run a Pulumi Up.
Now of course here we have to run it non-interactive mode
because we don't want to wait for anyone to have to click
anything on that on the terminal,
but we're in a situation right now where we have
an application that's actually being brought up.
Once we have the application up,
we just perform some extra tasks, okay?
We export the stock.
We re-import it to make sure
that the integrity of the stack is there.
Then we run a preview after we import it and another Pulumi
Up to make sure everything works as we expected.
And then what we actually do is we perform
that extra run-time validation
that we actually encoded as part of our tests.
So here, we're actually going off to the endpoint
of our load balancer that we created in DigitalOcean,
and work actually testing that the N-point actually brings back
the correct HTP request, which, as we saw is,
"Welcome to Engine X."
And once we're happy with that, then we can destroy
our application, and we can remove our stack.
So, you know, this is the entire workflow encoded
in our application acceptance test,
and that way, we can continually run the same workflow
that we would do in our CI/CD pipeline
as part of our testing and poll requests.
So we're trying to bring that system
into actually what we're doing here.
So we actually believe that as we start to move towards
this Modern Infrastructure as Cloud that Pulumi
is going to be the easiest way for a lot of teams
to actually get that going because we have thought
about what different teams need to do as they go forward.
We thought about the testability, the security,
the CI/CD pipeline,
the reusability, the versioning and the packaging
because we believe that this is actually what a lot of teams
are actually going to need to do,
what a lot of teams want to do and what a lot of teams
have to move towards in the future
as they start to make their cloud journey faster
and their velocity
to deploy their applications in the cloud much faster.
So I'm going to say thank you so much for your time today!
I'm going to say enjoy the rest of your conference!
If you need any further information about anything
I've shown you today, you can get in contact with me,
paul@pulumi.com, or you can tweet me @Stack72.
I'm always on Twitter. I'm always available,
and thank you so much for having me.