Welcome back to Stage three
of this advanced demo series.
In this stage, you're going to create a CodePipeline
which will include source and build stages.
Now the idea is that instead of having
to manually run the CodeBuild stage,
you're going to configure CodePipeline
so that whenever anything is committed
to the CodeCommit repository,
the build step is automatically executed
and a new version of the Docker image is created.
So let's go ahead and get started.
And to do that, we need to move to the CodePipeline console.
So in the search box at the top,
type CodePipeline,
and let's open that in a brand new tab
and move to that tab.
Now we're going to create a pipeline.
So go ahead and click Create pipeline.
And for the pipeline name,
put catpipeline.
Now as with most AWS services,
if we need to provide permissions
for CodePipeline to interact with anything else,
then that's done via a service role.
So in this case, we're going to allow CodePipeline
to create a new service role
and we're going to use the default name.
So AWSCodePipelineServicerole-US-East-1-catpipeline.
Now in addition to this, I just want to draw your attention
to this Advanced settings area,
so expand that.
And in here, we can set where the artifact store is located.
And this is where the output of any of the stages
within a CodePipeline can be stored,
if they're configured to do that,
and where the inputs for any stage can be loaded from.
And this is how you tie individual stages together
within a CodePipeline.
Now we're going to leave this as the default location
which will create a default S3 bucket in this account.
We could also choose a custom location
but we're going to use the default.
And we won't be changing the encryption key settings.
We'll be using the default AWS managed key.
Okay, so let's go ahead and move on.
And first we need to define the source stage,
so the source provider.
So we're gonna click in the dropdown
and select AWS CodeCommit.
We'll need to select the CodeCommit repository
that we've been using to this point.
So click in the name box and select catpipeline-CodeCommit
and whatever random numbers or letters
you defined on the end.
So go ahead and click your repository.
Just as before, we're going to be using a branch.
So clicking that box and select master.
This is the name of the branch that we have.
You might also have main,
or in production usage,
you might have a different set of branches.
But in my case, it's master.
Now in terms of the Change detection options,
you can have CodePipeline periodically check
the repository for any changes,
but we're not going to use that.
Or you can use CloudWatch Events.
And this means that whenever any events happen
to the CodeCommit repository,
this will automatically be detected in an event-driven way
by CodePipeline.
And that's what we want.
So we're going to pick Amazon CloudWatch Events.
And for output format,
we're just going to use the CodePipeline default.
This means that the default zip format
will be used for artifacts in the pipeline.
And that's fine, that's all we need.
So go ahead and move on to the next step
which is the build stage.
For Build provider, we're going to pick CodeBuild.
We can also use Jenkins,
which is a third party build product.
I'll be talking about that elsewhere in the course
but for now we can go ahead and use CodeBuild.
You need to pick the region
that the build project is located in,
in our case, it's US East 1,
so we can leave this selected.
And then for Project name, click in this box
and pick catpipeline-build.
Now this build project already exists
and so we don't need to provide
any other environment variables.
For Build type, we're going to be using a single build
because this is a simple project.
For more complex builds,
you might use a batch build
but for this demo lesson,
we're gonna go ahead and use single build.
So until this point, all we're doing
is including the previously configured elements,
so the CodeCommit repository and the CodeBuild project
which we've previously created and configured.
Now let's go ahead and move on to the next stage
which is the deploy stage.
And we're not actually going to configure this
in this stage of the demo series.
That's coming up next.
So we're going to skip the deploy stage.
So click Skip deploy stage.
You need to confirm that process,
so click on Skip.
We're given the opportunity
to review everything we've configured,
everything looks good.
Scroll down to the bottom
and then Create the pipeline.
Now the pipeline's going to perform an initial execution.
And so if you leave this screen displayed,
what you'll see is the source stage will run
and it should succeed.
And then it will move on to the build stage.
And the build stage will execute
based on what's inside the CodeCommit repository.
And you should find that this build stage
runs without any problems.
What I'm going to do is I'm going to go ahead
and pause the video
until my build stage either succeeds or fails.
You should do the same and we can continue
once the status changes to succeeded or failed.
Now in my case, the build stage was successful.
If yours failed, one very common reason
is throttling that's performed by Docker Hub.
Because we're using Docker Hub as part of the Docker file
to download the CentOS-7 image,
it's possible that that is throttled.
And so if you're doing this at a busy part of the day,
it's possible that you might be throttled.
Now this build step succeeding
is not essential for this part of the demo series.
So don't worry if your build step did fail.
At this point, we're not going to worry about it.
And if this was in production,
we probably wouldn't be using Docker Hub
to download the CentOS-7 image.
We would probably be using ECR.
For now though, regardless if the build step
was successful or not, we're going to continue.
Now, what I want to draw your attention to
is the artifact bucket in S3.
So click in this Search box at the top
and type S3 and then open a new tab to the S3 console.
Move to that tab
and open the bucket,
which begins with CodePipeline-US-East-1.
Inside that bucket,
there should be a folder called catpipeline.
Go in there.
And inside there,
currently there will be a SourceArti folder.
So inside there are the source artifacts,
and this is just a zip file.
Now we don't need to worry about that for now.
We're just gonna go back up one level
by clicking on catpipeline.
So this is where the artifacts
will be stored for our CodePipeline.
And currently, we're not generating any
as part of the build step.
Now we do need to generate an artifact
so that we can later use CodePipeline with CodeDeploy.
So the next step is we're going to update
our buildspec.yml file.
And we're going to add some additional lines
including a directive to generate some artifacts.
So I'm going to move across to my code editor.
So this is the existing buildspec.yml file.
This is the one that we've already got
on our local copy of the repository.
What I want you to do
is to refer to the updated code snippet.
So there's an updated buildspec.yml file
contained within the Stage three instructions.
So I want you to copy all of that into your clipboard,
and then select all of the existing version,
delete it, and paste in the new version.
Now as before, I want to make sure
that there are no tabs in this file.
So I'm going to select all of this.
In my case, with this editor,
I can click on Editor and Convert tabs to spaces.
Now depending on what code editor you use,
this might be in a different location
but you need to make sure
that there are no tabs in this document
because strictly speaking,
you can't have tabs in a YAML document.
So we need to make sure that they're all spaces.
So this updated buildspec.yml file
does a number of things.
Firstly, it gets the commit hash from CodeCommit.
So whenever you commit anything
into a CodeCommit repository
or most source control systems,
you get a commit hash or a commit ID.
And what we're doing on this line
is getting that and putting it into a variable
called commit_hash.
In addition to that,
we're also setting a variable called image tag
and we are doing that within pre-build.
We're still authenticating against ECR
using AWS ECR get-login-password,
as well as Docker login in the pre-built section.
In the build section,
we're still outputting these diagnostic messages.
And we're still running a Docker build
against the current directory,
so that's using the Docker file to generate a Docker image.
And then we're tagging that Docker image.
Now where the bulk of the changes are,
are in the post-build section.
So what we're doing now in the post-build section
is we're running two different Docker pushes.
One, which is pushing with the latest tag,
and one, which is pushing with the image tag.
Now the image tag will include the commit hash.
And that means that every time
we push a new Docker image into our repository,
we get a uniquely identified Docker image
based on the commit ID
that we make to the CodeCommit repository.
In addition, we're updating the latest tag
to always represent the latest Docker image.
So basically what that means
is we'll have a history of any Docker images
and they'll be mapped back to specific commits,
and then we'll always have the latest tag.
Now in addition, what we're doing is we're creating a file
called imagedefinitions.json.
And this is what's used by CodeDeploy
when we're deploying through to ECS.
So this is an ECS-specific implementation detail.
So we're generating imagedefinitions.json,
and then we're defining this as an artifact.
So this means that this is going to be stored
within the artifact bucket,
and it's going to be available
to the next step of the CodePipeline when we create it
which is going to be the deploy stage.
So now that we've got that updated buildspec.yml,
I want you to go ahead and save that
into your local copy of this repository.
So save that file.
Then I want you to go back to the AWS console
and go back to the CodePipeline screen.
I want you to click on Pipelines,
and then click on catpipeline.
And you can see the previous step is succeeded.
So this is currently not running.
So the source stage is not running
and the build stage is not running.
What I want to demonstrate now is what happens
when we do a commit to the repository.
Now we've currently updated the buildspec.yml file.
So this is a great thing that we can commit
to the repository and see exactly what happens.
So go ahead and move to your terminal.
I'm gonna clear the screen to make it easier to see
and then I'm going to rerun the same three commands
that we're used to.
So, git add -A.
And this will add all of the files in this folder
including the updated buildspec.yml.
Next, git commit -M
and then "" to add a message.
And then in there, I'm going to put
"updated buildspec.yml".
And that means that we just know what we've updated.
And then I'm going to use a, git push, to push that commit
into the AWS CodeCommit remote repository.
Now, because we've set up this CodePipeline,
what's gonna happen when we push this commit
is the CodeCommit repository
is going to get this updated commit
and it's going to generate an event.
That event is going to be seen by CloudWatch Events.
And that's going to be used to initiate
a run of our CodePipeline.
So let's do this push
and move back to the AWS console.
Now, what we should see when we're back at the console
is that once this commit has occurred,
once this event has been generated,
we should see that this pipeline begins an execution.
So I'm gonna refresh.
And we can see that the source stage has succeeded
and the build stage is currently in progress.
And that means that this has worked.
We've run a commit to the repository,
this is the commit ID.
And the comment is updated buildspec.yml,
which is what we've just done.
So we've generated an event using CloudWatch Events.
That's caused this pipeline to execute.
The source stage has succeeded
and now the build stage is in progress.
And this is generating a brand new Docker image
with the updated contents of the repository.
So this is now a completely automated process.
Whenever we commit anything to our CodeCommit repository,
this pipeline is going to execute,
and it's going to generate us an updated Docker image.
Now we just need to give this a few minutes to run.
Once it does complete, I want to show you
exactly what's been produced.
So we're going to have a look in ECR
as well as have a look in the artifacts bucket
because both of these are important to understand.
Now, again, if you do get a failure in this build stage,
you can go into the build stage and retry it.
It's possible that you might get a throttle error,
again, because Docker Hub are throttling
requests made against it from AWS.
Hopefully though, in my case, it's going to be successful.
And it is, it has succeeded with the build.
So now I want to show you two things.
So first, we're going to go back to ECR
and we're going to do a refresh
of the catpipeline repository.
Now, instead of just the image tag of latest,
now we have the image tag of latest,
comma, and then the commit ID.
So because we've only got one updated Docker image,
this is tagged both with latest and with the commit ID.
If we rerun the process that we've just run,
we get a new docket image.
That one would be tagged latest,
and this one would only have the commit ID.
So we're maintaining the history
of any images that we're creating
by using the commit hash or the commit ID.
So that means that we can always
map back these Docker images
to the commit that was used to generate them.
And that's important, especially in the real world.
Now if we go to the S3 console,
you should still have a tab open to it.
And if we refresh,
now in addition to the source artifacts,
we now have another folder
which represents the build artifacts.
If I go inside there,
we can see a single file.
So if I click on it,
click on download,
that will download it to your local machine.
If I go to my downloads folder,
here's that file.
If I click on it and add.zip onto the end,
and then extract that file,
what we can see inside it is imagedefinitions.json.
And this is the file that's been generated by CodeBuild.
It's set as an artifact,
it's stored in the artifacts bucket,
and it's going to be used during the next stage
when we use a deploy stage to deploy through to ECS.
So I'm just gonna go ahead and open this file.
And this is the contents of that file.
It essentially specifies the catpipeline name
as well as an image URI.
And this image URI points to the docket image
that we've just generated.
And so this can be used during a deploy stage
to deploy this out to ECS.
And we're going to be using this
during the next stage of this demo series.
Now at this point, that's everything that you need to do
in Stage three of this demo series.
So in Stage three, you've successfully set up
the first few stages of a CodePipeline;
the source stage and the build stage.
And you've made it so that now we have a build operation
that's performed whenever anything is committed
into our CodeCommit repository.
And, of course, the output of that
is a brand new version of the Docker image
together with the image definitions file
which I've just demonstrated.
Now in the next and final stage of this demo series,
we're going to bring everything together
by adding a deploy stage
which will be deploying our docket image out to ECS.
But at this point though,
that's everything I wanted to cover.
So go ahead and complete this lesson.
And when you're ready,
I look forward to you joining me
in the next stage of this demo series.