Complete GitHub CI/CD Pipeline & CodeBuild In CloudFormation!

As software development teams continue to embrace DevOps methodologies, the need for efficient and reliable continuous integration and continuous delivery (CI/CD) pipelines has become increasingly important. GitHub, one of the most popular code hosting platforms, offers a robust set of tools for building and deploying applications using CI/CD.

In this article, we’ll walk through how to set up a complete CI/CD pipeline in GitHub using CloudFormation templates. We’ll cover everything from creating a new repository to building your code and deploying your application on AWS.

So, without further ado, grab your coffee, and let’s dig in!

What is CloudFormation?

AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so you can spend less time managing those resources and more time focusing on your applications that run in AWS. You create a template that describes all the AWS resources you want (like Amazon EC2 instances or Amazon RDS DB instances), and AWS CloudFormation provides and configures those resources for you.

Why use CloudFormation for CI/CD?

1. Infrastructure as Code: By defining your infrastructure as code, you can version control it alongside your application code. This makes it easier to manage changes to your infrastructure over time.
2. Consistency: By using the same template to create environments across different stages of your pipeline (e.g., dev, test, approval, and prod), you ensure consistency in configuration and reduce the risk of errors.
3. Automation: With CloudFormation, you can automate the creation and destruction of environments as needed, reducing manual effort and increasing efficiency.

Now let’s dive into setting up our CI/CD pipeline using GitHub and CloudFormation.

Step 1: Generate a GitHub Personal Access Token

This will be used for authentication when we will use GitHub API.

In GitHub, go to Settings> Developer Setting > personal access token > click on generate a new token.

Then name it — AWS_Token for instance.

Select the repo and admin repo hook checkbox for the token, then save the token in the AWS Secret Manager for further use.

AWS Secret Manager

Step 2: Create a new GitHub Repo

The second step is to create a new repository in GitHub where we will store our application code.

To do this:

1. Log in to GitHub.
2. Click on the “+” icon in the top right corner.
3. Select “New repository” from the dropdown menu.
4. Give your repository a name (e.g., aws-cicd-repo).
5. Choose whether it should be public or private.
6. Select Add a README file and Add .gitignore file.
7. Click “Create repository”.

Step 3: Set Up Your Development Environment

Before we start building our pipeline in CloudFormation, we need to make sure we have everything set up locally:

1. Install Git if you haven’t already.
2. Clone your newly created repository (aws-cicd-repo) onto your local PC using Git.

> git clone git@github.com:saedHasan/aws-cicd-repo.git

The file structure should be like the below:

3. Add some comments in the README file and Commit to Git with a message like “Initial commit”.

> git add . 
> git commit -am "Initial commit "
> git push origin main #if your master branch called main 

Step 4: Create Your Pipeline Template

Now that we have our application code committed to Git, we can start building our pipeline template:

1. Log in to AWS Console
2. Navigate to AWS Cloudformation Service
3. Click on “Create Stack”
4. Select “Upload a template file”
5. Upload the “pipeline.yml” file which will include all necessary infrastructure components required for CI/CD Pipeline ( GitHub, CodeBuild, and S3 ).

For the stack details, enter the Stack name github-cicd-cf , and for other details, it’s auto-filled from the Parameter section in the pipeline.yaml file ( CodeBuildEnvironmentImage , GitHubBranch ,GitHubRepository and GitHubOwner )

Once the template is executed, you can check the Events of creation of the resources and as well as check the Resources created from the template.

Another option is to run the template in the CLI:

$ aws cloudformation create-stack - stack-name my-pipeline 
-template-body file://pipeline.yml -parameters 
ParameterKey=GitHubUsername,
ParameterValue=<your_github_username>,
ParameterKey=GitHubPersonalAccessToken,
ParameterValue=<your_github_personal_access_token>,
ParameterKey=GitHubRepositoryName,
ParameterValue=<your_github_repository_name>,
ParameterKey=BranchName,
ParameterValue=<your_branch_name>

It will take a few minutes until all infrastructure components are created successfully by cloudformation stack creation process completed successfully.

The “pipeline.yml” file includes several components such as S3 Bucket which will be used as an artifact storage location where build artifacts will be stored after the build process is completed by CodeBuild service; CodeBuild Project which will be used for building LINUX_CONTAINER images; CodePipeline which will orchestrate entire process including source pulling from Github Repo; then to Build stage and Dev and AQ stage, then to Deploy stage. It also has the Service Role needed for CodeBuild and CodePipeline services.

Step 5: Create Develop branch and pull the requests

After the creation of the pipeline, we need to simulate and test the CI/CD in GitHub with CodePipeline, which means, we need to trigger the pipeline whenever there’s a change committed to the github repo from the developer, the pipeline then will run the four stages ( Sorce, Build, Dev, AQ, and Prod).

To do this; first, create the “develop” branch in git command

> git checkout -b develop

next, commit the change to “develop” branch

> git push origin develop

next, you can pull the request from develop to main branch to validate the webhook.

after clicking on the pull request and comparing the changes, click on the Create Pull request button.

Notice a link to the CodeBuild project should appear as described below.

From the drop-down menu in the button Merge pull request, select the second option Squash and merge.

Then you can validate the pull request is trigger the build command in the CodeBuild console.

click on the Squash and merge and then click on click Confirm squash and merge.

Step 6: Add Validation and Branch Protection Role

To add validation and branch protection role to prevent merging of pull requests until successful build completion, follow these steps:

1. Navigate to the repository settings and click on “Branches” tab.
2. Select the branch you want to protect and click on “Edit”.
3. Scroll down to “Branch Protection Rules” section and click on “Add rule”.
4. Add branch name “ main “ , then check the box for “Require status checks to pass before merging” and select the required check(s) from the dropdown list.
5. Check the box for “Include administrators” if you want to include them in this protection rule.
6. Click on “Create” to save the changes.

Now, whenever a pull request is created, it will not be able to merge until all required checks have passed successfully. This ensures that only high-quality code is merged into your repository’s protected branches.

Step 7: Configure the Pipeline template for the webhook

Webhook configuration allows us to trigger the entire pipeline when there is any push event occurs within configured branch name.

To do this, we added the below block in the pipeline.yaml file ( edit the SecreteToken with your github key name stored in the Secret Manager )

  CodePipelineWebhook:
    Type: 'AWS::CodePipeline::Webhook'
    Properties:
      Authentication: GITHUB_HMAC
      AuthenticationConfiguration:
        SecretToken: '{{resolve:secretsmanager:YOUR_GITHUB_TOKEN_NAME:
        SecretString:YUOR_GITHUB_KEY}}'
      RegisterWithThirdParty: 'true'
      Filters:
        - JsonPath: $.ref
          MatchEquals: 'refs/heads/{Branch}'
        - JsonPath: $.repository.name
          MatchEquals: '{Repo}'
      TargetPipeline: !Ref CodePipeline
      TargetAction: Source
      TargetPipelineVersion: !GetAtt 
        - CodePipeline
        - Version

Next, update cloudformation template in the console and execute the new template.yaml file, then CodePipeline service should automatically pick up these changes based on configured webhook URL; then orchestrate the entire build & deployment process automatically via pre-defined steps within the defined cloudformation template.

Step 8: Test Your Pipeline

With everything configured correctly now, it’s time to test our newly created CI/CD pipeline by pushing some changes into our Github repository;

Before that, checkout to develop branch.

> git checkout develop

1. Open the pipeline.yaml file within cloned repository folder locally

2. Change content by adding some comments

3. Commit changes with a message like “Update pipeline template content”

4. Push changes into develop branch

5. Pull the request in GitHub and merge the changes from develop to the main branch.

If everything goes well without any errors during these steps then congratulations! You’ve successfully built & deployed an application with a fully automated CI/CD pipeline using Github & AWS services together!

CodePipeline Stages

If you found this article interesting and helpful, please show your appreciation by giving it a clap!

If you faced a problem while doing this, re-fill your coffee and retry again :)

Photo by Tyler Nix on Unsplash

Links to previous stories:

CORS Error in API Gateway: 5 Points When Troubleshooting

How to Setup SSH with GitHub Repo?

Setup Jenkins With AWS EC2 Instance

Linux Networking Commands You Should Know!