Otse Obande's Blog

Using Github Actions to run automated tests and deployment to AWS EC2 with CodeDeploy.

Using Github Actions to run automated tests and deployment to AWS EC2 with CodeDeploy.

Otse Obande's photo
Otse Obande
·

0 min read

Automation is the heart of building a modern Continuous Integration and Continuous Delivery (CI/CD) pipeline or workflow and helps saves time and effort for the development and operations team allowing them to focus on building great features. In this article we would be exploring how to setup a CI/CD pipeline using Github Actions. I chose to use Github Actions because the project repository was hosted on Github and I wanted to explore the newly added Github Actions feature. We would be using AWS CodeDeploy because the application servers are hosted on AWS EC2 instances.

Prerequisites

These are required to setup the pipeline using the steps described in this article:

  • AWS access keys with AWS CodeDeploy and AWS S3 access.
  • Two identical AWS EC2 instances with a running PM2 cluster on each.
  • Administrator access to the project's Github repository.
  • AWS CLI

Our application would be called api-server. Before we begin, we need to plan how we intend to implement this automation setup. It should be noted that the workflow used by the development team is the Gitflow workflow. What we intend to achieve is an automated pipeline that runs tests on push to every branch, deploys to the staging server after tests pass on the develop branch and deploys to the production server afters tests pass on the master branch.

CodeDeploy

We'll be using CodeDeploy for this setup so we have to create a CodeDeploy application for our project and two deployment groups for the application. One for staging and the other for production. Deployment groups are a set of individual EC2 instances or lambda functions our application would be deployed to. Creating these AWS resources can be done using the AWS Console but in this article we'll be using the CLI to create AWS resources.

To create the api-server CodeDeploy application using AWS CLI, we run this on our terminal:

aws deploy create-application \
--application-name api-server \
--compute-platform Server
Deployment Groups

Next, we create the deployment groups for our application. Before we can proceed to create a deployment group we need to create a Service Role for CodeDeploy using IAM and attach the AWSCodeDeployRole (As stated previously, this can be easily done using the AWS console but we'll be using the CLI through out this article). Before we run the cli command to create the service role, we need to create a file with IAM specifications for the role, copy the content below into it and name it CodeDeployDemo-Trust.json

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "Service": [
                    "codedeploy.amazonaws.com"
                ]
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

We can now create the role by running:

aws iam create-role \
--role-name CodeDeployServiceRole \
--assume-role-policy-document file://CodeDeployDemo-Trust.json

After the role is created we attach the AWSCodeDeployRole policy to the role

aws iam attach-role-policy \
--role-name CodeDeployServiceRole \
--policy-arn arn:aws:iam::aws:policy/service-role/AWSCodeDeployRole

To create a deployment group we would be needing the service role ARN. This can be gotten from the output of the create-role command but we can run the command below to get it again.

aws iam get-role \
--role-name CodeDeployServiceRole \
--query "Role.Arn" \
--output text

The ARN should look something like arn:aws:iam::[account-number]:role/CodeDeployServiceRole.

Phew! Creating a service role with the AWS CLI took more steps than I thought it would. Let's go on to create a deployment group for the staging environment.

aws deploy create-deployment-group \
--application-name api-server \
--deployment-group-name staging \
--service-role-arn arn:aws:iam::[account-number]:role/CodeDeployServiceRole \
--ec2-tag-filters Key=Name,Value=staging,Type=KEY_AND_VALUE

To create deployment group for the production environment we just need to duplicate the step above and change the --deployment-group-name to production and the --ec2-tag-filters to Key=Name,Value=production,Type=KEY_AND_VALUE

AppSpec File

We've created our CodeDeploy application with the necessary deployment groups. The next stage is to setup our appspec file. An AppSpec file is used by CodeDeploy to manage deployments. It can be in either JSON or YAML. We'll be using the YAML format to setup some hooks to deploy our application and validate the deployment.

Create a file named appspec.yml in the project directory and paste this content into it:

version: 0.0
os: linux
files:
  - source: .
    destination: /srv/api-server
hooks:
  AfterInstall:
    - location: scripts/reload_pm2.sh
      timeout: 300
      runas: ubuntu
  ValidateService:
    - location: scripts/validate_service.sh
      timeout: 300

Let's break down the appspec file down. The files section contains an array of objects to specify source and destination for application files. These can be modified to suit other applications and server locations. The hooks specify actions to be taken during the deployment process. For this deployment, we would be using the AfterInstall and the ValidateService hooks. The runas key specifies the server user account to run the scripts as.

The AfterInstall hook runs a script after the deployment artifacts are sent to the server. This is usually used to start the application. Our current application uses a PM2 cluster so we can perform a no downtime deployment by reloading the running process. The content of the scripts/reload_pm2.sh file looks like this.

#!/bin/bash
pm2 reload api-server

It's just a command to reload the api-server process (you would need to name the process on an initial deploy using pm2).

The ValidateService hook runs after every thing has been set up to test that the service runs fine. In this case we would use curl and grep to validate a text appears in the return value of an endpoint but you can use any form of validation you find necessary. The content of the scripts/validate_service.sh should look like this for this deployment:

#!/bin/bash
curl -v --silent localhost:3000 2>&1 | grep success

Github Actions

As we discussed, we would be using Github Actions to setup the CI/CD pipeline. First step is to create the workflow file and setup the automated tests. To create the workflow file, create a .github folder in the project repository and create a workflows sub folder in the .github folder. The workflows folder would contain yaml configurations for Github Actions. Next up, we'll create a workflow file. I'll call this one ci-cd.yml and add the test job to run tests.

name: CI/CD

on: push

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v1
    - name: Install Node.js
      uses: actions/setup-node@v1
      with:
        node-version: '10.x'
    - name: Install npm dependencies
      run: npm install
    - name: Run tests
      run: npm run test

This test job would be ran on every push to any branch of the Github repository. It uses the actions/checkout@v1 to checkout the current repository's branch and uses actions/setup-node@v1 to setup a node environment to run scripts in. The job then installs the project dependencies and runs the test script.

The next job we would configure is the staging-deploy job. To configure this job to perform actions on our AWS account we need to save our AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY as secrets on our Github repository so we can gain access to them in the job.

Screenshot 2020-02-16 at 16.20.54.png

After secrets have been updated we can go ahead to add the code for the staging-deploy job. 

# ... test job config

staging-deploy:
    runs-on: ubuntu-latest
    needs: [test]
    if: github.ref == 'refs/heads/develop'
    steps:
    - uses: actions/checkout@v1
    - name: Install Node.js
      uses: actions/setup-node@v1
      with:
        node-version: '10.x'
    - name: Install npm dependencies
      run: npm install
    - name: AWS Deploy push
      uses: ItsKarma/aws-cli@v1.70.0
      env:
        AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
        AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        AWS_DEFAULT_REGION: "us-east-2"
      with:
        args: >-
          deploy push
          --application-name api-server
          --description "This is a revision for the api-server"
          --ignore-hidden-files
          --s3-location s3://deployments/staging-api-server.zip
          --source .
    - name: AWS Create Deploy
      uses: ItsKarma/aws-cli@v1.70.0
      env:
        AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
        AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        AWS_DEFAULT_REGION: "us-east-2"
      with:
        args: >-
          deploy create-deployment
          --application-name api-server
          --deployment-config-name CodeDeployDefault.OneAtATime
          --deployment-group-name staging
          --file-exists-behavior OVERWRITE
          --s3-location bucket=deployments,key=staging-api-server.zip,bundleType=zip

For this deployment job we need it to wait for the test job to pass before proceeding to run the deployment so we add needs: [test]. We also need it to be ran only on push to the develop branch so we use an if clause to implement the check. This job steps starts out with steps similar to that of the test job because we still need to checkout the repository code and install dependencies. Things get a little different on the AWS Deploy push step. In this step we need to use the AWS CLI to push the application code as a deployment artifact to an S3 bucket. To ease the step of setting up a profile for aws cli, I used ItsKarma/aws-cli@v1.70.0 and used the AWS credentials saved as secrets as environment variables. The AWS Create Deploy step creates a deployment to the staging deployment group.

To setup the production-deploy job we would duplicate the staging-deploy job and modify the deployment group to production and change the S3 bucket. The complete ci-cd.yaml file should look like this:

name: CI/CD

on: push

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v1
    - name: Install Node.js
      uses: actions/setup-node@v1
      with:
        node-version: '10.x'
    - name: Install npm dependencies
      run: npm install
    - name: Run tests
      run: npm run test

  staging-deploy:
    runs-on: ubuntu-latest
    needs: [test]
    if: github.ref == 'refs/heads/develop'
    steps:
    - uses: actions/checkout@v1
    - name: Install Node.js
      uses: actions/setup-node@v1
      with:
        node-version: '10.x'
    - name: Install npm dependencies
      run: npm install
    - name: AWS Deploy push
      uses: ItsKarma/aws-cli@v1.70.0
      env:
        AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
        AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        AWS_DEFAULT_REGION: "us-east-2"
      with:
        args: >-
          deploy push
          --application-name api-server
          --description "This is a revision for the api-server"
          --ignore-hidden-files
          --s3-location s3://deployments/staging-api-server.zip
          --source .
    - name: AWS Create Deploy
      uses: ItsKarma/aws-cli@v1.70.0
      env:
        AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
        AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        AWS_DEFAULT_REGION: "us-east-2"
      with:
        args: >-
          deploy create-deployment
          --application-name api-server
          --deployment-config-name CodeDeployDefault.OneAtATime
          --deployment-group-name staging
          --file-exists-behavior OVERWRITE
          --s3-location bucket=deployments,key=staging-api-server.zip,bundleType=zip

  production-deploy:
    runs-on: ubuntu-latest
    needs: [test]
    if: github.ref == 'refs/heads/master'
    steps:
    - uses: actions/checkout@v1
    - name: Install Node.js
      uses: actions/setup-node@v1
      with:
        node-version: '10.x'
    - name: Install npm dependencies
      run: npm install
    - name: AWS Deploy push
      uses: ItsKarma/aws-cli@v1.70.0
      env:
        AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
        AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        AWS_DEFAULT_REGION: "us-east-2"
      with:
        args: >-
          deploy push
          --application-name api-server
          --description "This is a revision for the api-server"
          --ignore-hidden-files
          --s3-location s3://deployments/production-api-server.zip
          --source .
    - name: AWS Create Deploy
      uses: ItsKarma/aws-cli@v1.70.0
      env:
        AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
        AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        AWS_DEFAULT_REGION: "us-east-2"
      with:
        args: >-
          deploy create-deployment
          --application-name api-server
          --deployment-config-name CodeDeployDefault.OneAtATime
          --deployment-group-name production
          --file-exists-behavior OVERWRITE
          --s3-location bucket=deployments,key=production-api-server.zip,bundleType=zip

With these in place, our Github Action workflow is ready to deploy application changes to our servers automatically. 😁

Screenshot 2020-02-16 at 22.41.40.png

References

AWSec2CI/CD