10 min read Dec 16, 2021 Creating CDK Pipelines with Github Source and Codestar Connection
Binu Sebastian

10 min read

2021-12-16

What is CDK anyway?

AWS CDK (Cloud Development Kit) is an open-source framework which gives great depth to the concept of Infrastructure as Code. It’s made up of individual building blocks called constructs which are modules that allow us to configure and use various AWS services within a CDK project. It supports many popular programming languages like Python, Typescript, Java, etc. and for that matter, can be seen and maintained as any other code repository. This to a great extent helps blur the lines between application code and the infrastructure configuration that it requires to function.

So Why CDK Pipelines?

One of the key components in any application development process is a deployment pipeline that helps push code changes from a repository to the infrastructure where the application is deployed. Since CDK allows for the infrastructure, including said pipeline, to be part of the code itself, we are introduced to a grey area wherein the pipeline that is intended to deploy the code is also part of the same code.

CDK Pipelines is CDK’s way of handling this. While using the AWS CodePipeline under the hood to handle deployments, it brings to the table a feature called self mutation or self updation. This means that whenever changes are pushed from a CDK project configured with CDK Pipelines, it first checks for any changes made to the pipeline itself. If there are no changes to the pipeline, it goes ahead and deploys the actual infrastructure stack. However, if any changes have indeed been made to the pipeline, it will first deploy the changes to itself before triggering the deployment of the rest of the infrastructure stack. This means, once up and running, everything, including the pipeline that does the deployment itself, can be modified and updated by just pushing the code changes to the repository which triggers the pipeline.

In this post, we will go through the steps required to setup a simple Typescript CDK project with CDK Pipelines and see how we could use a Github repo as the source. We will also look at how an AWS CodeStar connection can be used to authorize the deployment of code from said repository.


Pre-requisites

  • AWS CLI to be installed and configured
  • AWS CDK to be installed
  • Nodejs

Setup CodeStar Connection

Even though, the CodePipelineSource construct, which we would use down the line to configure the source repository for the pipeline, contains a Github method, it requires a Github personal access token to be generated and stored in AWS Secrets Manager, which would then be used to authorize the pipeline with the repository. This could likely lead to issues when contributors to the repository change or when permissions are altered. On the flip side we have the CodeStar connection, which doesn’t require creating personal Github access tokens and uses a Github application called AWS Connector for Github. We will be using this approach.

  • Open the AWS CodePipeline console and navigate to the connections page from the settings dropdown in the left hand menu. Click Create Connection.

connections-dropdown.png

  • Select the Github provider, give the connection a name, and hit Connect to Github.

create-connection.png

  • On the next page, choose to install a new app.

install-github-app.png

  • Sign in with Github, select the repository the connector is to be used for, and click Install.

select-repository.png

  • Once back on the AWS console, make sure the newly installed app is selected and click Connect.

use-github-application.png

  • With that, we have successfully setup our Codestar connection. Make a note of the ARN of the connection, which we will later use to set up the pipeline.

connection-details.png


Now Let’s Get Our CDK Project Set up

  • Create a directory for the project, and run the following command to initialize your CDK Typescript project within that directory.

    cdk init --language typescript
  • Install the required constructs for our project. The @aws-cdk/pipelines construct will be used to set up the CDK Pipeline.

    npm i @aws-cdk/aws-apigateway @aws-cdk/aws-lambda @aws-cdk/pipelines
  • Open cdk.json in the project’s root directory and add the following element to the context property. This will allow the @aws-cdk/pipelines construct to use some new core features of the CDK framework called new style stack synthesis, which are required for our usecase.

    "@aws-cdk/core:newStyleStackSynthesis": true

newstylesynthesis.png

  • Now we use the cdk bootstrap command with the --cloudformation-execution-policies flag in order to provide the required permissions to deploy the pipeline to your AWS account. Make sure you have the AWS CLI installed and configured with the right AWS account to be used.
    cdk bootstrap --cloudformation-execution-policies arn:aws:iam::aws:policy/AdministratorAccess

Creating the Pipeline Stack

  • Create a new file in the /lib directory called pipeline-stack.ts. Here we use the CodePipelineSource.connection() method to specify the CodeStar connection we created.

    import * as cdk from "@aws-cdk/core";
    import {
    	CodeBuildStep,
    	CodePipeline,
    	CodePipelineSource
    } from "@aws-cdk/pipelines";
    
    export class MyPipelineStack extends cdk.Stack {
    	constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    		super(scope, id, props);
    
    		const pipeline = new CodePipeline(this, "BlogPipeline", {
    			pipelineName: "BlogPipeline",
    			synth: new CodeBuildStep("SynthStep", {
    				input: CodePipelineSource.connection(
    					"<Repo owner>/<Repo name>",
    					"main",
    					{
    						connectionArn:
    							"ARN OF THE CODESTAR CONNECTION MADE EARLIER"
    					}
    				),
    				installCommands: ["npm install -g aws-cdk"],
    				commands: ["npm ci", "npm run build", "npx cdk synth"]
    			})
    		});
    
    	}
    }
    
  • The next step is to make sure that this pipeline-stack will serve as the entry point for our CDK project and the resources defined within. So we update the /bin/<name-of-your-project>.ts file.

    #!/usr/bin/env node
    import "source-map-support/register";
    import * as cdk from "@aws-cdk/core";
    
    //Import our newly created pipeline stack
    import { MyPipelineStack } from "../lib/pipeline-stack";
    
    const app = new cdk.App();
    
    //Instantiate the pipeline stack
    new MyPipelineStack(app, "MyPipelineStack");
    
  • Now that we have updated the entry point, we need a way to connect our resource stack(s) (which will contain our actual AWS resources to be deployed) to the pipeline stack. For this, we will create a new CDK Stage and add it to the pipeline stack. So create a new file in /lib called pipeline-stage.ts.

    import { Stage, Construct, StageProps } from "@aws-cdk/core";
    
    export class BlogPipelineStage extends Stage {
    	constructor(scope: Construct, id: string, props?: StageProps) {
    		super(scope, id, props);
    
    	}
    }
    
  • Once the stage is created, update the /lib/pipeline-stack.ts and add the stage to the pipeline.

    import * as cdk from "@aws-cdk/core";
    import {
    	CodeBuildStep,
    	CodePipeline,
    	CodePipelineSource
    } from "@aws-cdk/pipelines";
    
    //***********Import the pipeline stage***********
    import { BlogPipelineStage } from "./pipeline-stage";
    
    export class MyPipelineStack extends cdk.Stack {
    	constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    		super(scope, id, props);
    
    		const pipeline = new CodePipeline(this, "BlogPipeline", {
    			pipelineName: "BlogPipeline",
    			synth: new CodeBuildStep("SynthStep", {
    				input: CodePipelineSource.connection(
    					"<Repo owner>/<Repo name>",
    					"main",
    					{
    						connectionArn:
    							"ARN OF THE CODESTAR CONNECTION MADE EARLIER"
    					}
    				),
    				installCommands: ["npm install -g aws-cdk"],
    				commands: ["npm ci", "npm run build", "npx cdk synth"]
    			})
    		});
    
        //***********Instantiate the stage and add it to the pipeline***********
    		const deploy = new BlogPipelineStage(this, "Deploy");
    		pipeline.addStage(deploy);
    	}
    }
    
  • Within our /lib directory, we will find that we have a default stack, which is where we will be defining our AWS resources. And the job of connecting this, and any other stacks we might have, to the pipeline stack is done by the pipeline stage. All we need to do is, instantiate our resource stack(s) within the pipeline stage.

    import { Stage, Construct, StageProps } from "@aws-cdk/core";
    
    //***********Import the resource stack***********
    import { CdkPipelinesBlogStack } from "./blog-resources-stack";
    
    export class BlogPipelineStage extends Stage {
    	constructor(scope: Construct, id: string, props?: StageProps) {
    		super(scope, id, props);
    
    		//***********Instantiate the resource stack***********
    		new CdkPipelinesBlogStack(this, `CdkPipelinesBlogStack`);
    	}
    }
    
  • With that, our pipeline is ready. Next, commit and push the code to the main branch of the Github repo selected while setting up the CodeStar connection.

Deploying the Pipeline

For the very first deployment, we are required to directly deploy the stack using the CDK CLI. This will setup the pipeline for us and deploy the code pushed into the github repo.

  • Build the project

    npm run build
  • Synthesise and deploy

    cdk synth && cdk deploy

Let’s move over to the AWS CodePipeline console to verify that the pipeline has been successfully deployed.

pipeline-successful.png


Creating the Resources Stack

Now that our pipeline is up and running, let’s start pushing some changes to see how it handles. We will setup a simple application with an API Gateway and a Lambda function

  • Create a directory called lambda in the root of our project and add a new file called index.js.

    exports.handler = async (event) => {
    	return {
    		statusCode: 200,
    		headers: { "Content-Type": "text/plain" },
    		body: `Congrats!!! You have reached the really useful API`
    	};
    };
    
  • Update the default resource stack /lib/<name-of-your-project>-stack.ts.

    import * as cdk from "@aws-cdk/core";
    
    //Import the lambda and apigateway constructs
    import * as lambda from "@aws-cdk/aws-lambda";
    import * as apigw from "@aws-cdk/aws-apigateway";
    
    export class BlogResourcesStack extends cdk.Stack {
    	constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    		super(scope, id, props);
    
    		const blogLambda = new lambda.Function(this, `BlogLambdaHandler`, {
    			runtime: lambda.Runtime.NODEJS_14_X,
    			code: lambda.Code.fromAsset("lambda"),
    			handler: "index.handler"
    		});
    
    		new apigw.LambdaRestApi(this, `BlogEndpoint`, {
    			handler: blogLambda
    		});
    	}
    }
    
  • Now, let’s make some changes to the pipeline itself to see if the self-mutating nature of the pipeline kicks in. Update the name of the pipeline in /lib/pipeline-stack.ts from BlogPipeline to BlogPipelineChanged.

    import * as cdk from "@aws-cdk/core";
    import {
    	CodeBuildStep,
    	CodePipeline,
    	CodePipelineSource
    } from "@aws-cdk/pipelines";
    
    import { BlogPipelineStage } from "./pipeline-stage";
    
    export class MyPipelineStack extends cdk.Stack {
    	constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    		super(scope, id, props);
    
    		const pipeline = new CodePipeline(this, "BlogPipeline", {
    
        //***********Change the name of the pipeline***********
      		pipelineName: "BlogPipelineChanged",
        
    
    			synth: new CodeBuildStep("SynthStep", {
    				input: CodePipelineSource.connection(
    					"<Repo owner>/<Repo name>",
    					"main",
    					{
    						connectionArn:
    							"ARN OF THE CODESTAR CONNECTION MADE EARLIER"
    					}
    				),
    				installCommands: ["npm install -g aws-cdk"],
    				commands: ["npm ci", "npm run build", "npx cdk synth"]
    			})
    		});
    
    		const deploy = new BlogPipelineStage(this, "Deploy");
    		pipeline.addStage(deploy);
    	}
    }
  • Make sure to add !**/lambda/*.js to .gitignore to prevent the .js files in the lambda directory from being ignored

    *.js
    !jest.config.js
    
    #To prevent functions in our lambda directory from being ignored 
    !**/lambda/*.js
    
    *.d.ts
    node_modules
    
    # CDK asset staging directory
    .cdk.staging
    cdk.out
    

Let’s See if it Works

  • Commit and push the code to the main branch of the Github repo. This will trigger a deployment by BlogPipeline.

old-pipiline-inprogress.png

  • Since there was a change to the pipeline itself, self-mutation will be triggered.

old-pipeline-self-mutate.png

  • Once the self-mutation is complete, BlogPipelineChanged will deploy the resource stack, which contains an API Gateway backed by a Lambda.

new-pipeline-inprogress.png

new-pipiline-deployed.png

  • Now let’s test our application.

api-gateway-resource.png

working-app.png

The code used in this post can be found in this repo

References

For questions and suggestions, feel free to reach out to us on Twitter

Further Reading

Up and running with Azure Functions

Jan 25, 2021

A step by step walkthrough of how to get up and running with Azure serverless functions

6 min read

6 min read

Jan 25, 2021

This website stores cookies on your computer.

These cookies are used to collect information about how you interact with this website and allow us to remember you. We use this information in order to improve and customize your browsing experience and for analytics and metrics about our visitors on this website.

If you decline, your information won’t be tracked when you visit this website. A single cookie will be used in your browser to remember your preference not to be tracked.