A Lambda bot that responds to GitHub pushes and deploys to S3

This walkthrough automates deployment to an S3 website triggered by pushes to a GitHub repository. The following steps will create the necessary S3 bucket, GitHub webhook, and Lambda application. This article assumes that you already have a GitHub account and repo, that you are proficient in using git, branches, and pushing to a remote repository, that you have AWS-CLI installed and configured, and that you have the Serverless.js framework installed and configured for your AWS account(s).

Step 1: Create Two S3 buckets and configure for web hosting

If you already have S3 buckets configured for web hosting you can skip to Step 2. However I would recommend testing this on a non-production website first. I recommend creating a primary bucket for pushes to the 'master' branch and a second dev bucket for pushes to the 'dev' branch.  The steps below are only the very basic necessary to create an S3 bucket for web hosting, there are far more configuration options available at: http://docs.aws.amazon.com/gettingstarted/latest/swh/website-hosting-intro.html

NOTE:: it should go without saying, but please replace www.mysite.com below with your domain unless you actually own mysite.com. 

1. Go to the Amazon S3 console.
2. Click on [Create Bucket].
3. Use your domain name as the bucket name. eg: www.mysite.com
4. Click on [Create].
5. Select the newly created bucket and click on [Properties].
6. Click on Static Website Hosting and select Use this bucket to host a website.
7. Enter the name of your index document and error document and click [Save]. eg: index.html and 404.html
8. Click on Permissions and then [Bucket policy].
9. Enter the following bucket policy and click [Save].

Step 2: Create deploy.json file and assign deploy targets

The GitHubtoS3 lambda bot will search your repo for a deploy.json file in order to know how and where to deploy your website. This file specifies the type of deployment “S3”, as well as the bucket names for pushes to the “master” and “dev” branches. Again, please replace www.mysite.com and dev.mysite.com below with your actual bucket names.

1. In the root of your repository, create a text file named ‘deploy.json’.
2. Enter the following JSON example into the file and save it.

3. Make sure this file is added to the git repo and committed.

Step 3: Create GitHub access token and Webhook Secret

You can create access tokens for your own account, but then everything the Lambda function does will appear as if you yourself did it. This may be fine for a repo that only you contribute to, but for a larger team it makes sense to create a new GitHub account for your bot to authenticate as.

1. Log in to GitHub
2. Browse to https://github.com/settings/tokens
   IMPORTANT: This is your only opportunity to copy the string.
3. Click on [Generate new token].
4. Enter “GitHubtoS3” as the “Token description”.
5. Check the boxes next to “repo” and “notifications”.
6. Click on [Generate Token].

Step 4: Clone the GitHubtoS3 repo and configure

Clone or download the GitHubtoS3 repo to your local environment to configure your aws-cli profiles, personal access token, and S3 bucket names. The example config file provides examples for 2 separate AWS-CLI accounts configured but more can be added or removed as necessary. It also contains examples for 3 S3 buckets per AWS account, again more can be added or removed as needed.

1. Go to https://github.com/kyle138/githubtos3 and download the repo (or latest release).
2. Copy config/config_example.json to config/config.json.
3. Edit config/config.json and modify to match your information. It will initially appear as below:

4. Change "aws-profile1" and replace it with the name of your aws-cli profile. If you do not remember your aws-cli profile names they can be found in the ~/.aws/config file.
5. Change the value of "github_personal_access_token":  "SuperSecretPersonalAccessToken1" and replace it with the Personal Access Token you created in Step 3.
6. You haven't created your "github_webhook_secret": yet, but change the "UseTheSameWebhookSecretForAllReposDeployedToThisAWSAcct" value to a long random string. You will enter this later in Step 7.
7. Modify the "s3_targets" array with the ARN strings for the S3 buckets you created in Step 1.
8. Repeat substeps 4-7 for "aws_profile2" if you have multiple AWS-CLI profiles configured, otherwise delete that section from the JSON file.
9. Save the config/config.json file.

Step 5: Install the necessary NPM modules

The GitHubtoS3 application depends on a number of NPM modules, those aren't included within this repo and will need to be installed.

1. Change to the layers/CommonModules/nodejs directory in terminal.
   eg: cd layers/CommonModules/nodejs
2. Run npm install to download the required modules.
3. If successful you should now see a node_modules directory.
   

Step 6: Deploy GitHubtoS3

GitHubtoS3 should now be configured and ready to be deployed to your AWS account using Serverless.js. This will create the two lambda functions, the SNS topic, and the API Gateway endpoint that the GitHub webhook will use.

1. Deploy using the following command:
   sls --aws-profile aws-profile1 -s v1 deploy
   (Replace aws-profile1 with your actual AWS-CLI profile)
2. If successful the output should appear like this:
   
3. Locate the POST endpoint highlighted in the example above and copy this URL. You will need it to create your GitHub webhook in Step 7.

Step 7: Create the GitHub webhook

GitHub webhooks will trigger an external service in response to certain events. In this case "Push" events will be published to the API Gateway endpoint created in the previous step. You will also need the GitHub webhook secret you created in Step 4. If you configure multiple repos to use GitHubtoS3 use the same GitHub webhook secret for all of them. Make sure each of these repos contains a deploy.json file created in Step 2.

1. Log in to GitHub and browse to your repository.
2. Click on the “Settings” tab along the top navigation bar.
3. Click on "Webhooks" in the left menu.
4. Click on [Add webhook].
5. In the Payload URL field enter the API Gateway POST endpoint you copied in Step 6.
6. For Content type select "application/json".
7. In the Secret field enter the GitHub Webhook Secret you created in Step 4.
8. For Which events would you like to trigger this webhook select (*) Just the push event.
9. Make sure the [x] Active box is checked.
10. Click on [Add webhook].
    
11. You should now see your new webhook listed, click on it.
12. At the bottom of the page you should see Recent Deliveries. When the webhook was created GitHub sent a "ping" event to the API Gateway endpoint, if everything has been configured successfully you should see a Good  200  Response.
    

The following links provide more information for GitHub webhooks:

• https://developer.github.com/webhooks/creating/
• https://developer.github.com/webhooks/securing/
• https://developer.github.com/webhooks/testing/

Step 8: Test the Setup
That's it, everything should now be functioning. You can test it by making a change to your repo and pushing to either the master or dev branches to trigger the webhook. You can monitor the progress in the following places:

1. GitHub:

1. Browse to the Webhook - Recent Deliveries same as in Step 7.12. You should see a new delivery listed with the timestamp matching your code push. It should have a green check mark signifying a good response.

2. CloudWatch:

1. Log in to the AWS Console.
2. Browse to CloudWatch > Log groups and click on /aws/lambda/SLS-GithubToS3-v1-listener.
3. Locate the log stream with the timestamp matching your code push.
4. Browse back to CloudWatch > Log groups and click on
   /aws/lambda/SLS-GithubToS3-v1-deployer.
5. Locate the log stream with the timestamp matching your code push.

3. S3:

1. Log in to the AWS Console.
2. Browse to S3  and select the bucket you created in Step 1.

1. The contents of the bucket should match your repo.

Credits:

By no means did I come up with all of this by myself. I drew heavy inspiration (and code) from the links below::

Dynamic GitHub Actions with AWS Lambda

https://aws.amazon.com/blogs/compute/dynamic-github-actions-with-aws-lambda/

JavaScript GitHub API for Node.JS

http://mikedeboer.github.io/node-github/

S3 Static Website Hosting:

http://docs.aws.amazon.com/gettingstarted/latest/swh/website-hosting-intro.html

The changes in V3.0.x which moved the trigger for this function from GitHub's SNS service to an API Gateway webhook drew heavy inspiration from:

https://serverless.com/examples/aws-node-github-webhook-listener/