Using Fastly with Terraform, Automation, and CICD

17 min readMar 23, 2021

Building a continuous edge delivery pipeline for any organization, small or large.

Before we begin

Full Disclosure: This article is not sponsored (or promoted) by Fastly, Terraform, GitLab or any other organization. The sole purpose of this article is support the opensource community. 🖥️

Intro 💡

Over the past year, many organizations have gone through a transitional phase to adopt faster, smarter, and more developer friendly technologies. For many companies, this involved migrating their entire CDN (Content Delivery Network) to Fastly. For these organizations it can also be the prime opportunity to adopt Infrastructure as Code (IaC) methodologies and a robust pipeline for Continuous Delivery.

Key Terms:

CDN: A “Content Delivery Network” (CDN) is a geographically distributed network of servers to deliver web content to users. Think images, html, JavaScript, and API responses. Fastly is the CDN used in this article.
IaC: “Infrastructure as Code” is the process of managing and provisioning infrastructure through machine-readable definition files, rather than physical configuration or interactive configuration tools. We will use Terraform as our tool for IaC in this article.
CI/CD: “Continuous Integration and Continuous Delivery” is an engineering principal for the building, testing and deployment of applications. We will be using GitLab CI in this article.

The Open Source Fastly-Framework

The entire framework for this article and project can be found on GitHub. The source code also contains a lot of docs pages and in-line documentation for usage. Full Link: https://github.com/GrantBirki/fastly-framework

Benefits

There are many benefits to using these three technologies together — here are just a few:

Using Git as a version control system for all Fastly changes
Eliminate code reuse through shared VCL files, Snippets, and Terraform configuration blocks
Test your services through a CICD pipeline before deploying them
Integrate with ChatOps for deployments (Example: Slack)
Quickly create new services from templates with make service - Using Jinja and Python
Adopt Infrastructure as Code methodologies with Terraform
Promote a peer-review culture through merge/pull requests
Create your own pipeline stages for robust testing, alerts, approval, and much more

Let’s dive in!

Prerequisites 📝

Here are the prerequisites you will need to follow along with this article:

A Fastly account — Free!
A GitLab account — Free!
An AWS account if you wish to use Terraform Remote State — Free tier eligible
You own domain — Replace all occurrences of example.com in this guide and the framework repo with your domain.

Fastly ⏰

I may be a little biased and have only had the opportunity to work on a couple of CDNs but I must say, Fastly is awesome. Don’t just take it from me, here are other companies that use Fastly:

GitHub, Imgur, Reddit, Stripe, New Relic, The New York Times, Kickstarter, Yelp, Shopify, BuzzFeed, Kayak, USA Today, The Guardian, and many more

As the name states, Fastly is fast, especially when it comes to deployment times. When you make a change to a service in Fastly, your changes are deployed globally in under 60 seconds. Other CDNs that are out there have ~10 minute deployment times. With Fastly, you are now able to build, test, deploy, and validate a service before other CDNs can even activate a service!

Not only is Fastly fast but it is also developer centric. This means that everything you can do in the Fastly console, you can also do via the Fastly API or with Terraform.

Let’s break down Fastly for understanding and then explain how we can leverage Terraform to build Fastly services:

Fastly is a CDN. This means there are servers all over the world serving requests for Fastly Services.
Fastly serves requests based on domains. We give Fastly a domain and it listens for requests to this domain: www.example.com
Fastly fetches content from backends. We provide Fastly with a backend (ex: S3 bucket with images) and Fastly will serve content from these backends to clients.
Fastly uses VCL. Varnish Configuration Language is the code we write to fine tune how Fastly responds, caches, and processes requests to our domains and backends.

Example of a Fastly Service

A Fastly service that listens for incoming requests all around the world to www.example.com.

A request comes in to www.example.com/cookie.jpg and Fastly begins processing.
Fastly executes the service’s VCL code which we uploaded.
The VCL code states that all requests with .jpg file extensions should go to a static S3 bucket to get assets.
Fastly checks its cache for this image before requesting it from the backend. Fastly determines the image is not in it’s cache.
Fastly requests cookie.jpg from the S3 backend.
Fastly responds to the client with cookie.jpg
www.example.com/cookie.jpg renders on the client’s browser.

To accomplish the example above, we need to build a Fastly service, define the domains to listen on, backends to fetch data from, and VCL to process requests. Luckily with Terraform, we can define all this as code!

Now let’s check out how you can get started writing some Infrastructure as Code and build a Fastly service and configure these components with Terraform!

Fastly Service with Terraform ⚙️

The snippet below shows how you can make a simple Fastly service with Terraform:

resource "fastly_service_v1" "fastly-service" {
  name            = "www.example.com"
  activate        = false
  version_comment = "Hello World" domain {
  name    = "www.example.com"
  comment = "Example Domain"
  }backend { 
  name          = "S3_Example"
  address       = "example.s3-website-us-west-2.amazonaws.com"
  override_host = "example.s3-website-us-west-2.amazonaws.com"
  port          = 80
  }vcl {
  name    = "main"
  content = file("fastly.vcl")
  main    = true
}}

This example would build a basic Fastly service that serves requests to www.example.com from a S3 website backend.

You will need to replace all occurrences of example.com with your own domain

Note: you will need to create the fastly.vcl file listed above and place it into the same directory as you are running your Terraform commands. The fastly.vcl file needs to contain the Fastly boilerplate as a starting point. Simply paste the boilerplate into your fastly.vcl file. For reference you may view the example service folder for using Fastly + Terraform + the adapted VCL Boilerplate in the framework here.

Building a Fastly Service

Once you have Terraform installed, a valid fastly.tf file, and a fastly.vcl file (with the boilerplate) you are ready to build your service!

cd into the same directory as your files listed above
Get a Fastly API key from your account page and set it as an environment variable like so: export FASTLY_API_KEY="<your_key_here>"
Run terraform init
Run terraform plan
Run terraform apply

Check the Fastly console to see your new service!

Note — If you are having any difficulties with this step please refer to the following documents as guides:

Building a CI/CD Pipeline 🔨

So far we have seen the core components of a Fastly service and how we can create one with Terraform by hand. However, in the real world we do not want a bunch of engineers making changes by hand, with no version control, and without a process to make changes uniformly. This is where pipelines come into play.

Intro

A CI/CD pipeline is a series of steps that must be performed in order to deliver a new version of a service. They are repeatable, automated, and reliable ways to release and deploy code.

There are several big players in the space of CI/CD:

This project is using GitLab-CI for the CI/CD pipeline. However, you can use any CI/CD platform you like and follow the Fastly-Framework as a guide for what you can create.

Create Your Repository

The first step to creating a CI/CD pipeline is to create a Git Repo where our code and configuration will live. Since we are using GitLab in this article we can create a repo there.

If you haven’t cloned the Fastly-Framework yet, please do so now:
git clone https://github.com/GrantBirki/fastly-framework.git

Create a new repo in GitLab

Clone your new repo locally:
git clone https://gitlab.example.com/<username>/fastly.git
Copy the Fastly-Framework contents into your new GitLab repo locally:
cp -r fastly-framework/. fastly/
Create a new branch:
git checkout -b "initial-fastly-build"
Add, Stage, and Commit all files:
git add -A && git commit -m "Initial Fastly Repo Commit"
Push your changes into GitLab:
git push --set-upstream origin initial-fastly-build
Check back into GitLab. You should see your branch and have the option to create a Merge Request now:

View the Merge Request and the CI/CD pipeline which was automatically created:

You will notice that the pipeline starts to run right away in the Merge Request above. However, it fails on the first stage. This is expected as we have not yet configured the pipeline… yet.

Configuring Pipeline Stages

The first step to getting our pipeline to actually run successfully is to configure pipeline stages.

For this section we will referencing the .gitlab-ci.yml file frequently. It can be found in the Fastly-Framework here.

Edit the .gitlab-ci.yml file to configure the CI/CD Stages you wish to use.

stages:  
  - repo-check 🗺️
  - plan 📝
  - test 🧪
  # - metrics build-and-push 📊 (optional)
  # - approval 📯 (optional)
  - apply ⚙️
  - deploy 🚀
  - rapid-rollback 🔄
  # - metrics deploy 📊 (optional)

All the stages listed as (optional) above may be removed:

approval 📯 is used for ServiceNow automated change requests. More info can be found here.

metrics build-and-push 📊 and metrics deploy 📊 are used for aggregated metrics collection and publishing to New Relic. More info can be found here.

For the sake of simplicity, this would leave us with the following stages once the optional ones are removed:

stages:  
  - repo-check 🗺️ #(runs on merge_requests)
  - plan 📝 #(runs on merge_requests)
  - test 🧪 #(runs on merge_requests)
  - apply ⚙️ #(runs on master branch)
  - deploy 🚀 #(runs on master branch)
  - rapid-rollback 🔄 #(runs on master branch)

To see what each stage does, please see the Pipeline documentation in the Fastly-Framework. As a helpful tip, you can see where each stage will run above: merge_requests or merges to the master branch

Note: If you delete the optional stages above please also delete their related references later on in the .gitlab-ci.yml file. For example, if you are not using the approval stage, make sure to delete the block below (it should already be commented out anyways):

Build our Default Pipeline Image

Now that we have our initial stages setup, we can begin configuring the basics of our pipeline. First off, we will need a default image for our pipeline. This image will need the following dependencies:

Terraform
Python
AWS CLI (If using a remote S3 backend for Terraform) — suggested
CURL

To create this image, you can view the code/ci/docker folder of the Fastly-Framework for instructions. This folder also contains a Dockerfile to easily build the image.

Once you have built the image, you will need to push it up to GitLab’s container registry so the pipeline can easily access it.

Run the following commands from the code/ci/docker folder:

docker login registry.gitlab.com
docker build -t registry.gitlab.com/<account>/<repo>/<image>:<tag> .
docker push registry.gitlab.com/<account>/<repo>/<image>:<tag>

The exact <path>/<repo>/<image>:<tag> to your image my differ depending on what you name it and your registry’s org structure. No matter what you name it you just need to ensure that the default: image: line in your .gitlab-ci.yml points to this image.

stages:
  ...
  ..
  .default:
  image:
    <GitLab URL>/<repo>/<image>:<tag>

Now our pipeline will be able to access this image and it will use it as the default for all jobs and stages unless another image is specified.

Configure Pipeline Variables + Terraform State

The pipeline needs two types of authentication in order to run. It needs to be able to authenticate to Fastly via an API key to deploy changes, and it needs to be able to authenticate to a remote backend like AWS for Terraform state.

Both of these variables/credentials can be configured via the GitLab console. Steps for doing so in the GitLab console can be found here.

For Fastly, this authentication is very straightforward. Simply follow these steps to create an API token with Fastly.

Add the API token for Fastly to GitLab CI variables as a key:value
pair:
FASTLY_API_KEY: <value>

For Terraform Remote State authentication, this can be done in a variety of ways and you will need to configure this on your own. A common workflow is to use AWS as a remote state for Terraform with S3 and DynamoDB. There are many ways to authenticate to AWS and a simple/common one is to use static IAM user credentials (there are safer methods but this is just an example). You could add your AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY as GitLab CI variables. Then when plan 📝 , test 🧪 , and apply ⚙️ stages run, credentials are automatically set as they are present as environment variables. To see how this works you can checkout AWS documentation here.

If you go with this method above, it should just work. If you do any method requiring AWS tokens, or use a service like Vault you will need to edit the following files (below) and add your custom logic to get necessary remote state credentials.

code/ci/plan/plan.sh
code/ci/test/test.sh
code/ci/apply/apply.sh

It is highly suggested to use a Terraform Remote State. This framework uses the assumption that you have Terraform remote backend configured using AWS S3 and DynamoDB. To set this up, please reference the following guide.

If you do use this method, you will need to make a few additional edits:

Reminder: Make sure to be replacing example.com with your own domain in all occurrences.

Enter the services/ folder
Enter both folders www.example.com and nonprod.example.com and make the same following edits to the config.tf file

terraform {
  backend "s3" {
    bucket         = "example-terraform-state-bucket" # set to your own S3 bucket name
    key            = "fastly/services/www.example.com/terraform.tfstate" # change www.example.com
    region         = "us-west-2" # put your desired region here
    dynamodb_table = "terraform-lock"
    encrypt        = true
  }
}
provider "aws" {
  region = "us-west-2" # put your desired region here
}

Make very similar edits to the code/ci/test/test.tf file

terraform {
  backend "s3" {
    bucket         = "example-terraform-state-bucket" # set to your own S3 bucket name
    key            = "fastly/services/test-servicename/terraform.tfstate" #ID0001 - #Do NOT change this line
    region         = "us-west-2" # put your desired region here
    dynamodb_table = "terraform-lock"
    encrypt        = true
  }
}

provider "aws" {
  region = "us-west-2" # put your desired region here
}

The edits you make to these files will be directly related to how you setup your Terraform Remote State in AWS.

All three files you just made edits to should have the same bucket and region . Each file will have its own key as that will be the unique path to your state file for each Fastly service you are building with Terraform. The only oddball is the test.tf file. This is because the test 🧪 stage of the pipeline works a little differently… It works by creating an ephemeral Fastly service. This is so that you can validate the VCL you are uploading before you actually make changes to your own service to avoid tainting your Terraform state. It also allows you to write custom tests against this ephemeral service. The test 🧪 stage essentially just creates your service with a unique “test” name and then instantly delete it (unless you write custom tests in before deletion). The ephemeral test service name will look something like this in Fastly for its short existence: $CI_COMMIT_SHORT_SHA-TEST<domain> . This is set through the test.sh file, the bash sed command against the test.tf file, and the domain block name = “${var.FastlyEnv}<domain>" in each services/<service> folder.

Whew! We just covered a lot there… Let’s summarize the Variables + State section:

Set an environment variable named FASTLY_API_KEY with your Fastly API key through the GitLab UI.
Setup credentials that the pipeline can access and authenticate with for your Terraform Remote State (Ex: AWS access/secret keys).
Edit each fastly.tf file for each service in your services/ folder to point to your Terraform Remote State locations.
Edit the code/ci/test/test.tf file in a very similar manner to the previous step. Follow the #comments in the file.

Trigger and Run the Pipeline

Now that we have configured our .gitlab-ci.yml file for our pipeline, the next step is to trigger our pipeline and test it to ensure all the pieces work!

You should still have the same Merge Request/Pull Request open from when we first pushed our code up to GitLab/GitHub. If you don’t, follow the steps above to create another MR. If your MR is still open, let’s make a new commit with our changes and push it up!

On every commit to our open Merge Request, GitLab will re-run all jobs that reference the merge_requests requirements. Example:

only:
  refs:
    - merge_requests

However, it will only run merge_requests jobs if all other criteria is met. If you take a look at the .gitlab-ci.yml file, you will notice that we try to build two Fastly services: www.example.com and nonprod.example.com . Taking a look at the plan stage for www.example.com in our yml file we can see the following job defined:

plan:www.example.com:
  stage: plan 📝
  script:
    - sh code/ci/plan/plan.sh
  only:
    refs:
      - merge_requests
    changes:
      - services/www.example.com/*
      - code/logs/log_format.json
      - code/snippets/*
      - code/terraform/*
      - code/vcl/*
  artifacts:
    untracked: false
    expire_in: 1 days
    when: always
    paths:
        - "services/*/*plan*"

Lets break down what this job is doing:

Running a job called plan:www.example.com
The job is attached to the plan 📝 stage
The job will execute the code/ci/plan/plan.sh script
The job will only run on merge_requests
The job will only run if changes are made in any of the following locations: services/www.example.com/* , code/logs/log_format.json , code/snippets/* , code/terraform/* , code/vcl/*
The job will produce an artifact and save it for 1 day. Note: The artifact that is being saved is the plan file that is created after running terraform plan

Now that we know the criteria for triggering this job lets push up another commit to our merge_request . The only change we need to make is add a newline or perhaps a #comment to any file services/www.example.com/* . This will make the pipeline think that a “change” has occurred in our service and trigger related pipeline jobs. For this example, I will add a single newline to both services/www.example.com/fastly.tf and services/nonprod.example.com/fastly.tf .

This will trigger the pipeline to build or push new versions of these services to Fastly. Lets check our pipeline status in GitLab:

Our merge_request pipeline has passed! 🎉

Let’s merge our change now to the master or main branch and trigger our deployment pipeline!

After clicking merge in the GitLab UI we can see our deployment pipeline is immediately kicked off:

Now if we check our deployment pipeline we will see that the apply ⚙️ stage has kicked off right away:

Remember, you can check out the pipeline.md docs to get more info on a pipeline stage.

Apply ⚙️ — The apply phase pushes up an inactive service which you can review in the console. This is useful for a final review before deploying to production.

Let’s view the Fastly service which the apply ⚙️ stage has created for us in Fastly:

It is always good practice to take a look at the service and its associated version in Fastly before triggering the manual Deploy 🚀 stage. In Fastly, this can easily be done by clicking on “Diff versions” in the UI. An example of how to do this can be seen below:

Note: Since we are pushing up our first ever service with this pipeline it will be Version 1 and we will not be able to run a “Diff” on the service. Keep this in mind though when running your next pipeline.

Now let’s move onto the deploy stage…

Deploy 🚀 — This phase activates the service via an API call to Fastly.

Click on the job for the service you want to deploy. You can also click the top “play” button to deploy all services at once (risky — You should always run your nonprod services first).

Once our nonprod service is deployed and it looks good we can deploy prod (www.example.com):

Successful GitLab CI Deployment Pipeline

Yahoo! Our deploy pipeline has successfully passed and our Fastly services are activated! 🎉

Note: The Rapid-rollback 🔄 stage at the end of the pipeline is manual and is for rolling back a service if the deployment causes issues:

Rapid Rollback 🔄 — This phase is a break glass option to rollback the change made to a service. It should be used only if needed. Documentation Link

This means that if you have made it through the Deploy 🚀 stage you have successfully pushed out your first Fastly change with Terraform and a CI/CD pipeline. Congrats!

Summary ⭐

We just covered a ton of info and if all went well, you now have a working CI/CD pipeline to consistently deploy Fastly services in an automated fashion. Let’s summarize what we just did to connect some neurons:

Fastly

Created a fastly.tf file with our general domain and backend info
Created a fastly.vcl file with the Fastly VCL boilerplate
Used Terraform commands locally to build a Fastly service

Pipeline and Terraform

Built a GitLab repository using the Fastly-Framework
Built a default image with Docker to run pipeline jobs
Pushed our default image to the GitLab container registry
Setup Terraform Remote State using AWS S3 + DynamoDB or a comparable method — Related Guide
Set FASTLY_API_KEY and AWS credentials as environment variables for our pipeline jobs
Point each services/<service>/config.tf file and code/ci/test/test.tf config file to your Terraform Remote State
Create a new branch and merge request to trigger our pipeline
Merge our change with the main or master branch and run the deployment pipeline
View our shiny new services in Fastly!

Conclusion 🎇

Pipelines, Infrastructure as Code, and Automation are here to stay. Being able to leverage these technologies for consistent, fast, and reliable deployments is incredibly powerful. These benefits can be amplified when using critical services like Fastly that are the entry point for entire domains. Whether you are an organization of 10 people or 10,000 people, you can benefit from all that CI/CD methodologies have to offer.

I hope you enjoyed this article, learned a thing or two, and got a useful intro into the world of CI/CD with Fastly. If you haven’t already, please checkout the open source framework of this project on GitHub. There is a lot more documentation, code examples, and notes in the repo to help you get a working pipeline stood up with Fastly + Terraform.

❤️ Opensource

This project is 100% opensource and free for anyone/everyone to use. All contributors are welcome. Feel free to open pull requests, leave comments, or fork this project for your own use.