Deploy Kubernetes in an Existing AWS VPC with Kops and Terraform

Kops is a relatively new tool that can be used to deploy production-ready Kubernetes clusters on AWS. It has the ability to create a highly-available cluster spanning multiple availability zones and supports a private networking topology. By default, Kops will create all of the required resources on AWS for you — the EC2 instances, the VPC and subnets, the required DNS entries in Route53, the load balancers for exposing the Kubernetes API, and all of the other necessary infrastructure components.

For organizations that use Terraform, Kops can instead be used to generate a Terraform configuration for all of the aforementioned AWS resources. This will allow them to use the familiar terraform plan and terraform apply workflow to build and update their Kubernetes infrastructure. The Terraform configuration that Kops generates will include new VPC, subnet, and route resources.

But what if you want to use Kops to generate a Terraform configuration for a Kubernetes cluster in an existing VPC? In this post, I will walk through the process to achieve this.

Create a VPC with Terraform

To simulate this process, we need an existing VPC infrastructure to work with. In the repository associated with this post, I have some Terraform modules that will let us easily create a VPC with public / private subnet pairs across multiple availability zones. It will also create NAT gateways to allow outbound internet traffic for instances on the private subnets.

Let’s create this infrastructure. Go ahead and clone the repository.

git clone https://github.com/ryane/kubernetes-aws-vpc-kops-terraform.git

Before we run terraform apply, we need to configure some variables. In variables.tf, you need to set the name variable. It is used in several places in our configuration and should be set to the domain name you are going to be using for this cluster. You can either modify the variables.tf file directly or use one of the supported mechanisms to assign Terraform variables.

Optionally, you can configure the region and availability zone variables. By default, we are going to be creating a highly available cluster with Kubernetes masters in us-east-1a, us-east-1c, us-east-1d. You can also configure the env and vpc_cidr variables, if desired.

Let’s take a look at main.tf. Here is how we define our VPC:

module "vpc" {
  source   = "./modules/vpc"
  name     = "${var.name}"
  env      = "${var.env}"
  vpc_cidr = "${var.vpc_cidr}"

  tags {
    Infra             = "${var.name}"
    Environment       = "${var.env}"
    Terraformed       = "true"
    KubernetesCluster = "${var.env}.${var.name}"
  }
}

module "subnet_pair" {
  source              = "./modules/subnet-pair"
  name                = "${var.name}"
  env                 = "${var.env}"
  vpc_id              = "${module.vpc.vpc_id}"
  vpc_cidr            = "${module.vpc.cidr_block}"
  internet_gateway_id = "${module.vpc.internet_gateway_id}"
  availability_zones  = "${var.azs}"

  tags {
    Infra             = "${var.name}"
    Environment       = "${var.env}"
    Terraformed       = "true"
    KubernetesCluster = "${var.env}.${var.name}"
  }
}

Most of the heavy lifting is done in the vpc and subnet-pair modules. Those modules are responsible for creating the VPC, private and public subnets, NAT Gateways, routes, and security groups. One thing to note is the KubernetesCluster tag that we are setting on our resources. This tag is required by some of the Kubernetes AWS integration features (such as creating a LoadBalancer service that is backed by an ELB). If you are installing Kubernetes in your own existing VPC and want to take advantage of the Kubernetes AWS integration, you’ll need to ensure that this tag is added to your resources.

Besides the networking infrastructure, we also need to create the hosted zone for our cluster domain name in Route53. If you are following along and already have your domain name registered in Route53, you can remove this resource from your local configuration.

resource "aws_route53_zone" "public" {
  name          = "${var.name}"
  force_destroy = true
  ...
}

Finally, Kops also requires an S3 bucket for storing the state of the cluster. We create this bucket as part of our Terraform configuration:

resource "aws_s3_bucket" "state_store" {
  bucket        = "${var.name}-state"
  acl           = "private"
  force_destroy = true

  versioning {
    enabled = true
  }
  ...
}

Let’s go ahead and create our infrastructure. You will need to provide credentials for an IAM user that has sufficient privileges to create all of these resources. For simplicity, I am using a user that has the following policies associated:

• AmazonEC2FullAccess
• IAMFullAccess
• AmazonS3FullAccess
• AmazonVPCFullAccess
• AmazonRoute53FullAccess

export AWS_ACCESS_KEY_ID=<access key>
export AWS_SECRET_ACCESS_KEY=<secret key>

terraform get
terraform apply

The apply may take a few minutes but when it’s done, you should have a new VPC and associated resources in your AWS account.

Deploy Kubernetes with Kops and Terraform

At this point, we have our base AWS infrastructure up and running. Now, we can move on to using Kops to generate the Terraform for our Kubernetes cluster.

First, we should export a few environment variables that we will be using in our Kops commands.

export NAME=$(terraform output cluster_name)
export KOPS_STATE_STORE=$(terraform output state_store)
export ZONES=us-east-1a,us-east-1c,us-east-1d

The $NAME and $KOPS_STATE_STORE variables are populated by our Terraform outputs. $NAME should be set to <env>.<yourdomain.com> and $KOPS_STATE_STORE should be s3://<yourdomain.com>-state.

$ echo $NAME
staging.example.com

$ echo $KOPS_STATE_STORE
s3://example.com-state

You can explicitly set those variables if you are not working with the sample Terraform configuration from this post.

The $ZONES variable should set to the same availability zones that we are using in variables.tf.

export ZONES=$(terraform output -json availability_zones | jq -r '.value|join(",")')

Now we can run Kops. Here is the command we will use to create our cluster:

kops create cluster \
    --master-zones $ZONES \
    --zones $ZONES \
    --topology private \
    --dns-zone $(terraform output public_zone_id) \
    --networking calico \
    --vpc $(terraform output vpc_id) \
    --target=terraform \
    --out=. \
    ${NAME}

Let’s break this down.

• master-zones: tell Kops that we want one Kubernetes master in each zone in $ZONES. If you are using the default configuration in this post, that will be 3 masters — one each in us-east-1a, us-east-1c, and us-east-1d.
• zones: tells Kops that our Kubernetes nodes will live in those same availability zones.
• topology: tells Kops that we want to use a private network topology. Our Kubernetes instances will live in private subnets in each zone.
• dns-zone: specifies the zone ID for the domain name we registered in Route53. In this example, this is populated from our Terraform output but you can specify the zone ID manually if necessary.
• networking: we are using Calico for our cluster networking in this example. Since we are using a private topology, we cannot use the default kubenet mode.
• vpc: tells Kops which VPC to use. This is populated by a Terraform output in this example.
• target: tells Kops that we want to generate a Terraform configuration (rather than its default mode of managing AWS resources directly).
• out: specifies the output directory to write the Terraform configuration to. In this case, we just want to use the current directory.

When you run this command, Kops does several things including:

1. Populating the KOPS_STATE_STORE S3 bucket with the Kubernetes cluster configuration.
2. Creating several record sets in the Route53 hosted zone for your domain (for Kubernetes APIs and etcd).
3. Creating IAM policy files, user data scripts, and an SSH key in the ./data directory.
4. Generating a Terraform configuration for all of the Kubernetes resources. This will be saved in a file called kubernetes.tf.

The kubernetes.tf includes all of the resources required to deploy the cluster. However, we are not ready to apply this yet as it will want to create new subnets, routes, and NAT gateways. We want to deploy Kubernetes in our existing subnets. Before we run terraform apply, we need to edit the cluster configuration so that Kops knows about our existing network resources. The kops edit cluster command will open your $EDITOR with your cluster settings in YAML format. We need to replace the subnets section with our existing vpc and subnet information.

kops edit cluster ${NAME}

Your subnets map should look something like this:

subnets:
- cidr: 10.20.32.0/19
  name: us-east-1a
  type: Private
  zone: us-east-1a
- cidr: 10.20.64.0/19
  name: us-east-1c
  type: Private
  zone: us-east-1c
- cidr: 10.20.96.0/19
  name: us-east-1d
  type: Private
  zone: us-east-1d
- cidr: 10.20.0.0/22
  name: utility-us-east-1a
  type: Utility
  zone: us-east-1a
- cidr: 10.20.4.0/22
  name: utility-us-east-1c
  type: Utility
  zone: us-east-1c
- cidr: 10.20.8.0/22
  name: utility-us-east-1d
  type: Utility
  zone: us-east-1d

There should be one Private type subnet and one Utility (public) type subnet in each availability zone. We need to modify this section by replacing each cidr with the corresponding existing subnet ID for that region. For the Private subnets, we also need to specify our NAT gateway ID in an egress key. Modify your subnets section to look like this:

subnets:
- egress: nat-0b2f7f77b15041515
  id: subnet-8db395d6
  name: us-east-1a
  type: Private
  zone: us-east-1a
- egress: nat-059d239e3f86f6da9
  id: subnet-fd6b41d0
  name: us-east-1c
  type: Private
  zone: us-east-1c
- egress: nat-0231eef9a93386f4a
  id: subnet-5fc6dd16
  name: us-east-1d
  type: Private
  zone: us-east-1d
- id: subnet-0ab39551
  name: utility-us-east-1a
  type: Utility
  zone: us-east-1a
- id: subnet-656b4148
  name: utility-us-east-1c
  type: Utility
  zone: us-east-1c
- id: subnet-cdc7dc84
  name: utility-us-east-1d
  type: Utility
  zone: us-east-1d

Of course the IDs will be different for you if you are following along. You can use terraform output (or the AWS console/api) to find the correct IDs.

terraform output -json | docker run --rm -i ryane/gensubnets:0.1

After you edit and save your cluster configuration with the updated subnets section, Kops updates the cluster configuration stored in the S3 state store. However, it will have not yet updated the kubernetes.tf file. To do that, we need to run kops update cluster:

kops update cluster \
  --out=. \
  --target=terraform \
  ${NAME}

If you look at the updated kubernetes.tf, you will see that it references our existing VPC infrastructure instead of creating new resources. Perfect!

Now we can actually build our Kubernetes cluster. Run terraform plan to make sure everything looks sane to you and then run terraform apply.

After the apply finishes, it will take another few minutes for the Kubernetes cluster to initialize and become healthy. But, eventually, you should have a working, highly-available Kubernetes cluster!

$ kubectl get nodes
NAME                            STATUS         AGE
ip-10-20-101-252.ec2.internal   Ready,master   7m
ip-10-20-103-232.ec2.internal   Ready,master   7m
ip-10-20-103-75.ec2.internal    Ready          5m
ip-10-20-104-127.ec2.internal   Ready,master   6m
ip-10-20-104-6.ec2.internal     Ready          5m

Cleaning Up

If you want to delete all of the infrastructure we created in this post, you just have to run terraform destroy. If you used a different S3 bucket for your $KOPS_STATE_STORE, you may also want to run kops delete cluster to remove the Kops state. Otherwise, the entire S3 bucket will be destroyed along with the rest of the infrastructure.

There is a lot more to Kops than we covered here. I encourage you to check out the documentation.