Category Archives: Cloud

Creating a Route 53 Public Hosted Zone with a reusable delegation set ID in CDK

What’s a reuable delegation set anyway?

When you create a Route 53 public hosted zone, four DNS nameservers are allocated to the zone. You then use these name servers with your domain registrar to delegate DNS resolution to Route 53 for your domain.

However: each time you re-create a Route 53 hosted zone, the DNS nameservers allocated will change. If you’re using CloudFormation to manage your public hosted zone this means a destroy and recreate breaks your domain’s name resolution until you manually update your registrar’s records with the new combination of nameservers.

Route 53 reusable delegation sets are stable collections of Route 53 nameservers that you can create once and then reference when creating a public hosted zone. That zone will now have a fixed set of nameservers, regardless of how often it’s destroyed and recreated.

Shame it’s not in CloudFormation

There’s a problem though. You can only create route 53 reusable delegation sets using the AWS CLI or the AWS API. There’s no CloudFormation resource that represents it (yet).

Worse, you can’t even reference an existing, manually-created delegation set using CloudFormation. Again, you can only do it by creating your public hosted zone using the CLI or API.

The AWS CloudFormation documentation makes reference to a ‘DelegationSetId’ element that doesn’t actually exist on the Route53::HostedZone resource. Nor is the element mentioned anywhere else in that article or any SDK. I’ve opened a documentation bug for that. Hopefully its presence indicates that we’re getting an enhancement to the Route53::HostedZone resource some time soon…

So how can we achieve our goal of defining a Route 53 public hosted zone in code, while still letting it reference a delegation set ID?

Enter CDK and AwsCustomResource

CDK generates CloudFormation templates from code. I tend to use TypeScript when building CDK stacks. On the face of it, CDK doesn’t help us as if we can’t do something by hand-cranking some CloudFormation, surely CDK can’t do it either.

Not so. CDK also exposes the AwsCustomResource construct that lets us call arbitrary AWS APIs as part of a CloudFormation deployment. It does this via some dynamic creation of Lambdas and other trickery. The upshot is that if it’s in the JavaScript SDK, you can call it as part of a CDK stack with very little extra work.

Let’s assume that we have an existing delegation set whose ID we know, and we want to create a public hosted zone linked to that delegation set. Wouldn’t it be great to be able to write something like:

new PublicHostedZoneWithReusableDelegationSet(this, "PublicHostedZone", {
    zoneName:  `whatever.example.com`,
    delegationSetId: "N05_more_alphanum_here_K"
 // Probably pulled from CI/CD
});

Well we can! Again in TypeScript, and you’ll need to reference the @aws-cdk/custom-resources package:

import { IPublicHostedZone, PublicHostedZone, PublicHostedZoneProps } from "@aws-cdk/aws-route53";
import { Construct, Fn, Names } from "@aws-cdk/core";
import { PhysicalResourceId } from "@aws-cdk/custom-resources";
import { AwsCustomResource, AwsCustomResourcePolicy } from "@aws-cdk/custom-resources";

export interface PublicHostedZoneWithReusableDelegationSetProps extends PublicHostedZoneProps {
    delegationSetId: string
};

export class PublicHostedZoneWithReusableDelegationSet extends Construct {
    private publicHostedZone: AwsCustomResource;
    private hostedZoneName: string;

    constructor(scope: Construct, id: string, props: PublicHostedZoneWithReusableDelegationSetProps) {
        super(scope, id);

        this.hostedZoneName = props.zoneName;

        const normaliseId = (id: string) => id.split("/").slice(-1)[0];
        const normalisedDelegationSetId = normaliseId(props.delegationSetId);

        this.publicHostedZone = new AwsCustomResource(this, "CreatePublicHostedZone", {
            onCreate: {
                service: "Route53",
                action: "createHostedZone",
                parameters: {
                    "CallerReference": Names.uniqueId(this),
                    "Name": this.hostedZoneName,
                    "DelegationSetId": normalisedDelegationSetId,
                    "HostedZoneConfig": {
                        "Comment": props.comment,
                        "PrivateZone": false
                    }
                },
                physicalResourceId: PhysicalResourceId.fromResponse("HostedZone.Id")
            },
            onUpdate: {
                service: "Route53",
                action: "getHostedZone",
                parameters: {
                    Id: new PhysicalResourceIdReference()
                },
                physicalResourceId: PhysicalResourceId.fromResponse("HostedZone.Id")
            },
            onDelete: {
                service: "Route53",
                action: "deleteHostedZone",
                parameters: {
                    "Id": new PhysicalResourceIdReference()
                }
            },
            policy: AwsCustomResourcePolicy.fromSdkCalls({ resources: AwsCustomResourcePolicy.ANY_RESOURCE })
        });
    }

    asPublicHostedZone() : IPublicHostedZone {
        return PublicHostedZone.fromHostedZoneAttributes(this, "CreatedPublicHostedZone", {
            hostedZoneId: Fn.select(2, Fn.split("/", this.publicHostedZone.getResponseField("HostedZone.Id"))),
            zoneName: this.hostedZoneName
        });
    }
}

Note: thanks to Hugh Evans for patching a bug in this where the CallerReference wasn’t adequately unique to support a destroy and re-deploy

How does it work?

The tricky bits of the process are handled entirely by CDK – all we’re doing is telling CDK that when we create a ‘PublicHostedZoneWithReusableDelegationSet‘ construct, we want it to call the Route53::createHostedZone API endpoint and supply the given DelegationSetId.

On creation we track the returned Id of the new hosted zone (which will be of the form ‘/hostedzone/the-hosted-zone-id’).

The above resource doesn’t support updates properly, but you can extend it as you wish. And the interface for PublicHostedZoneWithReusableDelegationSet is exactly the same as the standard PublicHostedZone, just with an extra property to supply the DelegationSetId – you can just drop in the new type for the old when needed.

When you want to reference the newly created PublicHostedZone, there’s the asPublicHostedZone method which you can use in downstream constructs.

Get your EKS pod to assume an IAM role using IRSA

IRSA, or ‘IAM Roles for Service Accounts’ is a new AWS mechanism that lets pods running in your EKS cluster automatically assume an IAM role when using other AWS resources.

For example, we might have a DynamoDB with a read-write role associated, and we want a payment processing pod in our cluster to write to it. IRSA should make this pretty transparent.

First off, follow this tutorial. The rest of this post is basically an errata sheet for the missing information and steps to get it working.

Our approach

We’re going to tackle this in two sections:

  • Get the s3-echoer example app working in our cluster – this proves the cluster setup is correct
  • Amend our application to correctly use IRSA

Getting the cluster configuration right

If you’re using Terraform…

Two possible screw-ups here depending on which tutorial you’re following – I went with this one, which needs some tweaks:

OpenID Provider CA thumprints missing

Using Terraform to spin out your cluster? This bug will currently leave the OpenID Connect provider in an invalid state, because no certificate thumbprints are added. You’ll need to either hack some Bash into your build, or hard-code the CA thumb in your .tf file. The hacking method is fine if you’re only ever targeting one AWS region. Without this step, you will find that the s3-echoer portion of the original tutorial fails.

Wrong role assumption policy

The tutorial adds a policy to your service account role that limits who can assume the role to your aws-node pods running in your cluster. Since your code will be running in your own pods using your own service account, you need to use your own namespace and service account name instead. For example, if our deployment were under a service user called ‘payments-service-user’ running in the default namespace, we’d want:

data "aws_iam_policy_document" "example_assume_role_policy" {
  statement {
    actions = ["sts:AssumeRoleWithWebIdentity"]
    effect  = "Allow"

    condition {
      test     = "StringEquals"
      variable = "${replace(aws_iam_openid_connect_provider.example.url, "https://", "")}:sub"
      values   = ["system:serviceaccount:default:payments-service-user"]
    }

    principals {
      identifiers = ["${aws_iam_openid_connect_provider.example.arn}"]
      type        = "Federated"
    }
  }
}

You should now be at a point that you can get the s3-echoer to run.

You can check if your own cluster’s service account is setup OK by Bash’ing into any pod and running the env command. We need the following three environment variables set:

  • AWS_DEFAULT_REGION
  • AWS_ROLE_ARN
  • AWS_WEB_IDENTITY_TOKEN_FILE

Amending our application

While the SDKs just work with other AWS hosting models, you need to do some manual work to get EKS IAM role assumption to work. This isn’t consistent across SDK languages – the Go SDK seems to fall into the Just Works category, but others like the Java SDK need a tweak.

In Java, in particular, you need to add an instance of STSAssumeRoleWithWebIdentitySessionCredentialsProvider to a credentials chain, and pass that custom chain to your SDK init code via the withCredentials builder method.

This class doesn’t automatically come as part of the credentials chain. Nor does it automatically initialise itself from environment variables the same way other providers do.

You’ll have to pass in the web identity token file, region name and role ARN to get it running.