Implementing a Service Directory for private API Gateways

AWS’s private API Gateways are great for setting up internal REST Services, but there are some issues with finding and authenticating to them. The problem comes down to the fact that the hostname of the API is generated at random when the API is created, but if you want to set up role-based authorization on your API, your clients will need to know this hostname in order to create a valid authenticated request. Our solution was to have our Cloudformation stacks add the hostname to AWS ParamStore and set up a public API to access the data.

Background

Amazon’s API Gateway service provides a simple mechanism for setting up HTTP endpoints backed by things like S3 files, DynamoDB tables, or Lambda functions, which is what we’re using in this scenario. We defined an API in API Gateway such that when a particular URL is hit, our configured Lambda function is invoked with information about the request, and the value returned by our function is used to create the response.

APIs in API Gateway can be public or private (or regional): a public API can be reached directly on the public Internet. You can still implement authorization policies on your API to make sure that even if someone is able to reach your endpoint, they won’t actually be authorized to invoke it, which means they‘ll get a 403 Forbidden response, and your lambda won’t be invoked.

For even more security, you can make your API private, which basically means it is only accessible from inside an AWS Virtual Private Cloud (“VPC”). It has no public IP address and is theoretically unreachable from the public Internet. This added security is great if your API is only intended for your own services, for instance in a cloud-based microservice platform.

You can even enable communications from other VPCs to your private API gateway, for instance, if different services in your platform run on different VPCs. To do this, you configure a VPC endpoint (“VPCe”) in the VPC that hosts your private API and in the VPCs that your clients will run in. To reach your API from within a VPC, clients will make the request to the address of your VPCe, which is an AWS generated domain name that resolves to a private network IP address in the 172.16.0.0/12 range. Through a bit of routing magic that AWS provides inside its VPCs, requests to this address will be routed from the client’s VPC, through Amazon’s private network, to your VPC where your private API can handle it, without it ever crossing the public Internet. Additionally, because this communication is happening over Amazon’s private networks, the API Gateway knows for sure what VPCe the request came from, which means you can implement an authorization policy that whitelists certain VPCe’s and denies any other requests.

Sometimes you’ll need more fine-grained authorization, for instance, if your clients are sharing a VPC with other services that you do not want to authorize. In this case, you might want to add an IAM role authentication layer: for requests to be accepted by your API, they will need to sign the request with a Sig v4 signature using an authorized IAM role.

With these two layers, IAM role authorization and VPCe authorization, the Resource Policy for your API will look something like this:

The Rub

To create an authorized request, it must be signed, and to create a valid signature, the request must have the correct Host header. The authorization policy is specified by your API’s resource policy, which is implemented by API Gateway, which means it must have the correct hostname for your API Gateway. This hostname is generated randomly when the API is created and looks something like:

abcd1234ef.execute-api.us-east-1.amazonaws.com

The problem is that i̶f̶ ̶y̶o̶u̶ ̶e̶v̶e̶r̶ when you come to a point where your API needs to be torn down and recreated from scratch (e.g., a borked stack update in CloudFormation), you’re going to get a brand new random host header and all your clients are going to need to know it in order to make authenticated requests.

Our Solution

To solve this conundrum, we came up with a pretty simple solution of writing the hostname to AWS ParamStore on every deploy. With CloudFormation, this is trivial to do: create a resource of type AWS::SSM::Parameter, and reference your API resource (type AWS::ApiGateway::RestApi) in order to get that random bit of text at the beginning of the host name. The rest of the hostname is predictable and can be created in CloudFormation using pseudoparams. The relevant parts of the CloudFormation template look something like this:

The name of the parameter will need to be predictable so you can find it, and unique so there’s clashing. In a typical deploy, you’ll have an app name and a stage or environment: basically a specific instance of the app; include these in the parameter name and you should be in good shape.

ParamStore doesn’t support resource sharing across accounts, so if you need to access these hostnames across accounts, there are two options:

1. Use IAM role assumption
2. Put a service in front of it.

We opted for number 2, creating a new Lambda in the same account that our apps are deployed to and giving it access to the hostname params. When invoked with the name of an app and the stage, the Lambda fetches the hostname from ParamStore and returns it to the caller.

To finish it off, we put it behind API Gateway to make it easy to invoke across accounts. To avoid a stack of turtles, this API Gateway needs to be public, or you’re just going to run into the same problem trying to reach it. Since this API doesn’t have any sensitive data, making it public shouldn’t be a problem.

Wrap Up

Private APIs in API Gateway are relatively new, having been introduced not quite a year ago. They introduce a significant improvement in security for APIs that don’t need to be publicly accessible, but there are a few holes in the service that make things a little more complicated than we’d like; specifically that you can’t specify a fixed hostname for private APIs. Based on the way things usually go in AWS, it seems likely that in the coming months or years, there will be updates to API Gateway that fill this hole.

In the meantime, a simple solution using ParamStore is easy to implement and solves the problem with minimal complication for clients or service owners.

This is by no means the only solution to the problem. Another AWS service, CloudMap, provides service discovery mechanisms and may be a workable solution. And there may very well be some kind of native API Gateway solution that we missed entirely, so if you’ve come across this problem and found another solution, let us know in the comments.

Otherwise, good luck and have fun!