Terraform is a great infrastructure-as-code tool which we love at Ensono Digital, but effectively implementing the aws_lambda_function resource in the real world can be a little bit challenging. I am going to explain an approach for provisioning lambda functions without needing a pre-built deployment package.
Terraform and lambda – chicken or the egg?
The lambda function resource is a bit special, as it requires a suitable deployment package containing your function code to exist before it can go ahead and create the function. This contrasts to typical Terraform resources, and infrastructure as code in general, where you can stand-up resources in advance. The necessary deployment package can be in the form of a local file or an s3 object.
(This is requirement comes from the underlying AWS CreateFunction API action.)
Local file vs s3 object
If you have a function which is using an interpreted language, and the code is never going to change, then storing a pre-packaged deployment package alongside your Terraform and using the local “filename” option is probably going to suffice. In which case, move along, nothing to see here.
However, functions which using a compiled language, which get built and packaged in a CI pipeline are going to require a more creative solution.
What Terraform wants you to do
What Terraform wants you to do feels something like this:
Great, except providing actual code to Terraform just feelswrong for many reasons:
Last time I checked, Terraform was an Infrastructure-as-codetool. Infrastructure! Since when did you have to provide application codeto create infrastructure?
There is a circular dependency between the function codeCI/CD pipeline and Terraform
You need to ensure Terraform always references the correct versionof the deployment package for each environment, this might not always bethe latest
How do you ensure Terraform doesn’t subsequently “update”the function code outside of your CI/CD process?
Surely there must be a better way….
What I want to do
I want to create the infrastructure (specifically, an empty shell of a lambda function, with it’s corresponding IAM role, triggers, log configuration, permissions, etc) first, so that a separate CI/CD pipeline can then build, package, and deploy the function code. Like this
My first plan was to simply trick Terraform into provisioning my lambda function by giving it an empty zip file, so I whipped out the trusty archive_file provider and pointed it to an empty dir and fired away:
Not so fast! Looks like someone has thought of this already, and the AWS API swiftly rejects my file with the following message:
OK then, so let’s just add some content into the zip file:
Now obviously any invocations of the lambda function will fail at this point, but it’s now super easy to deploy your actual code to the function from a CI/CD pipeline completely independently from Terraform, for example:
There are frameworks available which are designed specifically for implementing serverless architectures. These do a lot of the heavy lifting by assisting with creation of supporting resources and configuration such as IAM roles, triggers, permissions, etc. In addition to provisioning resources, they can also handle packaging and deployment of the function code, so if you have a significant amount of lambda functions, should really check them out:
There is overlap between Terraform and the “Serverless” / “SAM frameworks and as these can all provision supporting resources as well as the lambda function itself. Consider how the rest of your infrastructure is going to be managed, as if you end up using Terraform and the Serverless framework, this will cause some fragmentation in tooling. For example, what if you also have a significant container infrastructure provisioned using Terraform, but some containers share resources (s3, sqs, etc) with lambda functions, where should the shared resources get provisioned? Terraform or the Serverless framework?