Terraform code testing
Terraform code testing is a recurring subject that we keep hearing about. People are wondering a lot about infrastructure code testing, even if we don't see much adoption of testing in practice. First of all, should you test your code, and what are the different strategies to automate tests, and what are the best open source tools to do so?
Terraform code testing is key and yes, you should be testing it. There is a lot of ways you can do it. It is very similar to standard software engineering processes. It comes from the same culture, so there is no surprise here.
Terraform code testing : start by using linters
You can execute a linter directly on your laptop if that is what you want or execute it on your CI/CD system because you want to ensure others respects some coding conventions or standardization.
One linter in Terraform that I really like is named TFlint. It is open source and available on Github. This linter even has a deep linting feature that goes beyond simple linting. Imagine that you need to request an existing correct VM, let’s say a
T3 large instance. If your account is not allowed any more capacity (like for example you already launched all the VMs you were allowed to request), the linter will send a request to the AWS API for the correctness of what you request plus the ability of your request to be executed within your account limits. So, it obviously does all the things that a proper linter in development will do, like single quotes, quality, tabs, etc… and, it goes beyond the validator and the formatter that you can find directly on Terraform.
Perform unit testing on your Terraform code with TDD
A proper agile engineering workflow would begin with test-driven development (TDD). I’m pretty sure “absolutely everybody” does TDD here (…) but that’s the theory as well and we all would like to do it.
Basically when you start doing some TDD, you write unit tests, like : “well I want to create my VPC and my VPC is supposed to have this address space”. So, you will do exactly what you would do in engineering: you start by writing your tests, then you write your function, in Terraform it is called a
resource, then you execute that. I would use any
RSpec derivative tools for that: Serverspec, InSpec… and your TDD steps would look something like:
- Start by writing your test : it fails,
- Write the resource and it doesn’t fail because it will launch in an isolated environment (especially that VPC).
It’s going to test what you wanted and then it’s going to destroy it. You can do exactly what you would do in engineering as well, like in your CI, execute all the tests on pull requests. Choosing how much security you want once you get started on testing is a sensitive subject. I personally execute all the tests prior to merging anything, but I’m much lighter during the pull request phase, because launching all the tests, all the times, at each commit can be very long.
Do integration tests as well
Something you can do which is quite common in QA processes is to randomize things, like variables. Terratest is another tool which is really cool for integration tests. During this phase of integration, I would randomize items like regions names. You usually always test your code in the same region. By randomizing it, you will have different results, and you will ensure that it works.
Maybe your names are unique, and the names of your resources as well. For example, your
s3 bucket needs to be unique. It is the same process as an engineering project:
- Use the tools
- Lint your code
- Perform unit test on your files
- Randomize that
- Put the results in your CI
- Add that to your pull requests
- Tag your code and your releases, (like
- Use feature branches and then merge to master and then release a code version
So, what kind of tools do you recommend to test your infrastructure code?
I mentioned all the
RSpec derivative tools for that: Serverspec, InSpec… InSpec is made by Chef Serverspec is more community based, I think. That’s for the units testing but it’s not only for Terraform it’s more general. You can test anything from infrastructure to Docker images, to processes running on a server or that kind of things, but it’s readable it’s close to English.
There is another tool that I really like called GOSS. It is open source and really cool. It is very simple, based on
YAML. It is not meant at all to test Terraform code but rather designed to test results. For example, you created a security group and you want to ensure that
Port 22 for SSH is closed. You can test that, plus processes, packages, DNS resolver, users… or check if your http endpoint on kubernetes is answering, that kind of things…
I think it can even output results of the tests as a Nagios-compatible output so you are even able to launch it as a
/health point in your micro service to ensure that things are running properly. It is really simple and lightweight. I really like this tool. It works very well for Docker files etc …
Probably the most complete, but also but the most complex is Terratest from our friends at Gruntwork. Terratest is a Golang library. It requires skills to master (well it’s Go…) but allows you to test anything that has an API like AWS, Azure, GCP, kubernetes, Docker images, Packer builds… (Packer is the tool from Hashicorp to build AMIs: virtual machine disk image) and even Helm charts as well…
Basically you just need to create the name of your
resource_test.go, you import your libraries and you just execute this, but it requires a certain level of skills in Golang and a deep understanding of what you want to test.
Wraping things up
So I would say :
- GOSS for very high level, easy stuff with simple YAML, simple binary, very easy, straightforward.
- Serverspec and InSpec for all the unit testing. It’s all open source.
- And Terratest obviously for integration. It is very complete.
Integrate all that in your CI CD system, or at least anywhere before shipping/ releasing it and into the wild or you are going to have issues if you don’t 🙂
Terraform automation for growing teams
Speed up deployment cycles | Reduce mistakes | Empower your team
CloudSkiff is an Infrastructure as code platform that provides Terraform automation and collaboration. We help growing teams safely ship infrastructure in short cycles and make their code better.