Manage multiple Terraform environments
How do you start managing several Terraform environments?
The need to manage multiple Terraform environments is very common. Indeed, getting started is one thing but then you end up with various environments that you need to manage, several teams etc… So how do you manage terraform when you start having several environments like dev, staging, prod, and how do you manage the complexity?
Manage multiple Terraform environments : getting started with TF Files
If you need to manage several Terraform environments, there are a lot of ways to ramp up your approach and get started.
Getting started step by step with single
tf files. The task can be daunting to know all the good practices at first, and probably you might get at best distracted, and at worse discouraged. So you can get started very naively with one single
tf state. Create a simple Terraform file, call it
production.tf, write your
VMs or whatever in it. Very soon you will be able to create another environment. Let’s call it
staging.tf. It is going to be still a single
tf state file which is not bad, but it won’t scale. But that’s one way to start doing it step by step.
The Hashicorp Terraform main recommendation is to use what they call now workspaces.
workspaces is a Terraform sub command. It was previously named environments.
The goal of workspaces is to separate one
tfstate file per environment. So let’s say you have a
QA or a
staging and a
production environment : using that Terraform workspace sub command, will switch
tfstates. That is the recommended way to do things by Hashicorp, but it does not mean that you have to do it exactly that way.
Use folders in your Git repo
Another way to do it, is to split your state files not using workspaces but basically using folders. Just create folders in your
git repo, give each folder a name, like
production and just generate different
tfstate files from those folders. It’s really easy, and then then you are already using splits environments “TFstate-wise”.
Modules : a common way of managing Terraform environments
Using modules is kind of becoming a standard now. It is a bit advanced, but modules basically work by injecting
variables in them like
strings, etc… They are a very handy feature for Terraform.
Modules contain generic code. Let’s take a standard
vpc as an example. This VPC can take several values like the
subnetwork and a
name. What you can do basically, is create two folders, one for each environment and your first Terraform file will just specify what
address space your
staging vpc will have, while the other will have another address space. So you solve that issue just by injecting different values in modules.
So that would be the main ways I would suggest using environments.
Just get started easily, naively with a single TF state, create different folders, split the states by folders, use modules inject different types of values in them, or basically just stick to what Hashicorp suggest, like use workspaces.
Organize your resources and define your directory layout
Last thing to consider is how you will organize your resources and directory layout. Would you use one or several repos? etc..
There is no one answer to that, but one common set up is to have your modules handled separately and called from a single
repository. Let’s take an example: if you have a module that knows how to properly configure a
vpc from two
variables, then you have some proper Terraform code. This code can be stored in a distinct Terraform git repo, managed, versioned and released like any other project. But still, it cannot be used “as is”. You still need your infra repo to call this module, a bit like you would do with a standard
library in engineering.
For some people, the folder hierarchy matters too. It’s very common to see test folders for all the testing, separate folders for the modules, and other folders for the environments as well.
One last common pattern that I think of, is the use of variables files with all the different kind of values you can have, so that environments can be sent to the execution using those environment variables.