3 tips to manage drift from manual changes in Terraform
How do you manage drift from manual changes in Terraform? If someone changes something manually on the infrastructure, how do you handle it on Terraform, and how do you run Terraform plan next time?
There are different ways to manage drift from manual changes on your infrastructure in Terraform, depending on the case. Here are 3 options :
Manage drift from manual changes in Terraform : Change the code manually to update your State
It is always easier to start with an example. So, let’s say you have a security group that was changed manually by one of your team members, like opening an HTTP port for a specific subnet, and you discover this at the next
terraform apply. (This is an easy case).
Following this action, you will find a diff in your
terraform output (be it on CI, or on your laptop). That means that you can add this difference as a snippet directly on your terraform code and apply it. So what you will need to do is basically manually add the code after the deployment. And thus, when you next do a
terraform apply, Terraform will notice that the state doesn’t include this new item (in our example, allowing HTTP for a sub network) and will require your cloud provider (AWS for example) to do it, and AWS will return : “I already have it”. So, in this easy case, the difference just appears in the
TFstate file, and you solved your case manually. It’s a bit like fixing a conflict during a git merge.
So, the key steps are :
- Write the missing code
- Push it
- Apply it
- The delta disappears
- And the final state is updated on the state file.
Reverse Terraform the new existing environment with a tool like Terraformer or Terraforming.
If you have much more difficult case, like a full VM or even worse maybe a full Kubernetes with 10,000 nodes run manually, you obviously will not try to write the code manually like I just did for a very simple case. I would recommend you to use a tool like Terraformer or Terraforming.
This kind of tools scans your cloud provider accounts for resources and parameters. And once they have all the parameters (let’s say for example on a VM : hard disk, IP address, operating system type, etc…), they will turn all those elements into valid Terraform code and create a TFState from the code.
Terraformer originates Waze SRE team at Google in Tel Aviv. It will do something like 80% of the job, and you just need to adapt some things here and there.
So, using this tool, you will end up with 3 elements :
- the code that you just extracted,
- an existing manual deployment,
- and a state file
Problem now is that these elements are not really merged to your existing infrastructure repo. That’s the topic of the next subject : chose what you import.
Import specific parts of your new code with the Terraform import subcommand
Once you have your code, you can import it by using the
terraform import subcommand to be very precise / picky about what you import.
Let’s say, as an example that there is a specific IP address that you launched manually for an old load balancer, but production needs to keep this particular IP address and you want to be sure that it’s under control on Terraform. You can write the corresponding terraform code and import the corresponding ID on AWS.
It’s quite different from just pushing / applying some code and checking if “it works” on the other side. You have this code, and you know that if you execute it right now, it is going to be duplicate (in this precise example because you can have hundreds of IP address with a name; it was not the case in the previous example with the security group where you can only have one port 80 open).
So in this example, I would define this very specific IP address through an importer like Terraformer and copy/paste it on my existing code. Then I would not apply this directly but import it with the
terraform import subcommand.
Here’s an example of how the terraform import command looks like.
Just zooming in on the
terraform --help command
So, as a wrap-up, here the key tips to manage drift from manual changes in Terraform
The 3 ways I would recommend you to manage manual changes in your infrastructure with Terraform, depending on the situation would be:
- Just write the code and apply it for small changes that don’t need to be imported again, and let it reconcile with the state file automatically,
- Import your new environment and reverse-terraform it, with a tool like Terraformer,
- And/or import very precise bits manually using the terraform import subcommand.