Managing Terraform State from Local
Specifically for managing the Terraform state, it can be much easier to interrogate the state and potentially fix it up from your local command line than always trying to make it happen with GitHub Actions.
Make sure the terraform and terragrunt executables are runnable from the command line.
- Terraform should be v1.1.7
- Terragrunt can be any version that works with Terraform v1.1.7 (e.g. 0.39.2)
Your personal AWS credentials will time out. That's why the commands outlined in this document can't be fully scripted. In GitHub actions, the commands are performed by an agent running in Terraform Cloud which operates via a persistent service account. We don't get persistent service accounts.
- Login to AWS
- Choose the environment you want to connect to from command line
- Copy the AWS creds to a terminal ("Click for Credentials")
- Execute the commands in your local terminal
- run this line to get terraform token
aws ssm get-parameter --name "/octk/tfc/team-token" --with-decryption | jq -r '.Parameter.Value'
- copy the parameter into ~/.terraformrc (create this file if necessary)
credentials "app.terraform.io" {
token = "terraform-token-goes-here"
}
Alternatively can run command "terraform login" which will populate this file.
This only needs to be done once.
When Terraform runs from GitHub actions, the workflow at .github/workflows/reusable_terraform_server.yml sets up a bunch of variables in the "Create Terraform vars" step. The values for these variables are necessary when doing a "plan" or an "apply", but not when executing the "state" command. The file needs to be set up with dummy values in order for Terraform to actually work.
Create a file called "github.auto.tfvars" in terraform/{environment} depending on the environment you want to run against ("terraform/dev", "terraform/test", or "terraform/prod"). Make sure you don't check this into version control (use .gitignore if necessary).
db_cluster_snapshot_identifier = "no_value_required"
execute_flyway = "no_value_required"
dev_oidc_idir_idp_client_secret = "no_value_required"
test_oidc_idir_idp_client_secret = "no_value_required"
prod_oidc_idir_idp_client_secret = "no_value_required"
dev_oidc_bceid_business_idp_client_secret = "no_value_required"
test_oidc_bceid_business_idp_client_secret = "no_value_required"
prod_oidc_bceid_business_idp_client_secret = "no_value_required"
forest_client_api_api_key = "no_value_required"
dev_oidc_bcsc_idp_client_secret = "no_value_required"
test_oidc_bcsc_idp_client_secret = "no_value_required"
prod_oidc_bcsc_idp_client_secret = "no_value_required"
In the directory "infrastructure/server", create two zip files. They don't need any contents, they just need to be there in order to get the Terraform commands to not error out. Make sure you don't check these files into version control (use .gitignore if necessary).
- fam_auth_function.zip
- fam-ui-api.zip
export tfc_workspace=[AWS_LICENSE_PLATE]-[ENV]
When the above has been accomplished, you can change to the appropriate directory in your terminal and start running Terraform commands (depending on the environment you want to run against --"terraform/dev", "terraform/test", or "terraform/prod").
terragrunt state list
One problem we have is that if a KMS key is created by Terraform and then removed from the configuration, Terraform will try to delete it. This does not work due to the rules in the AWS Secure Accelerator Environment (SAE). If you can find the key that you want to remove, you can remove it from the Terraform state with:
terragrunt state rm [name_of_resource]
terragrunt state pull > backup.tfstate
terragrunt state push backup.tfstate
terraform force-unlock -force [replace with lock id]
The other way is to delete the lock file from AWS console. You need to know the lock ID that is having trouble with.
DO NOT delete the wrong lock file.
It's fully possible to execute all the other Terraform commands from a local environment, but the github.auto.tfvars file would need to be populated with actual values in order for it to work, and the two zip files would need to be built correctly as well. The commands for building the zip files can be pulled out of the GitHub Actions files. The various secrets live inside of GitHub Actions secrets but cannot be retrieved from that location. The FAM team has the secrets stored securely somewhere very secret as these values should not be floating around in chat or show up in repositories or logs!!
The Terragrunt setup in "terraform-frontend" could be run in the same way from local command line, but the dependencies would need to be set up in advance. If you want to do it, mimic the steps in the "reusable_terraform_frontend.yml" workflow and give it a shot.