Advanced Troubleshooting

---

Introduction

Terraform is powerful but can sometimes exhibit unexpected behavior. This chapter covers the most common problems and their solutions.

---

1. State problems

Stuck State Lock

Symptom:

Error: Error acquiring the state lock
Lock Info:
  ID:        12345678-xxxx-xxxx-xxxx-xxxxxxxxxxxx
  Path:      s3://bucket/terraform.tfstate
  Operation: OperationTypeApply
  Created:   2024-01-15 10:30:00

Solutions:

# 1. Vérifier si un autre process tourne
ps aux | grep terraform

# 2. Vérifier dans DynamoDB (AWS)
aws dynamodb get-item \
    --table-name terraform-locks \
    --key '{"LockID": {"S": "s3://bucket/terraform.tfstate"}}'

# 3. Force unlock (DANGEREUX - s'assurer qu'aucun process n'est actif)
terraform force-unlock 12345678-xxxx-xxxx-xxxx-xxxxxxxxxxxx

# 4. Supprimer manuellement dans DynamoDB (dernier recours)
aws dynamodb delete-item \
    --table-name terraform-locks \
    --key '{"LockID": {"S": "s3://bucket/terraform.tfstate"}}'

Corrupted State

Symptom:

Error: Failed to load state: state snapshot was created by Terraform v1.5.0

Solutions:

# 1. Télécharger le state
aws s3 cp s3://bucket/terraform.tfstate ./terraform.tfstate.backup

# 2. Vérifier le JSON
jq '.' terraform.tfstate.backup

# 3. Récupérer une version précédente (S3 versioning)
aws s3api list-object-versions \
    --bucket bucket \
    --prefix terraform.tfstate

aws s3api get-object \
    --bucket bucket \
    --key terraform.tfstate \
    --version-id "PREVIOUS_VERSION_ID" \
    ./terraform.tfstate.restored

# 4. Restaurer le state
terraform state push terraform.tfstate.restored

State Drift

Symptom: Resources modified manually, state out of sync

# Détecter le drift
terraform plan -detailed-exitcode
# Exit code 2 = changes detected

# Rafraîchir le state depuis l'infrastructure réelle
terraform refresh

# Ou avec apply
terraform apply -refresh-only

# Importer une ressource manquante
terraform import aws_instance.example i-1234567890abcdef0

# Supprimer une ressource du state (sans la détruire)
terraform state rm aws_instance.orphan

---

2. Provider problems

Incompatible version

Symptom:

Error: Unsupported Terraform Core version
This configuration does not support Terraform version 1.6.0.

Solution:

# Fixer les versions dans versions.tf
terraform {
  required_version = ">= 1.5.0, < 2.0.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"  # >= 5.0.0, < 6.0.0
    }
  }
}

# Mettre à jour les providers
terraform init -upgrade

# Utiliser tfenv pour gérer les versions
tfenv install 1.6.0
tfenv use 1.6.0

Provider Timeout

Symptom: Error: Timeout while waiting for state...

Solutions:

# Augmenter les timeouts
resource "aws_db_instance" "main" {
  # ...
  
  timeouts {
    create = "60m"
    update = "60m"
    delete = "60m"
  }
}

# Pour EKS (particulièrement long)
resource "aws_eks_cluster" "main" {
  # ...
  
  timeouts {
    create = "45m"
    update = "60m"
    delete = "30m"
  }
}

Expired credentials

Symptom:

Error: error configuring Terraform AWS Provider: 
ExpiredToken: The security token included in the request is expired

Solutions:

# 1. Rafraîchir les credentials
aws sts get-caller-identity

# 2. Avec SSO
aws sso login --profile myprofile

# 3. Avec assume role
aws sts assume-role \
    --role-arn arn:aws:iam::123456789012:role/MyRole \
    --role-session-name mysession

# 4. Variables d'environnement
export AWS_ACCESS_KEY_ID="..."
export AWS_SECRET_ACCESS_KEY="..."
export AWS_SESSION_TOKEN="..."  # Si STS

---

3. Dependency problems

Dependency cycle

Symptom:

Error: Cycle: aws_security_group.a, aws_security_group.b

Solution:

# MAUVAIS - Crée un cycle
resource "aws_security_group" "a" {
  ingress {
    security_groups = [aws_security_group.b.id]
  }
}

resource "aws_security_group" "b" {
  ingress {
    security_groups = [aws_security_group.a.id]
  }
}

# CORRECT - Utiliser des règles séparées
resource "aws_security_group" "a" {
  name = "sg-a"
}

resource "aws_security_group" "b" {
  name = "sg-b"
}

resource "aws_security_group_rule" "a_from_b" {
  type                     = "ingress"
  security_group_id        = aws_security_group.a.id
  source_security_group_id = aws_security_group.b.id
  from_port                = 443
  to_port                  = 443
  protocol                 = "tcp"
}

resource "aws_security_group_rule" "b_from_a" {
  type                     = "ingress"
  security_group_id        = aws_security_group.b.id
  source_security_group_id = aws_security_group.a.id
  from_port                = 443
  to_port                  = 443
  protocol                 = "tcp"
}

Resource created before its dependency

Solution:

# Utiliser depends_on pour les dépendances implicites
resource "aws_ecs_service" "main" {
  name            = "my-service"
  cluster         = aws_ecs_cluster.main.id
  task_definition = aws_ecs_task_definition.main.arn
  
  # Attendre que le listener soit créé
  depends_on = [aws_lb_listener.main]
}

# Ou créer une dépendance explicite
resource "null_resource" "wait_for_db" {
  depends_on = [aws_db_instance.main]

  provisioner "local-exec" {
    command = "sleep 30"
  }
}

resource "aws_ecs_service" "main" {
  depends_on = [null_resource.wait_for_db]
  # ...
}

---

4. Performance problems

Plan/Apply very slow

# Activer le parallelisme (défaut: 10)
terraform apply -parallelism=20

# Cibler des ressources spécifiques
terraform plan -target=module.vpc
terraform apply -target=aws_instance.specific

# Utiliser des data sources au lieu de remote state
# Plus rapide que terraform_remote_state pour les lookups simples

State too large

# Diviser l'infrastructure en stacks
# Stack 1: Network
# Stack 2: Compute
# Stack 3: Database

# Utiliser data sources pour référencer les autres stacks
data "terraform_remote_state" "network" {
  backend = "s3"
  config = {
    bucket = "my-terraform-state"
    key    = "network/terraform.tfstate"
    region = "eu-west-1"
  }
}

resource "aws_instance" "main" {
  subnet_id = data.terraform_remote_state.network.outputs.private_subnet_ids[0]
}

---

5. Module problems

Module not found

Symptom:

Error: Module not installed

This module is not yet installed. Run "terraform init" to install all modules.

Solutions:

# Réinitialiser les modules
rm -rf .terraform/modules
terraform init

# Vérifier le chemin du module
terraform get -update

# Pour les modules privés GitHub
export GITHUB_TOKEN="your-token"

Module version conflict

# Utiliser des contraintes de version strictes
module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "5.1.2"  # Version exacte
  # ou
  version = "~> 5.0"  # >= 5.0.0, < 6.0.0
  # ou
  version = ">= 5.0.0, < 5.2.0"  # Range spécifique
}

---

6. Advanced Debugging

Enable verbose logs

# Niveau de log
export TF_LOG=TRACE  # TRACE, DEBUG, INFO, WARN, ERROR

# Logger dans un fichier
export TF_LOG_PATH="./terraform.log"

# Logs pour un provider spécifique
export TF_LOG_PROVIDER=TRACE

# Exemple de debug
TF_LOG=DEBUG terraform plan 2>&1 | tee debug.log

Analyze the plan

# Générer un plan lisible
terraform plan -out=tfplan
terraform show -json tfplan | jq '.' > plan.json

# Analyser les changements
jq '.resource_changes[] | select(.change.actions | contains(["delete"]))' plan.json

# Compter les ressources par action
jq '.resource_changes | group_by(.change.actions[0]) | 
    map({action: .[0].change.actions[0], count: length})' plan.json

Dependency graph

# Générer le graphe
terraform graph | dot -Tpng > graph.png

# Graphe simplifié
terraform graph -type=plan | dot -Tsvg > plan.svg

# Utiliser blast-radius pour visualisation interactive
pip install blastradius
blast-radius --serve .

---

7. Emergency recovery

Accidentally deleted resource

# 1. Récupérer le state précédent (S3 versioning)
aws s3api list-object-versions \
    --bucket my-terraform-state \
    --prefix env/terraform.tfstate

# 2. Télécharger la version précédente
aws s3api get-object \
    --bucket my-terraform-state \
    --key env/terraform.tfstate \
    --version-id "VERSION_ID" \
    ./old-state.json

# 3. Extraire l'ID de la ressource
jq '.resources[] | select(.type == "aws_instance" and .name == "main")' old-state.json

# 4. Importer la ressource (si elle existe encore)
terraform import aws_instance.main i-1234567890abcdef0

Full rollback

# 1. Identifier la bonne version du state
aws s3api list-object-versions --bucket my-bucket --prefix terraform.tfstate

# 2. Télécharger cette version
aws s3api get-object \
    --bucket my-bucket \
    --key terraform.tfstate \
    --version-id "GOOD_VERSION" \
    ./terraform.tfstate.rollback

# 3. Vérifier ce que le rollback va faire
terraform plan -state=terraform.tfstate.rollback

# 4. Si OK, pousser le state
terraform state push terraform.tfstate.rollback

# 5. Appliquer pour synchroniser l'infrastructure
terraform apply

---

8. Troubleshooting Checklist

□ Vérifier la version de Terraform (terraform version)
□ Vérifier les credentials (aws sts get-caller-identity)
□ Réinitialiser les modules (terraform init -upgrade)
□ Vérifier les locks dans DynamoDB
□ Examiner les logs (TF_LOG=DEBUG)
□ Vérifier le state (terraform state list)
□ Tester avec -target sur une ressource spécifique
□ Vérifier le graphe de dépendances
□ Consulter les versions précédentes du state

---

Useful commands

# Valider la configuration
terraform validate

# Formater le code
terraform fmt -recursive

# Lister les ressources dans le state
terraform state list

# Voir les détails d'une ressource
terraform state show aws_instance.main

# Déplacer une ressource dans le state
terraform state mv aws_instance.old aws_instance.new

# Taint pour forcer la recréation
terraform taint aws_instance.main

# Untaint
terraform untaint aws_instance.main

# Remplacer une ressource au prochain apply
terraform apply -replace=aws_instance.main

---

← Project 8 | Certification Preparation →