LabHub

Blog

Terraform State Management and Module Design Practical Guide: Remote Backend, State Locking, Module Patterns, and Drift Detection

2026-03-11 · 17 min read · #devops#terraform#iac#state-management#module-design#2026-03

한국어English日本語

Terraform State Management

Introduction

Among the IaC (Infrastructure as Code) tools for managing infrastructure as code, Terraform is the most widely used. When Terraform is operated in a production environment, however, the most complex and important area is state management. If the state file is corrupted or a conflict occurs, the entire infrastructure operation can be paralyzed, and a poor module design increases maintenance cost exponentially.

This article covers everything needed in a production environment in depth, from the internal structure of the Terraform state file through remote backend configuration, the state locking mechanism, module design patterns, drift detection strategies, and real failure cases with their recovery procedures. Recent changes such as the S3 native locking introduced in Terraform 1.10+ are reflected as well.

Terraform State Architecture

The Role of the State File

Terraform records the state of the infrastructure resources it currently manages in a JSON file called terraform.tfstate. This file serves the following core roles.

State File Structure

The internal structure of the state file is as follows.

{
  "version": 4,
  "terraform_version": "1.10.3",
  "serial": 42,
  "lineage": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "outputs": {
    "vpc_id": {
      "value": "vpc-0abc123def456789",
      "type": "string"
    }
  },
  "resources": [
    {
      "mode": "managed",
      "type": "aws_vpc",
      "name": "main",
      "provider": "provider[\"registry.terraform.io/hashicorp/aws\"]",
      "instances": [
        {
          "schema_version": 1,
          "attributes": {
            "id": "vpc-0abc123def456789",
            "cidr_block": "10.0.0.0/16",
            "tags": {
              "Name": "production-vpc"
            }
          }
        }
      ]
    }
  ]
}

How Plan Works

terraform plan compares three pieces of information to produce an execution plan.

  1. Configuration files (.tf): the desired state the user has defined
  2. State file (.tfstate): the last applied state, the known state
  3. Actual infrastructure: the current state queried from the cloud provider, the actual state

Terraform first queries the actual infrastructure (refresh) to update the state file, then compares the updated state with the configuration files to produce the change plan.

Remote Backend Configuration

Limits of the Local Backend

By default Terraform stores the state file on the local filesystem. That is fine for a personal project, but in a team environment it has the following serious limits.

S3 + DynamoDB Backend Configuration

This is the most common configuration in an AWS environment. S3 is used as the state file store and DynamoDB for state locking.

First, bootstrap the backend infrastructure.

# backend-bootstrap/main.tf
resource "aws_s3_bucket" "terraform_state" {
  bucket = "my-company-terraform-state"

  lifecycle {
    prevent_destroy = true
  }
}

resource "aws_s3_bucket_versioning" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id
  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm     = "aws:kms"
      kms_master_key_id = aws_kms_key.terraform_state.arn
    }
  }
}

resource "aws_s3_bucket_public_access_block" "terraform_state" {
  bucket                  = aws_s3_bucket.terraform_state.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

resource "aws_dynamodb_table" "terraform_locks" {
  name         = "terraform-state-locks"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "LockID"

  attribute {
    name = "LockID"
    type = "S"
  }
}

resource "aws_kms_key" "terraform_state" {
  description             = "KMS key for Terraform state encryption"
  deletion_window_in_days = 30
  enable_key_rotation     = true
}

After that, configure the backend in the project.

# terraform block
terraform {
  required_version = ">= 1.10.0"

  backend "s3" {
    bucket         = "my-company-terraform-state"
    key            = "production/network/terraform.tfstate"
    region         = "ap-northeast-2"
    encrypt        = true
    kms_key_id     = "arn:aws:kms:ap-northeast-2:123456789012:key/abcd-1234"
    dynamodb_table = "terraform-state-locks"
  }
}

S3 Native Locking (Terraform 1.10+)

From Terraform 1.10.0 onward, S3 native state locking was introduced and DynamoDB is no longer required. It makes use of the conditional write feature of S3.

terraform {
  backend "s3" {
    bucket       = "my-company-terraform-state"
    key          = "production/network/terraform.tfstate"
    region       = "ap-northeast-2"
    encrypt      = true
    use_lockfile = true
  }
}

The use_lockfile = true setting alone manages locking directly in S3 without DynamoDB. To migrate from existing DynamoDB-based locking, remove the dynamodb_table setting, add use_lockfile = true, and then run terraform init -migrate-state.

GCS Backend Configuration

On GCP, Google Cloud Storage is used. The GCS backend has locking built in, so no separate lock table is needed.

terraform {
  backend "gcs" {
    bucket = "my-company-terraform-state"
    prefix = "production/network"
  }
}

Terraform Cloud / HCP Terraform

A managed service from HashiCorp that provides not only state management but also an execution environment, policy management, audit logs and more.

terraform {
  cloud {
    organization = "my-company"

    workspaces {
      name = "production-network"
    }
  }
}

State Locking and Concurrency Control

Why a Locking Mechanism Is Needed

If several users or CI/CD pipelines run terraform apply at the same time, state file conflicts occur, which can lead to infrastructure inconsistency, duplicate resource creation, and in the worst case state file corruption. State locking is the core mechanism that prevents this.

How DynamoDB-based Locking Works

  1. terraform plan or terraform apply runs
  2. Terraform creates a lock item in the DynamoDB table (LockID = the state file path)
  3. Once the lock is acquired, the operation proceeds
  4. If another user tries to work on the same state, a lock conflict error occurs
  5. The lock is released once the operation finishes

Resolving Lock Conflicts

Attempting an operation while a lock is held produces the following error.

Error: Error acquiring the state lock
Lock Info:
  ID:        a1b2c3d4-e5f6-7890
  Path:      my-company-terraform-state/production/network/terraform.tfstate
  Operation: OperationTypeApply
  Who:       user@hostname
  Version:   1.10.3
  Created:   2026-03-11 09:15:30.123456 +0000 UTC

Under normal circumstances you have to wait until the operation completes. If a lock has been left behind abnormally (a process crash, a network disconnect and so on), force-release it.

# Force-release the lock (only after confirming no other operation is running)
terraform force-unlock a1b2c3d4-e5f6-7890

Caution: force-releasing while another user is actually running apply can corrupt the state file, so it must only be run after confirming with the owner of that lock.

Module Design Patterns

The Standard Module Structure

The standard module structure HashiCorp recommends is as follows.

modules/
  vpc/
    main.tf          # Resource definitions
    variables.tf     # Input variables
    outputs.tf       # Output values
    versions.tf      # Provider and Terraform version constraints
    README.md        # Module usage documentation
    examples/
      simple/
        main.tf
      complete/
        main.tf
    tests/
      vpc_test.go    # Terratest-based tests

The Composition Pattern

A pattern that composes small unit modules into higher-level infrastructure. Each module follows the single responsibility principle, and dependencies are injected as arguments.

# environments/production/main.tf
module "vpc" {
  source  = "../../modules/vpc"
  name    = "production"
  cidr    = "10.0.0.0/16"
  azs     = ["ap-northeast-2a", "ap-northeast-2b", "ap-northeast-2c"]
}

module "security_groups" {
  source = "../../modules/security-groups"
  vpc_id = module.vpc.vpc_id
  environment = "production"
}

module "eks" {
  source             = "../../modules/eks"
  cluster_name       = "production-cluster"
  vpc_id             = module.vpc.vpc_id
  subnet_ids         = module.vpc.private_subnet_ids
  security_group_ids = [module.security_groups.eks_sg_id]
}

module "rds" {
  source             = "../../modules/rds"
  identifier         = "production-db"
  vpc_id             = module.vpc.vpc_id
  subnet_ids         = module.vpc.database_subnet_ids
  security_group_ids = [module.security_groups.rds_sg_id]
}

The Facade Pattern

A pattern that hides a complex combination of internal modules behind a simple interface. It gives the consumer a concise API while encapsulating the internal complexity.

# modules/web-application/main.tf
# Internally composes the VPC, ALB, ECS and RDS modules
module "vpc" {
  source = "../vpc"
  cidr   = var.vpc_cidr
}

module "alb" {
  source    = "../alb"
  vpc_id    = module.vpc.vpc_id
  subnet_ids = module.vpc.public_subnet_ids
}

module "ecs" {
  source       = "../ecs"
  cluster_name = var.app_name
  vpc_id       = module.vpc.vpc_id
  alb_arn      = module.alb.alb_arn
}

# The consumer uses it simply
# environments/production/main.tf
module "web_app" {
  source   = "../../modules/web-application"
  app_name = "my-web-app"
  vpc_cidr = "10.0.0.0/16"
}

Version Management Through a Registry

A private registry makes module version management and sharing across teams straightforward.

module "vpc" {
  source  = "app.terraform.io/my-company/vpc/aws"
  version = "~> 3.2.0"

  name = "production"
  cidr = "10.0.0.0/16"
}

The version constraint patterns are as follows.

Drift Detection and Remediation

What Is Drift

Drift means a state in which the actual infrastructure Terraform manages does not match the state file. The main causes are as follows.

Detection Through terraform plan

The most basic drift detection method is to run terraform plan regularly.

# Refresh the state only to check for drift (no infrastructure change)
terraform plan -refresh-only

# Check the changes with detailed output
terraform plan -refresh-only -detailed-exitcode
# Exit codes: 0 = no change, 1 = error, 2 = drift detected

Integrating Drift Detection into the CI/CD Pipeline

#!/bin/bash
# drift-detection.sh
set -euo pipefail

SLACK_WEBHOOK_URL="https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXX"

echo "Running drift detection..."
terraform init -backend=true -input=false

# -detailed-exitcode: exit code 2 means changes exist
if terraform plan -refresh-only -detailed-exitcode -input=false > plan_output.txt 2>&1; then
  echo "No drift detected."
  exit 0
fi

EXIT_CODE=$?

if [ "$EXIT_CODE" -eq 2 ]; then
  echo "Drift detected! Sending notification..."
  DRIFT_SUMMARY=$(grep -E "^  #|^  ~|^  -|^  \+" plan_output.txt | head -20)

  curl -X POST "$SLACK_WEBHOOK_URL" \
    -H 'Content-Type: application/json' \
    -d "{\"text\": \"Drift detected in production infrastructure:\n\`\`\`\n${DRIFT_SUMMARY}\n\`\`\`\"}"
  exit 2
else
  echo "Error running terraform plan"
  exit 1
fi

Drift Remediation Strategies

There are broadly three ways to respond once drift is found.

  1. Revert with Terraform: run terraform apply to return the infrastructure to the state defined in the code
  2. Reflect it in the code: if the manual change was intended, update the HCL code to reflect the current infrastructure state
  3. Refresh the state file: run terraform apply -refresh-only to update only the state file to match the current infrastructure

State Migration Strategies

Migrating from Local to a Remote Backend

# 1. After adding the backend configuration
terraform init -migrate-state

# 2. Enter yes at the migration confirmation prompt
# 3. Delete the local state files
rm terraform.tfstate terraform.tfstate.backup

Using terraform state mv

Used when renaming a resource or refactoring modules.

# Rename a resource
terraform state mv aws_instance.old_name aws_instance.new_name

# Move into a module
terraform state mv aws_vpc.main module.network.aws_vpc.main

# Move to a different state file
terraform state mv -state-out=other.tfstate aws_s3_bucket.data aws_s3_bucket.data

Importing Existing Resources with terraform import

# Bring an existing resource under Terraform management
terraform import aws_instance.web i-1234567890abcdef0

# Import a resource inside a module
terraform import module.vpc.aws_vpc.main vpc-0abc123def456789

In Terraform 1.5+, a declarative import using the import block is also possible.

import {
  to = aws_instance.web
  id = "i-1234567890abcdef0"
}

Refactoring with the moved Block (Terraform 1.1+)

moved {
  from = aws_instance.old_name
  to   = aws_instance.new_name
}

moved {
  from = aws_vpc.main
  to   = module.network.aws_vpc.main
}

Unlike terraform state mv, the moved block leaves the refactoring history declaratively in the code, and the state is migrated automatically when a teammate runs terraform plan.

Comparative Analysis

Backend Comparison Table

AspectS3 + DynamoDBS3 native (1.10+)GCSTerraform Cloud
Locking methodDynamoDB tableS3 conditional writeGCS object lockingBuilt in
Extra infraS3 bucket + DynamoDB tableS3 bucket onlyGCS bucket onlyNone (SaaS)
EncryptionSSE-S3/SSE-KMSSSE-S3/SSE-KMSGoogle-managed key/CMEKHashiCorp Vault
VersioningS3 versioningS3 versioningGCS object versionsAutomatic
Access controlIAM policyIAM policyIAM policyTeam/org based RBAC
CostS3 + DynamoDB costS3 cost onlyGCS costFree tier + paid
Setup complexityMediumLowLowVery low
Multi-cloudAWS onlyAWS onlyGCP onlyCloud agnostic

Module Design Pattern Comparison

PatternWhere it fitsStrengthsWeaknesses
Flat (single composition)Small projects, prototypesSimple, quick to startNot reusable, code duplication
CompositionMid to large projects, team collaborationReusability, easy to testUp-front design cost
FacadeComplex infrastructure, self-service platformsEase of use, consistencyLess flexibility, cost of keeping the abstraction
RegistryLarge organizations, multiple teamsGovernance, version managementOperational overhead

Operational Considerations

The State File Contains Sensitive Information

A Terraform state file can store sensitive information such as database passwords, API keys and certificates in plain text. The following must be observed.

output "database_password" {
  value     = aws_db_instance.main.password
  sensitive = true
}

State File Separation Strategy

In large infrastructure, managing every resource in a single state file causes the following problems.

The recommended separation structure is as follows.

environments/
  production/
    network/          # VPC, subnets, NAT GW (infra team)
    security/         # IAM, KMS, Security Group (security team)
    database/         # RDS, ElastiCache (DBA team)
    application/      # ECS, Lambda, API GW (dev team)
    monitoring/       # CloudWatch, Datadog (SRE team)
  staging/
    ...

Each directory has its own independent state file and references the output values of another state through the terraform_remote_state data source.

data "terraform_remote_state" "network" {
  backend = "s3"
  config = {
    bucket = "my-company-terraform-state"
    key    = "production/network/terraform.tfstate"
    region = "ap-northeast-2"
  }
}

resource "aws_instance" "web" {
  subnet_id = data.terraform_remote_state.network.outputs.private_subnet_ids[0]
}

Automatic terraform plan Runs Need Care

Running terraform plan automatically in CI/CD is recommended, but running terraform apply -auto-approve automatically is very dangerous in a production environment. An approval process is essential.

Failure Cases and Recovery Procedures

Case 1: State File Corruption

The most common cause of state file corruption is terraform apply being interrupted during a network failure.

Symptom: a JSON parse error occurs when terraform plan runs.

Recovery procedure:

# 1. Restore a previous version from S3 versioning
aws s3api list-object-versions \
  --bucket my-company-terraform-state \
  --prefix production/network/terraform.tfstate

# 2. Download a healthy previous version
aws s3api get-object \
  --bucket my-company-terraform-state \
  --key production/network/terraform.tfstate \
  --version-id "VERSION_ID_HERE" \
  restored-state.tfstate

# 3. Verify the state file
terraform show restored-state.tfstate

# 4. Upload the restored state file
aws s3 cp restored-state.tfstate \
  s3://my-company-terraform-state/production/network/terraform.tfstate

# 5. Sync to the latest state with refresh
terraform apply -refresh-only

Case 2: The Lock Is Never Released (Stuck Lock)

A case where a CI/CD pipeline terminated abnormally and left the lock behind.

Symptom: the Error acquiring the state lock error keeps occurring.

Recovery procedure:

# 1. Confirm whether another operation is really running
# Query the lock item in DynamoDB
aws dynamodb get-item \
  --table-name terraform-state-locks \
  --key '{"LockID": {"S": "my-company-terraform-state/production/network/terraform.tfstate"}}'

# 2. Force-release after checking the lock owner and time
terraform force-unlock LOCK_ID_HERE

# 3. Verify state integrity
terraform plan

Case 3: State Conflict (Serial Mismatch)

A case where two users ran apply almost simultaneously, leaving the serial numbers out of sync.

Symptom: the Error saving state: serial number mismatch error occurs.

Recovery procedure:

# 1. Download the current remote state
terraform state pull > remote-state.json

# 2. Check the serial number
python3 -c "import json; print(json.load(open('remote-state.json'))['serial'])"

# 3. Sync the state with refresh
terraform apply -refresh-only

# 4. Re-apply the changes
terraform plan
terraform apply

Production Checklist

This section collects the items that must be checked when adopting Terraform in production.

Remote backend configuration

State Locking

Module management

Drift Detection

Operational process

References

Conclusion

Terraform state management is the foundation of IaC operations. If the state file is not managed correctly, even the best-written HCL code cannot run reliably in a production environment. The content covered in this article can be summarized as follows.

First, a remote backend is essential. A local state file cannot be used in a team environment; state has to be managed safely through S3, GCS or Terraform Cloud. On Terraform 1.10 or later, S3 native locking can be used to remove the DynamoDB dependency.

Second, state locking is the core of concurrency control. Without locking, state file corruption is inevitable when several users apply at the same time. Force-releasing a lock must only be done after confirming that no operation is currently running.

Third, module design is the foundation of reusability and maintainability. Composing small modules with the composition pattern and managing versions through a registry keeps infrastructure consistent even in a large organization.

Fourth, drift detection has to run continuously. Manual changes can happen at any time, and if they are not detected early and reflected in the code, the gap between Terraform and the actual infrastructure accumulates until Terraform itself becomes unusable.

Finally, recovery procedures have to be verified in advance on the premise that failures will definitely happen. Restoring the state file through S3 versioning, force-releasing a lock, and resolving a serial conflict are procedures the whole team has to know in order to respond quickly during a real incident.

Building solid Terraform state management creates the foundation for making infrastructure changes with confidence. That is the core precondition for the fast, safe deployment that DevOps culture aims at.

Read the original: https://www.youngju.dev/blog/devops/2026-03-11-terraform-state-management-module-design-best-practices.en

Comments

No comments yet.

Sign in to leave a comment