Terraform State and Backends

By Michael Samuel Naeem · December 14, 2025 · 5 min read

Terraform state is a JSON file that knows your RDS instance is db-ABC123 and that the security group depends on the VPC. Lose it, corrupt it, or let two applies run concurrently, and you are in for an afternoon of terraform import and prayer. Remote backends with locking are table stakes; the harder problems are partitioning, secrets in state, and refactoring resources without destroy-and-recreate.

Local vs remote state

Local terraform.tfstate on a laptop works for solo experiments. It fails for teams:

• No locking → concurrent applies corrupt state
• Not shared → teammates recreate resources
• No versioning → no rollback

Remote backend minimum:

terraform {
  backend "s3" {
    bucket         = "org-terraform-state"
    key            = "prod/network/terraform.tfstate"
    region         = "eu-west-1"
    dynamodb_table = "terraform-locks"
    encrypt        = true
  }
}

Enable S3 versioning on the bucket — previous state versions are your undo button.

Locking mechanics

DynamoDB lock item prevents parallel apply/plan-with-lock. Stuck lock after CI crash:

terraform force-unlock LOCK_ID

Only after confirming no running apply. Force-unlock during active apply causes state corruption.

State partitioning strategy

Anti-pattern: one state for entire prod account — 45-minute plans, everyone blocked.

Cross-stack references:

data "terraform_remote_state" "network" {
  backend = "s3"
  config = {
    bucket = "org-terraform-state"
    key    = "prod/network/terraform.tfstate"
    region = "eu-west-1"
  }
}

subnet_ids = data.terraform_remote_state.network.outputs.private_subnet_ids

Terraform 1.5+ terraform stack and Cloud stacks evolve this pattern — same principle: explicit outputs, no duplicate resources.

Sensitive values in state

State contains resource attributes, including secrets marked sensitive in plan output. Treat state bucket as highly confidential:

• Encryption at rest (S3 SSE-KMS)
• IAM least privilege
• No public access
• Audit logging

Prefer referencing Secrets Manager/SSM rather than storing secret strings in Terraform resources when possible. If password must flow through Terraform, rotate after apply.

Import and adoption

Bring existing infrastructure under management:

terraform import aws_instance.web i-0abc123def456

Write matching .tf first, then import. Plan should show no changes after successful import.

Bulk adoption: tools like terraformer generate rough .tf from scan — always refactor into modules afterward.

Refactoring without destroy — moved and import blocks

Terraform 1.1+ moved block retargets state when renaming resources:

moved {
  from = aws_security_group.old
  to   = aws_security_group.web
}

import block (1.5+) declarative import in config:

import {
  to = aws_instance.web
  id = "i-0abc123"
}

Avoid manual terraform state mv when declarative alternatives exist — they are reviewable in PR.

Recovery playbook

Corrupted state: restore previous S3 version.

Accidental destroy: restore state version; re-apply if resources still exist, or recreate from module.

Resource drifted outside state: import or remove from config.

Splitting state: terraform state pull, carefully edit JSON (expert only), or use terraform state mv -state-out to new backend.

Always backup before manual state surgery.

CI integration

• terraform init -backend-config=...
• terraform plan on PR (post comment)
• terraform apply on merge to main with approval gate
• State access via OIDC role, not long-lived keys

State access control and auditing

Limit who can read prod state — it contains secrets and infrastructure topology useful to attackers. Use IAM policies on S3 bucket and DynamoDB lock table. Enable CloudTrail data events on state bucket. Terraform Cloud adds RBAC and audit log of who ran apply — worth it for regulated industries even if S3 backend suffices technically.

Workspace vs directory layout

# environments/prod/backend.tf
terraform {
  backend "s3" {
    bucket         = "tf-state-prod"
    key            = "network/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "tf-locks-prod"
    encrypt        = true
  }
}

Never share state files between environments — staging terraform destroy must not touch production resources.

State size management

Large states (>10 MB) slow every plan:

• Split stacks by lifecycle (network rarely changes, apps change weekly)
• Remove resources from state with terraform state rm after migration to new stack
• Use -target sparingly — it doesn't reduce state size, only plan scope

Monitor terraform plan duration — sudden 10× increase often means someone imported thousands of resources into one state file.

Pair with Terraform modules composition when splitting state along module boundaries.

Common production mistakes

Teams get state management backends wrong in predictable ways:

• Skipping failure-mode rehearsal — run a game day or fault injection exercise before peak traffic, not after the first outage.
• Missing correlation context — every error path should carry request, trace, or tenant identifiers so incidents are debuggable.
• Optimizing for demo, not steady state — load tests, cache warm-up, and cold-start paths matter more than local dev latency.
• Undocumented trade-offs — if you chose speed over strict correctness (or vice versa), write that down for the next engineer.

Terraform patterns for state management backends rot when emergency console edits never get codified, ignore_changes blocks multiply without documentation, and drift detection runs monthly instead of daily on production workspaces.

Debugging and triage workflow

When state management backends misbehaves in production, work top-down instead of guessing:

1. Confirm scope — one tenant, region, or deployment stage? Narrow blast radius before deep diving.
2. Check recent changes — deploys, flag flips, config pushes, and schema migrations in the last 24 hours.
3. Compare golden signals — latency, error rate, saturation, and traffic for the affected surface vs. baseline.
4. Reproduce minimally — smallest input or scenario that triggers the failure; capture traces/logs with correlation IDs.
5. Fix forward or rollback — if rollback is faster than root-cause during incident, rollback first, postmortem second.
6. Add a guard — alert, integration test, or circuit breaker so the same class of failure is caught earlier next time.

Document the timeline during triage. Future you (and on-call) will need timestamps, not just conclusions.

Resources

• Terraform backend types
• S3 backend documentation
• State locking
• moved block reference
• Import block (Terraform 1.5+)