Skip to main content

Terraform state migrations with tfmigrate

· 5 min read
Erik Osterman
Erik Osterman
Founder @ Cloud Posse

Refactoring Terraform code leaves state behind. Rename a resource, or move it between root modules, and every plan shows a destroy-and-recreate for infrastructure that never changed. You can fix this by hand with terraform state mv, but that command only fixes one workspace at a time. It is easy to get wrong, and no one can review it before it runs.

tfmigrate turns these state changes into migration files that you store under version control. Atmos runs these files for you — manually from the CLI, or automatically from Terraform lifecycle hooks — in the same component context as atmos terraform plan and apply.

atmos terraform migrate plan s3-bucket -s plat-ue2-dev
atmos terraform migrate apply s3-bucket -s plat-ue2-dev
atmos terraform migrate
00:00.0 / 00:00.0

What Changed​

The new atmos terraform migrate command family adds:

  • atmos terraform migrate plan to preview a migration.
  • atmos terraform migrate apply to apply a migration.
  • atmos terraform migrate list to inspect per-component hook and history context.
  • kind: tfmigrate hooks for running migrations from Terraform lifecycle events.
  • Zero-config history storage that reuses the component's Terraform backend.

Before tfmigrate runs, Atmos performs the normal Terraform component setup. This includes auth identity resolution, source provisioning, workdir provisioning, generated backend and varfiles, Terraform init, and workspace selection.

Dynamic Hooks​

kind: tfmigrate hooks default to mode: dynamic, so automation follows the Terraform operation:

hooks:
  state-migration:
    events:
      - before.terraform.plan
      - before.terraform.apply
    kind: tfmigrate
    mode: dynamic

before.terraform.plan runs tfmigrate plan. before.terraform.apply and before.terraform.deploy run tfmigrate apply. Use static mode: plan or mode: apply when a hook must always run one action.

Hooks run through Atmos, so they use the same identity as the Terraform operation. If the Terraform command or component selects an Atmos auth identity, the migration gets that same authenticated environment.

Recovering From Skipped Releases​

Provider removals are where this bites hardest. When a component release drops a provider configuration, every workspace that skips the intermediate release gets stuck. The state still holds resources from the removed provider, and Terraform refuses to plan:

Error: Provider configuration not present

To work with random_pet.legacy (orphan) its original provider configuration
at provider["registry.opentofu.org/hashicorp/random"].legacy is required,
but it has been removed.

Until now, the only fix was manual, and per workspace: restore a temporary provider override, apply, then delete the override again. Ship a migration alongside the release instead. The hook prunes the stale state entries before Terraform loads provider configurations, so workspaces can jump straight to the newest release:

migration "state" "drop_legacy_provider_state" {
  actions = [
    "rm random_pet.legacy",
  ]
}

Note: state rm abandons the remote object. It does not destroy it. This is usually what you want for provider-cleanup migrations. If you need to destroy the object, destroy it before you upgrade, or temporarily restore the provider configuration.

History Mode​

For idempotent migrations in automation, use tfmigrate history mode:

atmos terraform migrate apply s3-bucket -s plat-ue2-dev

History mode needs no configuration by default. Atmos looks for a custom tfmigrate config in four places: the hook's config field, the --tfmigrate-config / ATMOS_TFMIGRATE_CONFIG flag or environment variable, the TFMIGRATE_CONFIG environment variable, and a .tfmigrate.hcl file in the component. If none of these exist, Atmos generates a config on the fly.

The generated config stores migration history in the component's own Terraform backend. For an S3 or GCS backend, Atmos reuses the same bucket as the state. It stores history under a key namespaced by stack, component, and workspace, and it inherits the region, role ARN, and endpoint. For a local backend, Atmos stores the history file beside the state file.

Atmos records every applied migration and never reruns it. You don't need to set anything up.

To take control, provide your own config. Drop a .tfmigrate.hcl file in the component, set the hook's config field, or pass --tfmigrate-config / ATMOS_TFMIGRATE_CONFIG. Atmos exports stack, component, and workspace-scoped history variables, and copies the supported Terraform backend settings. Your custom config can then reuse the same bucket and identity setup:

tfmigrate {
  migration_dir = "./tfmigrate"

  history {
    storage "s3" {
      bucket   = env.ATMOS_TFMIGRATE_HISTORY_BUCKET
      key      = env.ATMOS_TFMIGRATE_HISTORY_KEY
      region   = env.ATMOS_TFMIGRATE_HISTORY_REGION
      role_arn = env.ATMOS_TFMIGRATE_HISTORY_ROLE_ARN
    }
  }
}

The default history key is:

tfmigrate/<stack>/<component>/<workspace>/history.json

That keeps multiple Atmos component instances from colliding when they share a Terraform backend bucket.

Important Limitation​

Single-file tfmigrate apply path.hcl is not idempotent by itself. A rerun can fail if a state address already moved or was removed. Prefer history mode instead. When the component's backend is S3 or GCS, the generated default gives you durable storage automatically. With a purely local backend, make sure the local history file survives between runs — for example, have your CI workflow persist it.

Learn More​