Component Retry

Configure per-component retry behavior so transient failures — provider download 502s, S3 backend timeouts, registry rate limits — recover automatically without manual re-runs.

Terraform, Helmfile, Packer, and Ansible commands often fail with transient infrastructure errors that have nothing to do with the component's own configuration. The most common is a 502 Bad Gateway during terraform init:

Error: Failed to install provider … 502 Bad Gateway returned from
https://github.com/opentofu/terraform-provider-local/releases/...

Configure a retry block on the component and Atmos will retry each subprocess invocation (init, workspace, plan/apply, sync, build, playbook run — whatever the component type shells out to) when the captured output matches one of the regex patterns you list as recoverable.

Supported component types

retry works the same way for every component type that shells out to a binary: Terraform, Helmfile, Packer, and Ansible.

Native Kubernetes and native Helm components do not shell out — they call Go SDKs directly, with no subprocess to capture and pattern-match. retry is not wired up for them yet.

Usage

components:
  terraform:
    vpc:
      retry:
        max_attempts: 5
        backoff_strategy: exponential
        initial_delay: 2s
        max_delay: 30s
        conditions:
          - /Bad Gateway/
          - /GOAWAY/
          - /5\d\d /
          - /connection reset/
          - /TLS handshake timeout/
          - /could not query provider registry/

How it works

When retry.conditions is configured, each call Atmos makes to the component's underlying binary (terraform/tofu, helmfile, packer, or ansible-playbook) is wrapped in an independent retry loop:

1. The subprocess runs as usual; its stdout and stderr stream to the user and are captured into an in-memory buffer.
2. When the subprocess exits with a non-zero status, the captured output is matched against every regex in conditions.
3. If at least one pattern matches, Atmos waits for the configured backoff and retries.
4. If no pattern matches, the error is returned immediately — real failures (terraform plan exit-code 2, schema errors, permission denials) are never silently retried.

Each subprocess invocation has its own retry loop. Retrying terraform init does not consume the apply retry budget — the same isolation applies across any two invocations for any component type.

Provider registry failures

Provider downloads can fail during init in more than one way. The example above shows a 502 Bad Gateway. The registry can also close the HTTP/2 connection with a GOAWAY error:

Error: Failed to resolve provider packages

Could not resolve provider cloudposse/awsutils: could not query provider
registry for registry.opentofu.org/cloudposse/awsutils: http2: server sent
GOAWAY and closed the connection; LastStreamID=5, ErrCode=NO_ERROR, debug=""

Both errors come from a temporary connection problem with the registry. They are not code problems. The conditions list above already matches both errors, through /could not query provider registry/ and /GOAWAY/.

This type of error can fail many components at the same time. A drift-detection run, a bulk apply, or a CI job can query the registry for many components in a short time window. One registry connection problem can then fail the whole batch.

You have four ways to share one retry policy across those components:

1. Configure retry once at the stack-manifest root — see Stack-Level Defaults. This applies to every supported component in the stack. No mixin, no shared base component, no per-component copies.
2. Configure retry on each affected component.
3. Configure retry on one shared abstract base component — see Inheritance. This only works when the components already share a base.
4. Configure retry once in a mixin, using overrides.retry. This works even when the components do not share a base component, and lets you scope the policy to part of a stack instead of all of it.

Any of the four lets each component recover on its own, independently, within its own max_attempts or max_elapsed_time budget. A component recovers only if the registry error clears before its budget runs out. If the outage lasts longer than that, the component still fails even with retry configured — a long enough outage can still fail part or all of the batch. You do not need to start a manual re-run for any component that recovers within its budget.

atmos terraform cache reduces the number of calls Atmos makes to the upstream registry. The cache does not retry a failed call. A cache miss still sends one call to the registry and returns any error from that call. Retry and the cache work together: the cache reduces how often you meet a registry problem, and retry recovers when you do.

Arguments

conditions

A list of regex patterns matched against captured stdout/stderr. Only errors whose output matches at least one pattern are retried. Patterns may be wrapped in /.../ for readability. Without conditions, no retry is attempted — this is a safety default so a typo never silently retries real failures.

max_attempts

Maximum number of attempts, including the first. Omit for unlimited; defaults to 1 (no retry) when retry is unset entirely.

backoff_strategy

One of constant, linear, or exponential. Defaults to exponential when initial_delay is set.

initial_delay

Delay before the second attempt as a Go duration string (e.g. "2s", "500ms").

max_delay

Upper bound on any single delay; caps exponential growth (e.g. "30s").

max_elapsed_time

Total time budget across all attempts (e.g. "5m"). After this the most recent error is returned regardless of max_attempts.

multiplier

Growth multiplier for exponential backoff. Defaults to 2.0.

random_jitter

Fractional jitter applied to each delay (0–1). Useful to avoid thundering-herd retries from CI fleets.

Inheritance

The retry block participates in component inheritance. Abstract components can define a default retry policy that concrete components inherit and override:

components:
  terraform:
    base/network:
      metadata:
        type: abstract
      retry:
        max_attempts: 3
        initial_delay: 2s
        backoff_strategy: exponential
        conditions:
          - /Bad Gateway/
          - /GOAWAY/
          - /5\d\d /

    vpc:
      metadata:
        inherits:
          - base/network
      # vpc inherits the base retry block. `metadata.inherits` pulls in
      # base/network's config; `metadata.component` alone does not — it only
      # selects which Terraform source to use, not which config to inherit.

    transit-gateway:
      metadata:
        inherits:
          - base/network
      retry:
        # Scalars are replaced: max_attempts becomes 5. `conditions` is a
        # list, so it follows settings.list_merge_strategy — by default
        # (replace) this list replaces the base's conditions entirely,
        # leaving only /TLS handshake timeout/. Set
        # `settings.list_merge_strategy: append` to accumulate conditions
        # across layers instead of replacing them.
        max_attempts: 5
        conditions:
          - /TLS handshake timeout/

Stack-Level Defaults

Configure retry once at the root of a stack manifest to apply the same policy to every supported component in that stack. No mixin, no shared base component, no per-component copies:

stacks/deploy/prod.yaml

retry:
  max_attempts: 5
  backoff_strategy: exponential
  initial_delay: 2s
  max_delay: 30s
  conditions:
    - /Bad Gateway/
    - /GOAWAY/
    - /could not query provider registry/

components:
  terraform:
    vpc:
      # ...
    transit-gateway:
      # ...

Precedence, lowest to highest: stack-root retry → abstract base component retry → concrete component retry → overrides.retry. A more specific layer wins on a conflicting key. Non-conflicting keys from lower layers still apply — a component that only sets its own max_attempts still inherits the stack-root conditions list.

Stack-root retry is scoped to the manifest file, and to any file that imports it — the same scoping rule as global vars, metadata, and hooks. Use a mixin instead when the policy should apply to only some components in a stack, not the whole stack.

Opting a component out

retry merges as a deep merge, like every other section. An empty override does not turn retry off:

components:
  terraform:
    vpc:
      retry: {}
      # This does NOT disable an inherited retry policy. An empty map
      # contributes no keys to the merge, so the component still gets the
      # full stack-root or base-component retry policy unchanged.

To opt a single component out of an inherited policy, set max_attempts: 1 on that component. A budget of one attempt means the first failure is returned immediately, regardless of conditions:

components:
  terraform:
    vpc:
      retry:
        max_attempts: 1

When NOT to use retry

• Real Terraform failures. A plan that exits with code 2 because of a configuration error or a validate that catches a typo should fail loudly. Keep conditions narrowly scoped to transient infra patterns.
• Stateful mutations with non-idempotent side effects. If a third-party API is invoked from a local-exec provisioner and is not safe to retry, do not configure retry conditions that could match its error output.
• Hooks. Atmos hooks have their own configuration and are not affected by component retry.

See also

• Workflow retry — retry configuration for multi-step workflows.
• Vendor retry — retry configuration for component vendoring.
• Terraform Registry Cache — reduces calls to the registry. It does not retry on its own.
• Mixins — share a retry policy across components that do not have a common base component.