Retry Policies

Use error events to retry individual component invocations. on selects the operation, rule selects the failures, and with configures the retry budget and backoff.

policy.d/retries.yaml

apiVersion: atmos/v1
kind: Policy
metadata:
  name: retries
spec:
  policies:
    retry-terraform-network-errors:
      description: >-
        Retry initialization, planning, and output retrieval when
        Terraform encounters transient network failures.
      on:
        - terraform.init.error
        - terraform.plan.error
        - terraform.output.error
      action: retry
      rule: input.output.matches('(?i)connection reset|TLS handshake timeout')
      with:
        max_attempts: 3
        initial_delay: 2s

    retry-packer-downloads:
      description: >-
        Retry Packer builds when an upstream download temporarily fails.
      on: packer.build.error
      action: retry
      rule: input.output.matches('(?i)bad gateway|connection reset')
      with:
        max_attempts: 3
        initial_delay: 5s

    retry-kubernetes-unavailable:
      description: >-
        Retry Kubernetes apply operations when the API server
        is temporarily unavailable.
      on: kubernetes.apply.error
      action: retry
      rule: input.error.message.matches('(?i)service unavailable')
      with:
        max_attempts: 3
        initial_delay: 2s

    retry-terraform-throttling:
      description: >-
        Retry any Terraform subcommand that fails because a service
        throttled its requests.
      on: terraform.*.error
      action: retry
      rule: input.output.matches('(?i)throttling|rate exceeded')
      with:
        max_attempts: 3
        initial_delay: 5s

Policies belong in the governing policy directory. They can use the existing match selector to restrict retries to component tags and labels.

Events

An event names the component type and the actual operation that failed:

Component type	Example events
Terraform and OpenTofu	terraform.init.error, terraform.workspace.select.error, terraform.validate.error, terraform.output.error, terraform.plan.error
Helmfile	helmfile.diff.error, helmfile.apply.error
Packer	packer.init.error, packer.build.error
Ansible	ansible.playbook.error
Native Helm	helm.apply.error, helm.diff.error, helm.delete.error, helm.template.error
Kubernetes	kubernetes.apply.error, kubernetes.diff.error, kubernetes.delete.error, kubernetes.validate.error
CloudFormation	aws/cloudformation.apply.error, aws/cloudformation.diff.error, aws/cloudformation.delete.error
Containers	container.build.error, container.up.error, container.ps.error
Emulators	emulator.up.error, emulator.reset.error, emulator.ps.error

Subprocess events apply to every command routed through the shared executor. An init failure during atmos terraform plan emits terraform.init.error. Nested Terraform commands use dots, such as terraform.state.list.error. OpenTofu uses Terraform events. Output retrieval through !terraform.output and atmos.Component() also emits separate init, workspace, and output events, including initialization with init.pass_vars.

on accepts one event or a nonempty list. A component wildcard, such as terraform.*.error, selects all of its subcommands, including nested commands. Component wildcards and partial command globs are not supported. Overlapping selectors in one rule evaluate that rule once.

Aliases use canonical lifecycle names: Terraform deploy becomes apply; Helmfile sync and deploy become apply; native Helm, Kubernetes, and CloudFormation deploy becomes apply and plan becomes diff. Native Helm also maps render to template and destroy to delete. Selectors accept these aliases too.

Failure input

Field	Meaning
input.event	Canonical event name
input.component_type, input.command	Component type and canonical command
input.component, input.stack	Component instance identity, when available
input.stage	Subprocess phase, SDK method, or HTTP method that failed
input.attempt	Failed attempt number, starting at 1
input.exit_code	Subprocess exit code; 0 when unavailable
input.stdout, input.stderr	Output captured from this attempt, when available
input.output	Combined captured output; empty for SDK calls without output
input.error.message	Error message from this attempt
input.error.code	SDK error code when supported; HTTP status for native Helm requests; otherwise empty

Use input.output for subprocess diagnostics and input.error.message or input.error.code for SDK failures. Each attempt receives fresh output. Successful invocations, including Terraform plan's detailed exit code 2, do not emit error events.

Retry settings and precedence

with.max_attempts is required and includes the first attempt. The other settings are initial_delay, max_delay, max_elapsed_time, backoff_strategy (constant, linear, or exponential), multiplier, and random_jitter. They use the existing retry settings. Use rule for conditions; with.conditions is rejected.

All applicable rules are evaluated after a failure. A matching deny blocks further attempts. Otherwise, the first matching retry policy in filename and rule-name order supplies the budget. Its predicate must continue to match subsequent failures; Atmos does not switch to another policy or restart a budget partway through a retry sequence. Policy evaluation errors and cancellation stop execution.

For subprocess components, the existing component retry configuration remains the fallback when no retry policy matches the first failure. Its conditions are still required. A selected retry policy replaces that invocation's component retry budget.

SDK execution boundaries

Kubernetes and CloudFormation retry individual SDK calls. Native Helm retries individual replayable Kubernetes HTTP requests and chart lookup/download calls. Containers and emulators retry individual runtime calls. A failed polling request does not rerun an already successful create, apply, or install operation. Each call has its own budget. Native Helm does not retry streaming request bodies that cannot be replayed.

An error can still follow a mutation accepted by a remote service. Choose predicates appropriate for that operation's retry semantics. These policies do not make remote operations transactional and do not perform rollback.

Custom component providers can use pkg/policy/recovery.New with their registered component type and actual command, then recovery.Call or recovery.Do around individual SDK calls. Runner.Run accepts captured subprocess output. The same event selectors, policy context, and budget handling apply without extending a central event enum.