Testing Automation

A test step groups checks and reports their results. Use it to test script logic, command behavior, or deployed services in workflows, custom commands, and hooks.

Each check can be a script, an HTTP request, a shell command, or another supported step. A successful script passes; call fail() when an expectation is not met.

Test syntax

Add a type: test step to a workflow or custom command's steps list:

- name: checks
  type: test
  title: Configuration checks
  steps:
    - name: required-files
      type: script
      interpreter: starlark
      script: |
        if not fs.exists("atmos.yaml"):
            fail("atmos.yaml is missing")

title labels the group; each child's name identifies a check in the results. For a lifecycle hook, select kind: step, type: test, and put the test's title and steps under with. Set on_failure: fail on the hook to make failed checks fail the parent operation or CI job; hooks default to on_failure: warn. See hook syntax.

Read results and use them in CI

Successful logs stay hidden by default. Failed checks display their buffered output and errors; set output: all on the test step to include successful logs. Interactive terminals show a live tree, while CI and redirected output use static results.

The default failure policy lets independent checks finish, then fails the group if unhandled failures remain. Nest parallel or matrix groups when checks can run concurrently or need several input combinations. Keep the default failing policy for suites intended to gate CI.

Test steps also work inside custom commands and lifecycle hooks. See the test step reference for failure policies, supported children, counts, and hook configuration. The language overview describes output values and process execution; the test example shows a larger suite.

Example: test a workflow

With Atmos installed and on your PATH, save this as atmos.yaml. The first example uses only built-in functions and needs no external tools:

atmos.yaml

base_path: .
workflows:
  base_path: workflows

Save this complete workflow as workflows/checks.yaml. It produces a structured result, then checks both that result and a reusable calculation:

workflows/checks.yaml

workflows:
  check:
    steps:
      - name: calculate
        type: script
        interpreter: starlark
        script: |
          output = {"replicas": 3, "workers": 12}
      - name: checks
        type: test
        title: Capacity checks
        steps:
          - name: result
            type: script
            interpreter: starlark
            env:
              RESULT: '{{ .steps.calculate.value }}'
            script: |
              actual = json.decode(env["RESULT"])
              if actual["workers"] != actual["replicas"] * 4:
                  fail("worker count must equal replicas times four")
          - name: calculation
            type: script
            interpreter: starlark
            script: |
              def capacity(replicas):
                  return replicas * 4
              for replicas, expected in [(1, 4), (2, 8), (3, 12)]:
                  actual = capacity(replicas)
                  if actual != expected:
                      fail("expected {}, got {}".format(expected, actual))

Run from the directory containing atmos.yaml:

atmos workflow check -f checks

Both checks pass and the command exits with code zero. The output value from the first step is JSON-encoded; the test receives it through its environment and decodes it into a dictionary. Shared logic can live in a .star module loaded by both your automation and its checks.

Change the produced worker count from 12 to 11 and rerun. The result check fails with worker count must equal replicas times four, the independent calculation check still passes, and the workflow exits nonzero.

Example: test a custom command

Test the command your team runs as well as its internal logic. For the custom command guide's atmos capacity, add this child under the test step's steps list in the same project:

- name: rejects-zero
  type: script
  interpreter: starlark
  script: |
    result = exec.run(
        ["atmos", "capacity", "--replicas", "0"],
        check=False,
        output="capture",
    )
    if result.exit_code == 0:
        fail("capacity accepted zero replicas")
    if "replicas must be positive" not in result.stdout + result.stderr:
        fail("capacity failed for an unexpected reason")

This runs the real custom command. check=False returns an unsuccessful process result so the script can check both the exit code and the diagnostic. Launch or cancellation errors still fail the step. output="capture" keeps the expected error out of the live output while making it available to the check.

The same approach tests standalone scripts and other tools. Use fixed inputs for deterministic checks; subprocess tests still need the tools they invoke.