Workflows

Workflows are a way of combining multiple commands into one executable unit of work.

You can use Atmos Custom Commands in Atmos Workflows, and Atmos Workflows in Atmos Custom Commands

Simple Example

Here's an example workflow called eks-up which runs a few commands that will bring up the EKS cluster:

workflows:
  eks-up:
    description: |
      Bring up the EKS cluster.
    steps:
      - command: terraform apply vpc -auto-approve
      - command: terraform apply eks/cluster -auto-approve
      - command: terraform apply eks/alb-controller -auto-approve

Retry Configuration

The command section of the workflow schema has been updated to allow for retrying of failed commands. The retry section accepts the following parameters:

• max_attempts: The maximum number of times the command will be retried. The default is 1.
• delay: The amount of time to delay between retries. The default is 5s.
• backoff_strategy: The backoff strategy to use. The default is constant. The other options are exponential and linear.
• initial_delay: The initial delay to use when retrying. The default is 5s.
• random_jitter: the random jitter number to the delay between retries. The default is 0.0.
• multiplier: The multiplier to use when retrying. The default is 2.
• max_elapsed_time: The maximum time allocated to the command before the retry fails. The default is 30 minutes.

Here is an example of a workflow with retry configuration:

workflows:
  eks-up:
    description: Bring up the eks cluster
    steps:
      - command: terraform apply vpc -auto-approve
        retry:
          max_attempts: 3
          backoff_strategy: exponential # could be exponential, constant and linear
          initial_delay: 3s # the time used by the backoff_strategy
          random_jitter: 0.0 # the random jitter to be added while retrying.
          multiplier: 2 # used during exponential strategy as multiplier that would be raised by configuration
          max_elapsed_time: 4m # Default (30 min for all commands). This sets the total duration alloated to the command before the retry fails.
      - command: terraform apply eks/cluster -auto-approve
      - command: terraform apply eks/alb-controller -auto-approve

note

The workflow name can be anything you want, and the workflow can also accept command-line parameters (e.g. stack name)

If you define this workflow in the file workflow1.yaml, it can we executed like this to provision the vpc, eks/cluster and eks/alb-controller Atmos Components into the tenant1-ue2-dev Atmos Stack:

atmos workflow eks-up -f workflow1 --stack tenant1-ue2-dev

tip

Refer to atmos workflow for the complete description of the CLI command

Configuration

To configure and execute Atmos workflows, follow these steps:

• Configure workflows in atmos.yaml CLI config file
• Create workflow files and define workflows using the workflow schema

Configure Workflows in atmos.yaml

In atmos.yaml CLI config file, add the following sections related to Atmos workflows:

# Base path for components, stacks and workflows configurations.
# Can also be set using 'ATMOS_BASE_PATH' ENV var, or '--base-path' command-line argument.
# Supports both absolute and relative paths.
# If not provided or is an empty string, 'components.terraform.base_path', 'components.helmfile.base_path', 'stacks.base_path'
# and 'workflows.base_path' are independent settings (supporting both absolute and relative paths).
# If 'base_path' is provided, 'components.terraform.base_path', 'components.helmfile.base_path', 'stacks.base_path'
# and 'workflows.base_path' are considered paths relative to 'base_path'.
base_path: ""

workflows:
  # Can also be set using 'ATMOS_WORKFLOWS_BASE_PATH' ENV var, or '--workflows-dir' command-line arguments
  # Supports both absolute and relative paths
  base_path: "stacks/workflows"

where:

base_path

The base path for components, stacks and workflows configurations

workflows.base_path

The base path to Atmos workflow files

Create Workflow Files

In atmos.yaml, we set workflows.base_path to stacks/workflows. The folder is relative to the root of the repository.

Refer to networking.yaml for an example.

We put the workflow files into the folder. The workflow file names can be anything you want, but we recommend naming them according to the functions they perform, e.g. create separate workflow files per environment, account, team, or service.

For example, you can have a workflow file stacks/workflows/workflows-eks.yaml to define all EKS-related workflows.

Or, you can have a workflow file stacks/workflows/workflows-dev.yaml to define all workflows to provision resources into the dev account. Similarly, you can create a workflow file stacks/workflows/workflows-prod.yaml to define all workflows to provision resources into the prod account.

You can segregate the workflow files even further, e.g. per account and service. For example, in the workflow file stacks/workflows/workflows-dev-eks.yaml you can define all EKS-related workflows for the dev account.

Use Workflow Schema

Workflow files must confirm to the following schema:

workflows:

  workflow-1:
    description: "Description of Workflow #1"
    steps: []

  workflow-2:
    description: "Description of Workflow #2"
    steps: []

Each workflow file must have the workflows: top-level section with a map of workflow definitions.

Each workflow definition must confirm to the following schema:

  workflow-1:
    description: "Description of Workflow #1"
    stack: <Atmos stack> # optional
    steps:
      - command: <Atmos command to execute>
        name: <step name>>  # optional
        type: atmos  # optional
        stack: <Atmos stack> # optional
      - command: <Atmos command to execute>
        name: <step name>>  # optional
        stack: <Atmos stack> # optional
      - command: <shell script>
        name: <step name>>  # optional
        type: shell  # required for the steps of type `shell`

Schema Definitions

description

The workflow description

stack

Workflow-level Atmos stack (optional). If specified, all workflow steps of type atmos will be executed for this Atmos stack. It can be overridden in each step or on the command line by using the --stack flag (-s for shorthand)

steps

A list of workflow steps which are executed sequentially in the order they are specified

command

The command to execute. Can be either an Atmos CLI command (without the atmos binary name in front of it, for example command: terraform apply vpc), or a shell script. The type of the command is specified by the type attribute

name

Step name (optional). It's used to find the first step from which to start executing the workflow when the command-line flag --from-step is specified. If the name is omitted, a friendly name will be generated for you consisting of a prefix of step and followed by the index of the step (the index starts with 1, so the first generated step name would be step1).

type

The type of the command. Can be either atmos or shell. Type atmos is implicit, you don't have to specify it if the command is an Atmos CLI command. Type shell is required if the command is a shell script. When executing a step of type atmos, Atmos prepends the atmos binary name to the provided command before executing it

stack

Step-level Atmos stack (optional). If specified, the command will be executed for this Atmos stack. It overrides the workflow-level stack attribute, and can itself be overridden on the command line by using the --stack flag (-s for shorthand)

note

A workflow command of type shell can be any simple or complex shell command or script. You can use YAML Multiline Strings to create complex multi-line shell scripts.

Executing Workflow from a Named Step

Each workflow step can be given an arbitrary name (step's identifier) using the name attribute. For example:

workflows:
  test-1:
    description: "Test workflow"
    steps:
      - command: echo Command 1
        name: step1
        type: shell
      - command: echo Command 2
        name: step2
        type: shell
      - command: echo Command 3
        name: step3
        type: shell
      - command: echo Command 4
        type: shell

The step's name can be used in the --from-step command-line flag to start the workflow execution from the step.

For example, the following command will skip the first two steps and will start executing the workflow from step3:

atmos workflow test-1 -f workflow1 --from-step step3

This is useful when you want to restart the workflow from a particular step.

For example:

• You run the workflow first time with the command atmos workflow test-1 -f workflow1
• step1 and step2 succeed, but step3 fails
• You fix the issue with the step3 command
• You don't want to execute step1 and step2 again (to not spend time on it, or if they are not idempotent)
• You run the command atmos workflow test-1 -f workflow1 --from-step step3 to restart the workflow from step3

If the name attribute in a workflow step is omitted, a friendly name will be generated for you consisting of a prefix of step and followed by the index of the step. The index starts with 1, so the first generated step name would be step1.

The test-1 workflow defined above does not have the name attribute for the last workflow step. When we execute the atmos workflow commands, Atmos automatically generates the names for the steps where the name attribute is omitted. In this case, the generated name for the last step will be step4.

For example:

atmos workflow test-1 -f workflow1 --from-step step4

Executing the workflow 'test-1' from 'stacks/workflows/workflow1.yaml'

description: Test workflow
steps:
- name: step1
  command: echo Command 1
  type: shell
- name: step2
  command: echo Command 2
  type: shell
- name: step3
  command: echo Command 3
  type: shell
- name: step4
  command: echo Command 4
  type: shell

Executing workflow step: echo Command 4

Executing command: echo Command 4
Command 4

Executing workflow step: echo Command 5

Executing command: echo Command 5
Command 5

Workflow Examples

The following workflow defines four steps of type atmos (implicit type) without specifying the workflow-level or step-level stack attribute. Since the workflow does not specify the stack, it's generic and can be executed for any Atmos stack. In this case, the stack needs to be provided on the command line.

workflows:
  terraform-plan-all-test-components:
    description: |
      Run 'terraform plan' on 'test/test-component' and all its derived components.
      The stack must be provided on the command line:
      `atmos workflow terraform-plan-all-test-components -f workflow1 -s <stack>`
    steps:
      # Inline scripts are also supported
      # Refer to https://yaml-multiline.info for more details
      - type: shell
        command: |
          echo "Starting the workflow execution..."
          read -p "Press any key to continue... " -n1 -s
      - command: terraform plan test/test-component
      - command: terraform plan test/test-component-override
      - command: terraform plan test/test-component-override-2
      - command: terraform plan test/test-component-override-3
      - type: shell
        command: >-
          echo "All done!"

To run this workflow for the tenant1-ue2-dev stack, execute the following command:

atmos workflow terraform-plan-all-test-components -f workflow1 -s tenant1-ue2-dev

The following workflow executes terraform plan on test/test-component-override-2 component in all stacks. In this case, the stack is specified inline for each workflow command.

workflows:
  terraform-plan-test-component-override-2-all-stacks:
    description: Run 'terraform plan' on 'test/test-component-override-2' component in all stacks
    steps:
      - command: terraform plan test/test-component-override-2 -s tenant1-ue2-dev
      - command: terraform plan test/test-component-override-2 -s tenant1-ue2-staging
      - command: terraform plan test/test-component-override-2 -s tenant1-ue2-prod
      - command: terraform plan test/test-component-override-2 -s tenant2-ue2-dev
      - command: terraform plan test/test-component-override-2 -s tenant2-ue2-staging
      - command: terraform plan test/test-component-override-2 -s tenant2-ue2-prod
      - type: shell
        command: echo "All done!"

To run this workflow, execute the following command:

atmos workflow terraform-plan-test-component-override-2-all-stacks -f workflow1

The following workflow is similar to the above, but the stack for each command is specified in the step-level stack attribute.

workflows:
  terraform-plan-test-component-override-3-all-stacks:
    description: Run 'terraform plan' on 'test/test-component-override-3' component in all stacks
    steps:
      - command: terraform plan test/test-component-override-3
        stack: tenant1-ue2-dev
      - command: terraform plan test/test-component-override-3
        stack: tenant1-ue2-staging
      - command: terraform plan test/test-component-override-3
        stack: tenant1-ue2-prod
      - command: terraform plan test/test-component-override-3
        stack: tenant2-ue2-dev
      - command: terraform plan test/test-component-override-3
        stack: tenant2-ue2-staging
      - command: terraform plan test/test-component-override-3
        stack: tenant2-ue2-prod

To run this workflow, execute the following command:

atmos workflow terraform-plan-test-component-override-3-all-stacks -f workflow1

Working with Atmos Stacks in Workflows

The Atmos stack used by the workflow commands of type atmos can be specified in four different ways:

Inline in the command itself

steps:
  - command: terraform plan test/test-component-override-2 -s tenant1-ue2-dev

In the workflow-level stack attribute

workflows:
  my-workflow:
    stack: tenant1-ue2-dev
    steps:
      - command: terraform plan test/test-component

In the step-level stack attribute

steps:
  - command: terraform plan test/test-component
    stack: tenant1-ue2-dev

On the command line

atmos workflow my-workflow -f workflow1 -s tenant1-ue2-dev

Workflow Failure Handling and Resuming

When a workflow step fails, Atmos will:

1. Display which step failed
2. Show the exact command that failed
3. Provide a ready-to-use command to resume the workflow from the failed step

Given this workflow:

stacks/workflows/networking.yaml

workflows:
  provision-vpcs:
    description: "Deploy vpc components"
    steps:
      - command: terraform plan vpc -s plat-ue2-dev
        name: step-1
      - command: terraform plan vpc -s plat-ue2-staging
        name: step-2
      - command: terraform plan vpc -s plat-ue2-prod
        name: step-3

If step-2 fails, you'll see:

Step 'step-2' failed!

Command failed:
terraform plan vpc -s plat-ue2-staging

To resume the workflow from this step, run:
atmos workflow provision-vpcs -f networking --from-step step-2

Stack Precedence

The stack defined inline in the command itself has the lowest priority, it can and will be overridden by any other stack definition. The step-level stack will override the workflow-level stack. The command line --stack option will override all other stacks defined in the workflow itself. You can also use any combinations of the above (e.g. specify the stack at the workflow level, then override it at the step level for some commands, etc.).

While this provides a great flexibility in defining the stack for workflow commands, we recommend creating generic workflows without defining stacks in the workflow itself (the stack should be provided on the command line). This way, the workflow can be executed for any stack without any modifications and without dealing with multiple workflows that are similar but differ only by the environment where the resources are provisioned.