Import Stack Configurations

As your stacks grow taller with more and more component configurations, it often makes sense to start splitting them apart into different files. That's why you might want to take advantage of imports. This helps you keep your stack files smaller so they are easier to understand, while reusing their configuration in multiple places.

Each import overlays on top of others, and gets deep merged. Then we support inheritance and overrides to manage configuration variations, all without relying on templating. When none of these other methods work for your use-case, we provide an "Escape Hatch" with templating.

Use cases

DRY Configuration: Imports are how we reduce duplication of configurations.
Configuration Blueprints: Define reusable baselines or "defaults". Think of them almost as blueprints, that you can reuse anytime you want some particular combination of components in a stack.
Service Catalogs: Provide a "Service Catalog" for your team with reusable configurations that anyone can use to easily compose architectures with golden-path configurations.

Pitfalls!

Overusing imports can make configurations harder to understand. We recommend limiting import levels to maintain clarity. Review our best practices for practical guidance.

Imports may be used in Stack configurations together with inheritance and mixins to produce an exceptionally DRY configuration in a way that is logically organized and easier to maintain by your team.

Configuration

To import any stack configuration from the catalog/, simply define an import section at the top of any Stack configuration. Technically, it can be placed anywhere in the file, but by convention we recommend putting it at the top.

Here are some simple examples of how to import configurations:

import:
  - catalog/file1 # First import "file1" from the catalog
  - catalog/file2 # Second import "file2" from the catalog, deep merging on top of the first import
  - catalog/file3 # Third import "file3" from the catalog, deep merging on top of the preceeding imports

The base path for imports is specified in the atmos.yaml in the stacks.base_path section.

If no file extension is used, a .yaml extension is automatically appended.

It's also possible to specify file extensions, although we do not recommend it.

import:
  - catalog/file1.yml   # Expicitly load a file with a .yml extension
  - catalog/file2.yaml  # Expicitly load a file with a .yaml extension
  - catalog/file3.YAML  # Expicitly load a file with a .YAML extension

Automatic Template File Detection

When importing files without specifying an extension, Atmos will now automatically search for and use template versions of the files if they exist. The search order is:

.yaml
.yml
.yaml.tmpl
.yml.tmpl

For example, if you import catalog/file1, Atmos will:

First look for catalog/file1.yaml or catalog/file1.yml
If found, check if a template version exists (catalog/file1.yaml.tmpl or catalog/file1.yml.tmpl)
Use the template version if it exists, otherwise use the regular YAML file
If no files are found, default to using .yaml extension

This feature makes it easier to work with templated configurations as you don't need to explicitly specify the template file extension - Atmos will automatically use the template version when available.

Template File Validation

While template files are automatically detected and processed during normal operations (imports, etc.), they are excluded from YAML validation (atmos validate stacks) since they may contain template placeholders that are invalid YAML before being rendered.

This means:

Template files are fully supported for imports and normal operations
Template files are skipped during atmos validate stacks to prevent validation errors from unrendered templates
You don't need to explicitly specify template extensions - Atmos will find them automatically

Conventions

We recommend placing all baseline "imports" in the stacks/catalog folder, however, they can exist anywhere.

Use mixins for reusable snippets of configurations that alter the behavior of Stacks in some way.

Imports Schema

The import section supports two different formats, depending on whether the imported files use templates or not. One is a list of strings respresenting paths to the imported files, and the other is a list of objects with several feature flags.

Imports without Templates

For a list of paths to the imported files, just provide a list of strings like this:

stacks/orgs/cp/tenant1/test1/us-east-2.yaml
import:
  - mixins/region/us-east-2
  - orgs/cp/tenant1/test1/_defaults
  - catalog/terraform/top-level-component1
  - catalog/terraform/test-component
  - catalog/terraform/vpc
  - catalog/helmfile/echo-server

Imports with Templates

Sometimes you may want to import files that use Go templates. In this case, you can provide a list of objects with context that can be used to dynamically generate a stack configuration.

important

The imported files must have the .yaml.tmpl extension to be processed as Go templates and to be able to use the context variables. Additionally, the skip_templates_processing flag can be used to skip template processing for the imported file. Templating must be enabled in atmos.yaml for Atmos to process the imported files as Go templates.

For example, here we import a file with a template and provide a context to passing two variables.

import:
  - path: "catalog/something.yaml.tmpl" # Path to the imported file with the required .tmpl extension for Go templates
    context:
      foo: bar
      baz: qux
    skip_templates_processing: false
    ignore_missing_template_values: false
    skip_if_missing: false
  - path: "catalog/something.yaml.tmpl"
    context: {}
    skip_templates_processing: false
    ignore_missing_template_values: true
    skip_if_missing: true

The import section supports the following fields:

path - (string) required: The path to the imported file
context - (map): An optional freeform map of context variables that are applied as template variables to the imported file (if the imported file is a Go template)
skip_templates_processing - (boolean): Skip template processing for the imported file. Can be used if the imported file uses Go templates that should not be interpretted by atmos. For example, sometimes configurations for components may pass Go template strings not intended for atmos.
ignore_missing_template_values - (boolean): Ignore the missing template values in the imported file. Can be used if the imported file uses Go templates to configure external systems, e.g. Datadog. In this case, Atmos will process all template values that are provided in the context, and will skip the missing values in the templates for the external systems without throwing an error. The ignore_missing_template_values setting is different from skip_templates_processing in that skip_templates_processing skips the template processing completely in the imported file, while ignore_missing_template_values processes the templates using the values provided in the context and skips all the missing values
skip_if_missing - (boolean): Set it to true to ignore the imported manifest if it does not exist, and don't throw an error. This is useful when generating Atmos manifests using other tools, but the imported files are not present yet at the generation time.

A combination of the two formats is also supported:

import:
  - mixins/region/us-east-2
  - orgs/cp/tenant1/test1/_defaults
  - path: "<path_to_atmos_manifest1>"
  - path: "<path_to_atmos_manifest2>"
    context: {}
    skip_templates_processing: false
    ignore_missing_template_values: true

`Go` Templates in Imports

Atmos supports all the functionality of Go templates in imported stack configurations, including functions and Sprig functions.

Stack configurations can be templatized and then reused with different settings provided via the import context section.

For example, we can define the following configuration for EKS Atmos components in the catalog/terraform/eks_cluster.yaml.tmpl template file:

stacks/catalog/terraform/eks_cluster.yaml.tmpl
# Imports can also be parameterized using `Go` templates
import: []

components:
  terraform:
    "eks-{{ .flavor }}/cluster":
      metadata:
        component: "test/test-component"
      vars:
        enabled: "{{ .enabled }}"
        name: "eks-{{ .flavor }}"
        service_1_name: "{{ .service_1_name }}"
        service_2_name: "{{ .service_2_name }}"
        tags:
          flavor: "{{ .flavor }}"

note

Since Go processes files ending in .yaml.tmpl text files with templates, we can parameterize the Atmos component name eks-{{ .flavor }}/cluster and any values in any sections (vars, settings, env, backend, etc.), and even the import section in the imported file (if the file imports other configurations).

Then we can import the template into a top-level stack multiple times providing different context variables to each import:

stacks/orgs/cp/tenant1/test1/us-west-2.yaml
import:
  - path: "mixins/region/us-west-2"
  - path: "orgs/cp/tenant1/test1/_defaults"

  # This import with the provided context will dynamically generate
  # a new Atmos component `eks-blue/cluster` in the current stack
  - path: "catalog/terraform/eks_cluster.yaml.tmpl"
    context:
      flavor: "blue"
      enabled: true
      service_1_name: "blue-service-1"
      service_2_name: "blue-service-2"

  # This import with the provided context will dynamically generate
  # a new Atmos component `eks-green/cluster` in the current stack
  - path: "catalog/terraform/eks_cluster.yaml.tmpl"
    context:
      flavor: "green"
      enabled: false
      service_1_name: "green-service-1"
      service_2_name: "green-service-2"

Now we can execute the following Atmos commands to describe and provision the dynamically generated EKS components into the stack:

atmos describe component eks-blue/cluster -s tenant1-uw2-test-1
atmos describe component eks-green/cluster -s tenant1-uw2-test-1

atmos terraform apply eks-blue/cluster -s tenant1-uw2-test-1
atmos terraform apply eks-green/cluster -s tenant1-uw2-test-1

All the parameterized variables get their values from the context:

atmos describe component eks-blue/cluster -s tenant1-uw2-test-1
vars:
  enabled: true
  environment: uw2
  name: eks-blue
  namespace: cp
  region: us-west-2
  service_1_name: blue-service-1
  service_2_name: blue-service-2
  stage: test-1
  tags:
    flavor: blue
  tenant: tenant1

atmos describe component eks-green/cluster -s tenant1-uw2-test-1
vars:
  enabled: true
  environment: uw2
  name: eks-green
  namespace: cp
  region: us-west-2
  service_1_name: green-service-1
  service_2_name: green-service-2
  stage: test-1
  tags:
    flavor: green
  tenant: tenant1

Hierarchical Imports with Context

Atmos supports hierarchical imports with context. This will allow you to parameterize the entire chain of stack configurations and dynamically generate components in stacks.

For example, let's create the configuration stacks/catalog/terraform/eks_cluster_hierarchical.yaml.tmpl with the following content:

stacks/catalog/terraform/eks_cluster_hierarchical.yaml.tmpl
import:
  # Use `region.yaml.tmpl` `Go` template and provide `context` for it.
  # This can also be done by using `Go` templates in the import path itself.
  # - path: "mixins/region/{{ .region }}"
  - path: "mixins/region/region.yaml.tmpl"
    # `Go` templates in `context`
    context:
      region: "{{ .region }}"
      environment: "{{ .environment }}"

  # `Go` templates in the import path
  - path: "orgs/cp/{{ .tenant }}/{{ .stage }}/_defaults"

components:
  terraform:
    # Parameterize Atmos component name
    "eks-{{ .flavor }}/cluster":
      metadata:
        component: "test/test-component"
      vars:
        # Parameterize variables
        enabled: "{{ .enabled }}"
        name: "eks-{{ .flavor }}"
        service_1_name: "{{ .service_1_name }}"
        service_2_name: "{{ .service_2_name }}"
        tags:
          flavor: "{{ .flavor }}"

Then we can import the template into a top-level stack multiple times providing different context variables to each import and to the imports for the entire inheritance chain (which catalog/terraform/eks_cluster_hierarchical.yaml.tmpl imports itself):

stacks/orgs/cp/tenant1/test1/us-west-1.yaml
import:

  # This import with the provided hierarchical context will dynamically generate
  # a new Atmos component `eks-blue/cluster` in the `tenant1-uw1-test1` stack
  - path: "catalog/terraform/eks_cluster_hierarchical.yaml.tmpl"
    context:
      # Context variables for the EKS component
      flavor: "blue"
      enabled: true
      service_1_name: "blue-service-1"
      service_2_name: "blue-service-2"
      # Context variables for the hierarchical imports
      # `catalog/terraform/eks_cluster_hierarchical.yaml.tmpl` imports other parameterized configurations
      tenant: "tenant1"
      region: "us-west-1"
      environment: "uw1"
      stage: "test1"

  # This import with the provided hierarchical context will dynamically generate
  # a new Atmos component `eks-green/cluster` in the `tenant1-uw1-test1` stack
  - path: "catalog/terraform/eks_cluster_hierarchical.yaml.tmpl"
    context:
      # Context variables for the EKS component
      flavor: "green"
      enabled: false
      service_1_name: "green-service-1"
      service_2_name: "green-service-2"
      # Context variables for the hierarchical imports
      # `catalog/terraform/eks_cluster_hierarchical.yaml.tmpl` imports other parameterized configurations
      tenant: "tenant1"
      region: "us-west-1"
      environment: "uw1"
      stage: "test1"

In the case of hierarchical imports, Atmos performs the following steps:

Processes all the imports in the import section in the current configuration in the order they are specified providing the context to all imported files
For each imported file, Atmos deep-merges the parent context with the current context. Note that the current context (in the current file) takes precedence over the parent context and will override items with the same keys. Atmos does this hierarchically for all imports in all files, effectively processing a graph of imports and deep-merging the contexts on all levels

For example, in the stacks/orgs/cp/tenant1/test1/us-west-1.yaml configuration above, we first import the catalog/terraform/eks_cluster_hierarchical.yaml.tmpl and provide it with the context which includes the context variables for the EKS component itself, as well as the context variables for all the hierarchical imports. Then, when processing the stacks/catalog/terraform/eks_cluster_hierarchical.yaml.tmpl configuration, Atmos deep-merges the parent context (from stacks/orgs/cp/tenant1/test1/us-west-1.yaml) with the current context and processes the Go templates.

We are now able to dynamically generate the components eks-blue/cluster and eks-green/cluster in the stack tenant1-uw1-test1 and can execute the following Atmos commands to provision the components into the stack:

atmos terraform apply eks-blue/cluster -s tenant1-uw1-test-1
atmos terraform apply eks-green/cluster -s tenant1-uw1-test-1

All the parameterized variables get their values from the hierarchical context settings:

atmos describe component eks-blue/cluster -s tenant1-uw1-test-1
vars:
  enabled: true
  environment: uw1
  name: eks-blue
  namespace: cp
  region: us-west-1
  service_1_name: blue-service-1
  service_2_name: blue-service-2
  stage: test-1
  tags:
    flavor: blue
  tenant: tenant1

Handle with Care

Leveraging Go templating for Atmos stack generation grants significant power but demands equal responsibility. It can easily defy the principle of creating stack configurations that are straightforward and intuitive to read.

While templating fosters DRYer code, it comes at the expense of searchable components and introduces elements like conditionals, loops, and dynamic variables that impede understandability. It's a tool not for regular use, but for instances where code duplication becomes excessively cumbersome.

Before resorting to advanced Go templates in Atmos, rigorously evaluate the trade-off between the value added and the complexity introduced.

Advanced Examples of Templates in Atmos Configurations

Atmos supports all the functionality of Go templates, including functions and Sprig functions. The Sprig library provides over 70 template functions for Go's template language.

The following example shows how to dynamically include a variable in the Atmos component configuration by using the hasKey Sprig function. The hasKey function returns true if the given dictionary contains the given key.

components:
  terraform:
    eks/iam-role/{{ .app_name }}/{{ .service_environment }}:
      metadata:
        component: eks/iam-role
      settings:
        spacelift:
          workspace_enabled: true
      vars:
        enabled: {{ .enabled }}
        tags:
          Service: {{ .app_name }}
        service_account_name: {{ .app_name }}
        service_account_namespace: {{ .service_account_namespace }}
        {{ if hasKey . "iam_managed_policy_arns" }}
        iam_managed_policy_arns:
          {{ range $i, $iam_managed_policy_arn := .iam_managed_policy_arns }}
          - '{{ $iam_managed_policy_arn }}'
          {{ end }}
        {{- end }}
        {{ if hasKey . "iam_source_policy_documents" }}
        iam_source_policy_documents:
          {{ range $i, $iam_source_policy_document := .iam_source_policy_documents }}
          - '{{ $iam_source_policy_document }}'
          {{ end }}
        {{- end }}

The iam_managed_policy_arns and iam_source_policy_documents variables will be included in the component configuration only if the provided context object has the iam_managed_policy_arns and iam_source_policy_documents fields.

Summary

Using imports with context (and hierarchical imports with context) with parameterized config files will help you make the configurations extremely DRY. It's very useful in many cases, for example, when creating stacks and components for EKS blue-green deployment.

Use cases​

Configuration​

Automatic Template File Detection​

Conventions​

Imports Schema​

Imports without Templates​

Imports with Templates​

Go Templates in Imports​

Hierarchical Imports with Context​

Advanced Examples of Templates in Atmos Configurations​

Summary​

Related​