Vendor Manifest

The vendoring configuration is defined in the vendor.yaml manifest (vendor config file). The vendoring manifest is used to make copies of 3rd-party components, stacks, and other artifacts in your own repository.

It functions a little bit like the packages.json file in Node.js or the go.mod file in Go, but for infrastructure code.

How it works

Atmos searches for the vendoring manifest in the following locations and uses the first one found:

In the directory from which the atmos vendor pull command is executed, usually in the root of the infrastructure repo
In the directory pointed to by the base_path setting in the atmos.yaml CLI config file

After defining the vendor.yaml manifest, all the remote artifacts can be downloaded by running the following command:

atmos vendor pull

To vendor a particular component or other artifact, execute the following command:

atmos vendor pull -c <component>

To vendor components and artifacts tagged with specific tags, execute the following command:

atmos vendor pull --tags <tag1>,<tag2>

tip

Refer to atmos vendor pull CLI command for more details

Vendoring Manifest

To vendor remote artifacts, create a vendor.yaml file similar to the example below:

vendor.yaml

apiVersion: atmos/v1
kind: AtmosVendorConfig
metadata:
  name: example-vendor-config
  description: Atmos vendoring manifest
spec:
  # `imports` or `sources` (or both) must be defined in a vendoring manifest
  imports:
    - "vendor/vendor2"
    - "vendor/vendor3.yaml"

  sources:
    # `source` supports the following protocols: local paths (absolute and relative), OCI (https://opencontainers.org),
    # Git, Mercurial, HTTP, HTTPS, Amazon S3, Google GCP,
    # and all URL and archive formats as described in https://github.com/hashicorp/go-getter.
    # In 'source' and 'targets', Golang templates are supported  https://pkg.go.dev/text/template.
    # Currently the fields '{{.Component}}' and '{{.Version}}' are supported.
    # Download the component from the AWS public ECR registry (https://docs.aws.amazon.com/AmazonECR/latest/public/public-registries.html).
    - component: "vpc"
      source: "oci://public.ecr.aws/cloudposse/components/terraform/stable/aws/vpc:{{.Version}}"
      version: "latest"
      targets:
        - "components/terraform/infra/vpc3"
      # Only include the files that match the 'included_paths' patterns.
      # If 'included_paths' is not specified, all files will be matched except those that match the patterns from 'excluded_paths'.
      # 'included_paths' support POSIX-style Globs for file names/paths (double-star `**` is supported).
      # https://en.wikipedia.org/wiki/Glob_(programming)
      # https://github.com/bmatcuk/doublestar#patterns
      included_paths:
        - "**/*.tf"
        - "**/*.tfvars"
        - "**/*.md"
      # Tags can be used to vendor component that have the specific tags
      # `atmos vendor pull --tags test`
      # Refer to https://atmos.tools/cli/commands/vendor/pull
      tags:
        - test
        - networking
    - component: "vpc-flow-logs-bucket"
      source: "github.com/cloudposse/terraform-aws-components.git//modules/vpc-flow-logs-bucket?ref={{.Version}}"
      version: "1.323.0"
      targets:
        - "components/terraform/infra/{{.Component}}/{{.Version}}"
      excluded_paths:
        - "**/*.yaml"
        - "**/*.yml"
      # Tags can be used to vendor component that have the specific tags
      # `atmos vendor pull --tags networking,storage`
      # Refer to https://atmos.tools/cli/commands/vendor/pull
      tags:
        - test
        - storage
    - component: "vpc-mixin-1"
      source: "https://raw.githubusercontent.com/cloudposse/terraform-null-label/0.25.0/exports/context.tf"
      targets:
        - "components/terraform/infra/vpc3"
      # Tags can be used to vendor component that have the specific tags
      # `atmos vendor pull --tags test`
      # Refer to https://atmos.tools/cli/commands/vendor/pull
      tags:
        - test
    - component: "vpc-mixin-2"
      # Copy a local file into a local folder (keeping the same file name)
      # This `source` is relative to the current folder
      source: "components/terraform/mixins/context.tf"
      targets:
        - "components/terraform/infra/vpc3"
      # Tags can be used to vendor component that have the specific tags
      # `atmos vendor pull --tags test`
      # Refer to https://atmos.tools/cli/commands/vendor/pull
      tags:
        - test
    - component: "vpc-mixin-3"
      # Copy a local folder into a local folder
      # This `source` is relative to the current folder
      source: "components/terraform/mixins"
      targets:
        - "components/terraform/infra/vpc3"
      # Tags can be used to vendor component that have the specific tags
      # `atmos vendor pull --tags test`
      # Refer to https://atmos.tools/cli/commands/vendor/pull
      tags:
        - test
    - component: "vpc-mixin-4"
      # Copy a local file into a local file with a different file name
      # This `source` is relative to the current folder
      source: "components/terraform/mixins/context.tf"
      targets:
        - "components/terraform/infra/vpc3/context-copy.tf"
      # Tags can be used to vendor component that have the specific tags
      # `atmos vendor pull --tags test`
      # Refer to https://atmos.tools/cli/commands/vendor/pull
      tags:
        - test

With this configuration, it would be possible to run the following commands:

# atmos vendor pull 
# atmos vendor pull --everything
# atmos vendor pull --component vpc-mixin-1
# atmos vendor pull -c vpc-mixin-2
# atmos vendor pull -c vpc-mixin-3
# atmos vendor pull -c vpc-mixin-4
# atmos vendor pull --tags test
# atmos vendor pull --tags networking,storage

Vendoring Manifest Schema

The vendor.yaml vendoring manifest supports Kubernetes-style YAML config to describe vendoring configuration for components, stacks, and other artifacts. The file is placed into the directory from which the atmos vendor pull command is executed (usually the root of the repo).

version

The version attribute is used to specify the version of the artifact to download. The version attribute is used in the source and targets attributes as a template parameter using {{ .Version }}.

source

The source attribute supports all protocols (local files, Git, Mercurial, HTTP, HTTPS, Amazon S3, Google GCP), and all the URL and archive formats as described in go-getter, and also the oci:// scheme to download artifacts from OCI registries.

IMPORTANT: Include the {{ .Version }} parameter in your source URI to ensure the correct version of the artifact is downloaded.

For example, for http and https sources, use the following format:

source: "github.com/cloudposse/terraform-aws-components.git//modules/vpc-flow-logs-bucket?ref={{.Version}}"

For private Git repositories, prepend the URI with git:: and use the following format to pass a environment variable with the GitHub token:

source: "git::https://{{env "GITHUB_TOKEN"}}@github.com/some-org/some-private-repo/terraform/vpc.git?ref={{.Version}}"

Note, that GITHUB_TOKEN provided by GitHub Actions are only valid for the current repository, or repositories marked as internal within GitHub Enterprise organizations. For cross-repository access, make sure you provision a fine grained token with the necessary permissions.

ref: Pass the ref as a query string with either the tag, branch, or commit hash to download the correct version of the artifact. e.g. ?ref={{.Version}} will pass the version attribute to the ref query string.
depth: Pass the depth as a query string to download only the specified number of commits from the repository. e.g. ?depth=1 will download only the latest commit.

targets

The targets in each source supports absolute paths and relative paths (relative to the vendor.yaml file). Note: if the targets paths are set as relative, and if the vendor.yaml file is detected by Atmos using the base_path setting in atmos.yaml, the targets paths will be considered relative to the base_path. Multiple targets can be specified.

included_paths and excluded_paths

included_paths and excluded_paths support POSIX-style greedy Globs for filenames/paths (double-star/globstar ** is supported as well).

component

The component attribute in each source is optional. It's used in the atmos vendor pull -- component <component> command if the component is passed in. In this case, Atmos will vendor only the specified component instead of vendoring all the artifacts configured in the vendor.yaml manifest.

source and targets templates

The source and targets attributes support Go templates and Sprig Functions. This can be used to templatise the source and targets paths with the component name specified in the component attribute and artifact versions specified in the version attribute.

Here's an advanced example showcasing how templates and Sprig functions can be used together with targets:

targets:
  # Vendor a component into a major-minor versioned folder like 1.2
  - "components/terraform/infra/vpc-flow-logs-bucket/{{ (first 2 (splitList \".\" .Version)) | join \".\" }}"

tags

The tags in each source specifies a list of tags to apply to the component. This allows you to only vendor the components that have the specified tags by executing a command atmos vendor pull --tags <tag1>,<tag2>

imports

The imports section defines the additional vendoring manifests that are merged into the main manifest. Hierarchical imports are supported at many levels (one vendoring manifest can import another, which in turn can import other manifests, etc.). Atmos processes all imports and all sources in the imported manifests in the order they are defined.

note

The imported file extensions are optional. Imports that do not include file extensions will default to the .yaml extension.

vendor.yaml

vendor.yaml
spec:
  sources:
    - component: "vpc-flow-logs-bucket"
      source: "github.com/cloudposse/terraform-aws-components.git//modules/vpc-flow-logs-bucket?ref={{.Version}}"
      version: "1.323.0"
      targets:
        - "components/terraform/vpc-flow-logs-bucket"
      included_paths:
        - "**/**"
        # If the component's folder has the `modules` sub-folder, it needs to be explicitly defined
        - "**/modules/**"

warning

The glob library that Atmos uses to download remote artifacts does not treat the double-star ** as including sub-folders. If the component's folder has sub-folders, and you need to vendor them, they have to be explicitly defined as in the following example.

Template Parameters

The vendor manifest supports basic template parameters, which is useful for versioning and other dynamic values. The following template parameters are supported:

{{ .Component }}

Refers to the component attribute in the current section. The component attribute is used to specify the component name. This is useful to vendor components into folders by the same name.

targets:
  - "components/terraform/{{ .Component }}"

{{ .Version }}

Refers to the version attribute the current section. The version attribute is used to specify the version of the artifact to download. This is useful to version components into different folders.

targets:
  - "components/terraform/{{ .Component }}/{{ .Version }}"

When stacks need to pin to different versions of the same component, the {{ .Version }} template parameter can be used to ensure the components are vendored into different folders.

You can also use any of the hundreds of go-template functions. For example, to extract the major and minor version from the {{ .Version }} attribute, use the following template:

targets:
  - "components/terraform/{{ .Component }}/{{ (first 2 (splitList \".\" .Version)) | join \".\" }}"

Or to access an environment variable in the source attribute, use the following template:

source: "git::https://{{env "GITHUB_TOKEN"}}@github.com/some-org/some-private-repo/terraform/{{ .Component }}/{{ .Version }}.git?ref={{.Version}}"

This will enable vendoring to download the component into a versioned folder from a private repository, by reading the GitHub token from the GITHUB_TOKEN environment variable.

Hierarchical Imports in Vendoring Manifests

Use imports to split the main vendor.yaml manifest into smaller files for maintainability, or by their roles in the infrastructure.

For example, import separate manifests for networking, security, data management, CI/CD, and other layers:

imports:
  - "layers/networking"
  - "layers/security"
  - "layers/data"
  - "layers/analytics"
  - "layers/firewalls"
  - "layers/cicd"

Hierarchical imports are supported at many levels. For example, consider the following vendoring configurations:

vendor.yaml

apiVersion: atmos/v1
kind: AtmosVendorConfig
metadata:
  name: example-vendor-config
  description: Atmos vendoring manifest
spec:
  imports:
    - "vendor/vendor2"
    - "vendor/vendor3"

  sources:
    - component: "vpc"
      source: "oci://public.ecr.aws/cloudposse/components/terraform/stable/aws/vpc:{{.Version}}"
      version: "latest"
      targets:
        - "components/terraform/infra/vpc3"
    - component: "vpc-flow-logs-bucket"
      source: "github.com/cloudposse/terraform-aws-components.git//modules/vpc-flow-logs-bucket?ref={{.Version}}"
      version: "1.323.0"
      targets:
        - "components/terraform/infra/vpc-flow-logs-bucket/{{.Version}}"

vendor/vendor2.yaml

apiVersion: atmos/v1
kind: AtmosVendorConfig
metadata:
  name: example-vendor-config-2
  description: Atmos vendoring manifest
spec:
  imports:
    - "vendor/vendor4"

  sources:
    - component: "my-vpc1"
      source: "oci://public.ecr.aws/cloudposse/components/terraform/stable/aws/vpc:{{.Version}}"
      version: "1.0.2"
      targets:
        - "components/terraform/infra/my-vpc1"

vendor/vendor4.yaml

apiVersion: atmos/v1
kind: AtmosVendorConfig
metadata:
  name: example-vendor-config-4
  description: Atmos vendoring manifest
spec:
  imports:
    - "vendor/vendor5"

  sources:
    - component: "my-vpc4"
      source: "github.com/cloudposse/terraform-aws-components.git//modules/vpc?ref={{.Version}}"
      version: "1.319.0"
      targets:
        - "components/terraform/infra/my-vpc4"

When you execute the atmos vendor pull command, Atmos processes the import chain and the sources in the imported manifests in the order they are defined:

First, the main vendor.yaml file is read based on search paths
The vendor/vendor2 and vendor/vendor3 manifests (defined in the main vendor.yaml file) are imported
The vendor/vendor2 file is processed, and the vendor/vendor4 manifest is imported
The vendor/vendor4 file is processed, and the vendor/vendor5 manifest is imported
Etc.
Then all the sources from all the imported manifests are processed and the artifacts are downloaded into the paths defined by the targets

> atmos vendor pull

Processing vendor config file 'vendor.yaml'
Pulling sources for the component 'my-vpc6' from 'github.com/cloudposse/terraform-aws-components.git//modules/vpc?ref=1.315.0' into 'components/terraform/infra/my-vpc6'
Pulling sources for the component 'my-vpc5' from 'github.com/cloudposse/terraform-aws-components.git//modules/vpc?ref=1.317.0' into 'components/terraform/infra/my-vpc5'
Pulling sources for the component 'my-vpc4' from 'github.com/cloudposse/terraform-aws-components.git//modules/vpc?ref=1.319.0' into 'components/terraform/infra/my-vpc4'
Pulling sources for the component 'my-vpc1' from 'public.ecr.aws/cloudposse/components/terraform/stable/aws/vpc:1.0.2' into 'components/terraform/infra/my-vpc1'
Pulling sources for the component 'my-vpc2' from 'github.com/cloudposse/terraform-aws-components.git//modules/vpc?ref=1.320.0' into 'components/terraform/infra/my-vpc2'
Pulling sources for the component 'vpc' from 'public.ecr.aws/cloudposse/components/terraform/stable/aws/vpc:latest' into 'components/terraform/infra/vpc3'
Pulling sources for the component 'vpc-flow-logs-bucket' from 'github.com/cloudposse/terraform-aws-components.git//modules/vpc-flow-logs-bucket?ref=1.323.0' into 'components/terraform/infra/vpc-flow-logs-bucket/1.323.0'

Vendoring Multiple Versions of Components

Atmos supports vendoring multiple versions of the same component. This is useful when you need to pin stacks to different versions of the same component.

When vendoring multiple versions of the same component, use the {{ .Version }} template parameter in the targets attribute to ensure the components are vendored into different folders. Then update the stack configuration to point to the correct version of the component, and ensure the backend.s3.workspace_key_prefix is defined without the version to ensure you can seamlessly upgrade between versions of a component wihtout losing the state. By default the workspace_key_prefix incorporates the component relative path, which will include the version if it's included in the path.

components:
  terraform:
    # `vpc` is the Atmos component name
    vpc:
      # Backend configuration for the component
      backend:
        s3:
          # Ensure the path in the bucket is stable across versions
          # IMPORTANT: If not explicitly set, the `workspace_key_prefix` will include the version
          #            This will cause the state to be lost when upgrading between versions.
          workspace_key_prefix: vpc
      metadata:
        # Point to the Terraform component on the filesystem
        component: vpc/1.2.3

important

If not using the S3 backend, use the appropriate parameter for your backend to ensure the workspace is stable across versions of the component deployed.

Vendoring from OCI Registries

Atmos supports vendoring from OCI registries.

To specify a repository in an OCI registry, use the oci://<registry>/<repository>:tag scheme.

Artifacts from OCI repositories are downloaded as Docker image tarballs, then all the layers are processed, un-tarred and un-compressed, and the files are written into the directories specified by the targets attribute of each source.

For example, to vendor the vpc component from the public.ecr.aws/cloudposse/components/terraform/stable/aws/vpc AWS public ECR registry, use the following source:

vendor.yaml

# This is an example of how to download a Terraform component from an OCI registry (https://opencontainers.org), e.g. AWS Public ECR

apiVersion: atmos/v1
kind: AtmosVendorConfig
metadata:
  name: example-vendor-config
  description: Atmos vendoring manifest
spec:
  sources:
    - component: "vpc"
      source: "oci://public.ecr.aws/cloudposse/components/terraform/stable/aws/{{ .Component }}:{{ .Version }}"
      version: "latest"
      targets:
        - "components/terraform/{{ .Component }}"
      included_paths:
        - "**/*.tf"
        - "**/*.tfvars"
        - "**/*.md"
      excluded_paths: []

To vendor the vpc component, execute the following command:

atmos vendor pull -c vpc

How it works​

Vendoring Manifest​

vendor.yaml

Vendoring Manifest Schema​

vendor.yaml

Template Parameters​

Hierarchical Imports in Vendoring Manifests​

vendor.yaml

vendor/vendor2.yaml

vendor/vendor4.yaml

Vendoring Multiple Versions of Components​

Vendoring from OCI Registries​

vendor.yaml

How it works

Vendoring Manifest

Vendoring Manifest Schema

Template Parameters

Hierarchical Imports in Vendoring Manifests

Vendoring Multiple Versions of Components

Vendoring from OCI Registries