Vendor Manifest
The vendoring configuration is defined in the vendor.yaml
manifest (vendor config file). The vendoring manifest is used to make copies of 3rd-party components, stacks, and other artifacts in your own repository.
It functions a little bit like the packages.json
file in Node.js or the go.mod
file in Go, but for infrastructure code.
How it works
Atmos searches for the vendoring manifest in the following locations and uses the first one found:
-
In the directory from which the
atmos vendor pull
command is executed, usually in the root of the infrastructure repo -
In the directory pointed to by the
base_path
setting in theatmos.yaml
CLI config file
After defining the vendor.yaml
manifest, all the remote artifacts can be downloaded by running the following command:
atmos vendor pull
To vendor a particular component or other artifact, execute the following command:
atmos vendor pull -c <component>
To vendor components and artifacts tagged with specific tags, execute the following command:
atmos vendor pull --tags <tag1>,<tag2>
Refer to atmos vendor pull
CLI command for more details
Vendoring Manifest
To vendor remote artifacts, create a vendor.yaml
file similar to the example below:
vendor.yaml
With this configuration, it would be possible to run the following commands:
Vendoring Manifest Schema
The vendor.yaml
vendoring manifest supports Kubernetes-style YAML config to describe vendoring configuration for components, stacks,
and other artifacts. The file is placed into the directory from which the atmos vendor pull
command is executed (usually the root of the repo).
version
The
version
attribute is used to specify the version of the artifact to download. Theversion
attribute is used in thesource
andtargets
attributes as a template parameter using{{ .Version }}
.source
The
source
attribute supports all protocols (local files, Git, Mercurial, HTTP, HTTPS, Amazon S3, Google GCP), and all the URL and archive formats as described in go-getter, and also theoci://
scheme to download artifacts from OCI registries.IMPORTANT: Include the
{{ .Version }}
parameter in yoursource
URI to ensure the correct version of the artifact is downloaded.For example, for
http
andhttps
sources, use the following format:source: "github.com/cloudposse/terraform-aws-components.git//modules/vpc-flow-logs-bucket?ref={{.Version}}"
For private Git repositories, prepend the URI with
git::
and use the following format to pass a environment variable with the GitHub token:source: "git::https://{{env "GITHUB_TOKEN"}}@github.com/some-org/some-private-repo/terraform/vpc.git?ref={{.Version}}"
Note, that
GITHUB_TOKEN
provided by GitHub Actions are only valid for the current repository, or repositories marked asinternal
within GitHub Enterprise organizations. For cross-repository access, make sure you provision a fine grained token with the necessary permissions.ref
Pass the
ref
as a query string with either the tag, branch, or commit hash to download the correct version of the artifact. e.g.?ref={{.Version}}
will pass theversion
attribute to theref
query string.depth
Pass the
depth
as a query string to download only the specified number of commits from the repository. e.g.?depth=1
will download only the latest commit.
targets
The
targets
in each source supports absolute paths and relative paths (relative to thevendor.yaml
file). Note: if thetargets
paths are set as relative, and if thevendor.yaml
file is detected by Atmos using thebase_path
setting inatmos.yaml
, thetargets
paths will be considered relative to thebase_path
. Multiple targets can be specified.included_paths
andexcluded_paths
included_paths
andexcluded_paths
support POSIX-style greedy Globs for filenames/paths (double-star/globstar**
is supported as well). For more details, see Vendoring with Globs.component
The
component
attribute in each source is optional. It's used in theatmos vendor pull -- component <component>
command if the component is passed in. In this case, Atmos will vendor only the specified component instead of vendoring all the artifacts configured in thevendor.yaml
manifest.source
andtargets
templatesThe
source
andtargets
attributes support Go templates and Sprig Functions. This can be used to templatise thesource
andtargets
paths with the component name specified in thecomponent
attribute and artifact versions specified in theversion
attribute.Here's an advanced example showcasing how templates and Sprig functions can be used together with
targets
:targets:
# Vendor a component into a major-minor versioned folder like 1.2
- "components/terraform/infra/vpc-flow-logs-bucket/{{ (first 2 (splitList \".\" .Version)) | join \".\" }}"tags
The
tags
in each source specifies a list of tags to apply to the component. This allows you to only vendor the components that have the specified tags by executing a commandatmos vendor pull --tags <tag1>,<tag2>
imports
The
imports
section defines the additional vendoring manifests that are merged into the main manifest. Hierarchical imports are supported at many levels (one vendoring manifest can import another, which in turn can import other manifests, etc.). Atmos processes all imports and all sources in the imported manifests in the order they are defined.noteThe imported file extensions are optional. Imports that do not include file extensions will default to the
.yaml
extension.vendor.yaml
warningThe
glob
library that Atmos uses to download remote artifacts does not treat the double-star**
as including sub-folders. If the component's folder has sub-folders, and you need to vendor them, they have to be explicitly defined as in the following example.
Template Parameters
The vendor manifest supports basic template parameters, which is useful for versioning and other dynamic values. The following template parameters are supported:
{{ .Component }}
Refers to the
component
attribute in the current section. Thecomponent
attribute is used to specify the component name. This is useful to vendor components into folders by the same name.targets:
- "components/terraform/{{ .Component }}"{{ .Version }}
Refers to the
version
attribute the current section. Theversion
attribute is used to specify the version of the artifact to download. This is useful to version components into different folders.targets:
- "components/terraform/{{ .Component }}/{{ .Version }}"When stacks need to pin to different versions of the same component, the
{{ .Version }}
template parameter can be used to ensure the components are vendored into different folders.
You can also use any of the hundreds of go-template functions. For example, to extract the major and minor version from the {{ .Version }}
attribute, use the following template:
targets:
- "components/terraform/{{ .Component }}/{{ (first 2 (splitList \".\" .Version)) | join \".\" }}"
Or to access an environment variable in the source
attribute, use the following template:
source: "git::https://{{env "GITHUB_TOKEN"}}@github.com/some-org/some-private-repo/terraform/{{ .Component }}/{{ .Version }}.git?ref={{.Version}}"
This will enable vendoring to download the component into a versioned folder from a private repository, by reading the GitHub token from the GITHUB_TOKEN
environment variable.
Hierarchical Imports in Vendoring Manifests
Use imports
to split the main vendor.yaml
manifest into smaller files for maintainability, or by their roles in the infrastructure.
For example, import separate manifests for networking, security, data management, CI/CD, and other layers:
imports:
- "layers/networking"
- "layers/security"
- "layers/data"
- "layers/analytics"
- "layers/firewalls"
- "layers/cicd"
Hierarchical imports are supported at many levels. For example, consider the following vendoring configurations:
vendor.yaml
vendor/vendor2.yaml
vendor/vendor4.yaml
When you execute the atmos vendor pull
command, Atmos processes the import chain and the sources in the imported manifests in the order they
are defined:
- First, the main
vendor.yaml
file is read based on search paths - The
vendor/vendor2
andvendor/vendor3
manifests (defined in the mainvendor.yaml
file) are imported - The
vendor/vendor2
file is processed, and thevendor/vendor4
manifest is imported - The
vendor/vendor4
file is processed, and thevendor/vendor5
manifest is imported - Etc.
- Then all the sources from all the imported manifests are processed and the artifacts are downloaded into the paths defined by the
targets
Vendoring Multiple Versions of Components
Atmos supports vendoring multiple versions of the same component. This is useful when you need to pin stacks to different versions of the same component.
When vendoring multiple versions of the same component, use the {{ .Version }}
template parameter in the targets
attribute to ensure the components are vendored into different folders. Then update the stack configuration to point to the correct version of the component, and ensure the backend.s3.workspace_key_prefix
is defined without the version to ensure you can seamlessly upgrade between versions of a component wihtout losing the state. By default the workspace_key_prefix
incorporates the component
relative path, which will include the version if it's included in the path.
components:
terraform:
# `vpc` is the Atmos component name
vpc:
# Backend configuration for the component
backend:
s3:
# Ensure the path in the bucket is stable across versions
# IMPORTANT: If not explicitly set, the `workspace_key_prefix` will include the version
# This will cause the state to be lost when upgrading between versions.
workspace_key_prefix: vpc
metadata:
# Point to the Terraform component on the filesystem
component: vpc/1.2.3
If not using the S3 backend, use the appropriate parameter for your backend to ensure the workspace is stable across versions of the component deployed.
Vendoring from OCI Registries
Atmos supports vendoring from OCI registries.
To specify a repository in an OCI registry, use the oci://<registry>/<repository>:tag
scheme.
Artifacts from OCI repositories are downloaded as Docker image tarballs, then all the layers are processed, un-tarred and un-compressed,
and the files are written into the directories specified by the targets
attribute of each source
.
For example, to vendor the vpc
component from the public.ecr.aws/cloudposse/components/terraform/stable/aws/vpc
AWS public ECR registry, use the following source
:
vendor.yaml
To vendor the vpc
component, execute the following command:
Vendoring with Globs
In Atmos, glob patterns define which files and directories are included or excluded during vendoring. These patterns go beyond simple wildcard characters like *
—they follow specific rules that dictate how paths are matched. Understanding the difference between greedy (**
) and non-greedy (*
) patterns, along with other advanced glob syntax, ensures precise control over vendoring behavior.
Understanding Wildcards, Ranges, and Recursion
Glob patterns in Atmos provide flexible and powerful matching, that's simpler to understand than regular expressions:
*
(single asterisk)- Matches any sequence of characters within a single path segment.
- Example:
vendor/*.yaml
matchesvendor/config.yaml
but notvendor/subdir/config.yaml
. **
(double asterisk, also known as a "greedy glob")- Matches across multiple path segments recursively.
- Example:
vendor/**/*.yaml
matchesvendor/config.yaml
,vendor/subdir/config.yaml
, andvendor/deep/nested/config.yaml
. ?
(question mark)- Matches exactly one character in a path segment.
- Example:
file?.txt
matchesfile1.txt
andfileA.txt
but notfile10.txt
. [abc]
(character class)- Matches any single character inside the brackets.
- Example:
file[123].txt
matchesfile1.txt
,file2.txt
, andfile3.txt
, but notfile4.txt
orfile12.txt
. [a-z]
(character range)- Matches any single character within the specified range.
- Example:
file[a-c].txt
matchesfilea.txt
,fileb.txt
, andfilec.txt
. {a,b,c}
(brace expansion)- Matches any of the comma-separated patterns.
- Example:
*.{jpg,png,gif}
matchesimage.jpg
,image.png
, andimage.gif
.
This distinction is important when excluding specific directories or files while vendoring.
Example: Excluding a Subdirectory
Consider the following configuration:
included_paths:
- "**/demo-library/**"
excluded_paths:
- "**/demo-library/**/stargazers/**"
How it works:
- The
included_paths
rule**/demo-library/**
ensures all files insidedemo-library
(at any depth) are vendored. - The
excluded_paths
rule**/demo-library/**/stargazers/**
prevents any files insidestargazers
subdirectories from being vendored.
This means:
- All files within
demo-library
except those inside anystargazers
subdirectory are vendored. - Any other files outside
stargazers
are unaffected by this exclusion.
Example: A Non-Recursive Pattern That Doesn't Work
included_paths:
- "**/demo-library/*"
excluded_paths:
- "**/demo-library/**/stargazers/**"
In this case:
**/demo-library/*
only matches immediate children ofdemo-library
, not nested files or subdirectories.- This means
stargazers/
itself could be matched, but its contents might not be explicitly excluded. - To correctly capture all subdirectories and files while still excluding stargazers, use
**/demo-library/**/*
.
Using {...}
for Multiple Extensions or Patterns
Curly braces {...}
allow for expanding multiple patterns into separate glob matches. This is useful when selecting multiple file types or directories within a single glob pattern.
Example: Matching Multiple File Extensions
included_paths:
- "**/demo-library/**/*.{tf,md}"
This is equivalent to writing:
included_paths:
- "**/demo-library/**/*.tf"
- "**/demo-library/**/*.md"
The {tf,md}
part expands to both *.tf
and *.md
, making the rule more concise.
Example: Excluding Multiple Directories
excluded_paths:
- "**/demo-library/**/{stargazers,archive}/**"
This excludes both:
**/demo-library/**/stargazers/**
**/demo-library/**/archive/**
Using {...}
here prevents the need to write two separate exclusion rules.
Key Takeaways
- Use
**/
for recursive matching to include everything inside a directory. - Use
*
for single-segment matches, which won't include deeper subdirectories. - Use
{...}
to match multiple extensions or directories within a single pattern. - Exclusion rules must match nested paths explicitly when trying to exclude deep directories.
By carefully combining included_paths
, excluded_paths
, and {...}
expansion, you can precisely control which files are vendored while ensuring unwanted directories are omitted.