Vendor Manifest
The vendoring configuration is defined in the vendor.yaml manifest (vendor config file). The vendoring manifest is used to make copies of 3rd-party components, stacks, and other artifacts in your own repository.
It functions a little bit like the packages.json file in Node.js or the go.mod file in Go, but for infrastructure code.
How it works
Atmos searches for the vendoring manifest in the following locations and uses the first one found:
-
In the directory from which the
atmos vendor pullcommand is executed, usually in the root of the infrastructure repo -
In the directory pointed to by the
base_pathsetting in theatmos.yamlCLI config file
After defining the vendor.yaml manifest, all the remote artifacts can be downloaded by running the following command:
atmos vendor pull
To vendor a particular component or other artifact, execute the following command:
atmos vendor pull -c <component>
To vendor components and artifacts tagged with specific tags, execute the following command:
atmos vendor pull --tags <tag1>,<tag2>
Refer to atmos vendor pull CLI command for more details
Vendoring Manifest
To vendor remote artifacts, create a vendor.yaml file similar to the example below:
vendor.yaml
With this configuration, it would be possible to run the following commands:
Vendoring Manifest Schema
The vendor.yaml vendoring manifest supports Kubernetes-style YAML config to describe vendoring configuration for components, stacks,
and other artifacts. The file is placed into the directory from which the atmos vendor pull command is executed (usually the root of the repo).
versionThe
versionattribute is used to specify the version of the artifact to download. Theversionattribute is used in thesourceandtargetsattributes as a template parameter using{{ .Version }}.sourceThe
sourceattribute supports all protocols (local files, Git, Mercurial, HTTP, HTTPS, Amazon S3, Google GCP), and all the URL and archive formats as described in go-getter, and also theoci://scheme to download artifacts from OCI registries.See Vendor URL Syntax for complete documentation on supported URL formats, authentication, and subdirectory syntax.
IMPORTANT: Include the
{{ .Version }}parameter in yoursourceURI to ensure the correct version of the artifact is downloaded.For example, for
httpandhttpssources, use the following format:source: "github.com/cloudposse-terraform-components/aws-vpc-flow-logs-bucket.git?ref={{.Version}}"Authenticating to Private Git Repositories
Atmos provides automatic token injection for private repositories on GitHub, GitLab, and Bitbucket. This is the recommended approach for most users.
Automatic Token Injection (Recommended)The easiest and most secure way to authenticate to private Git repositories is to use Atmos's automatic token injection.
Step 1: Set the authentication token as an environment variable:
export GITHUB_TOKEN="your-personal-access-token"
# or
export ATMOS_GITHUB_TOKEN="your-personal-access-token"Step 2: Use simple URLs in your
vendor.yaml(no manual credentials):sources:
- component: "vpc"
source: "github.com/your-org/private-repo.git//terraform/vpc?ref={{.Version}}"
version: "1.0.0"
targets:
- "components/terraform/vpc"Step 3: Atmos automatically injects the token when downloading.
Supported environment variables:
- GitHub:
ATMOS_GITHUB_TOKENorGITHUB_TOKEN - GitLab:
ATMOS_GITLAB_TOKENorGITLAB_TOKEN - Bitbucket:
ATMOS_BITBUCKET_TOKENorBITBUCKET_TOKEN
Token injection settings in
atmos.yaml:inject_github_token: true(default: true) - Enables automatic GitHub token injectioninject_gitlab_token: true(default: false) - Enables automatic GitLab token injectioninject_bitbucket_token: true(default: false) - Enables automatic Bitbucket token injection
GitHub token injection is enabled by default. To disable it:
settings:
inject_github_token: falseTo enable GitLab or Bitbucket token injection:
settings:
inject_gitlab_token: true
inject_bitbucket_token: trueGitHub Actions Token ScopeThe
GITHUB_TOKENprovided by GitHub Actions is only valid for the current repository, or repositories marked asinternalwithin GitHub Enterprise organizations. For cross-repository access, provision a fine grained personal access token with the necessary permissions.Advanced: Manual Token Injection with Templates
For custom Git hosts or scenarios where you need explicit control over authentication, you can manually inject credentials using Go templates.
For GitHub (use correct authentication format):
sources:
- component: "vpc"
source: 'git::https://x-access-token:{{getenv "GITHUB_TOKEN"}}@github.com/org/private-repo.git?ref={{.Version}}'
version: "1.0.0"
targets:
- "components/terraform/vpc"For custom Git hosts:
sources:
- component: "vpc"
source: 'git::https://{{getenv "CUSTOM_GIT_TOKEN"}}@custom-git-host.com/org/repo.git?ref={{.Version}}'
version: "1.0.0"
targets:
- "components/terraform/vpc"YAML Quoting with Template FunctionsWhen using Go template functions like
{{getenv "TOKEN"}}that contain double quotes, you have two options to avoid YAML parsing errors:Option 1: Single-quoted YAML string (recommended)
source: 'git::https://x-access-token:{{getenv "GITHUB_TOKEN"}}@github.com/org/repo.git?ref={{.Version}}'Option 2: YAML folded scalar
source: >-
git::https://x-access-token:{{getenv "GITHUB_TOKEN"}}@github.com/org/repo.git?ref={{.Version}}What doesn't work:
source: "git::https://x-access-token:{{getenv "GITHUB_TOKEN"}}@github.com/org/repo.git?ref={{.Version}}"This is because YAML parsers interpret nested double quotes as string terminators. Template processing happens after YAML parsing, so the YAML must be valid before templates are evaluated.
Note: Atmos v1.194.0+ upgraded the YAML parser from v3.0.1 to v3.0.4, which enforces YAML spec compliance more strictly. Configurations with nested double quotes that may have worked in earlier versions (v1.193.x and below) will now fail YAML validation. Use single quotes or folded scalars as shown above.
GitHub Authentication FormatGitHub requires the username to be
x-access-tokenand the token to be the password:https://x-access-token:YOUR_TOKEN@github.com/...While
https://YOUR_TOKEN@github.com/...(token as username) may work with Git, using the documented format ensures compatibility.Why automatic injection is recommended:
- No need to manually specify
x-access-token:username format - Works automatically for GitHub, GitLab, and Bitbucket
- No YAML quoting complications
- Credentials are managed centrally via environment variables
- Easier to audit and rotate tokens
refPass the
refas a query string with either the tag, branch, or commit hash to download the correct version of the artifact. e.g.?ref={{.Version}}will pass theversionattribute to therefquery string.depthPass the
depthas a query string to download only the specified number of commits from the repository. e.g.?depth=1will download only the latest commit.
- GitHub:
targetsThe
targetsin each source supports absolute paths and relative paths (relative to thevendor.yamlfile). Note: if thetargetspaths are set as relative, and if thevendor.yamlfile is detected by Atmos using thebase_pathsetting inatmos.yaml, thetargetspaths will be considered relative to thebase_path. Multiple targets can be specified.included_pathsandexcluded_pathsincluded_pathsandexcluded_pathssupport POSIX-style greedy Globs for filenames/paths (double-star/globstar**is supported as well). For more details, see Vendoring with Globs.componentThe
componentattribute in each source is optional. It's used in theatmos vendor pull -- component <component>command if the component is passed in. In this case, Atmos will vendor only the specified component instead of vendoring all the artifacts configured in thevendor.yamlmanifest.sourceandtargetstemplatesThe
sourceandtargetsattributes support Go templates and Sprig Functions. This can be used to templatise thesourceandtargetspaths with the component name specified in thecomponentattribute and artifact versions specified in theversionattribute.Template Evaluation OrderTemplates in
sourceandtargetsare evaluated after YAML parsing. This means:- The
vendor.yamlfile itself must be valid YAML before template processing - Only the values of
sourceandtargetsfields are processed as templates - The file structure is not a template - it's parsed as standard YAML first
When using template functions that require double quotes (like
{{getenv "VAR"}}), wrap the entire YAML string value in single quotes to avoid nested double quote issues.Here's an advanced example showcasing how templates and Sprig functions can be used together with
targets:targets:
# Vendor a component into a major-minor versioned folder like 1.2
- "components/terraform/infra/vpc-flow-logs-bucket/{{ (first 2 (splitList \".\" .Version)) | join \".\" }}"- The
tagsThe
tagsin each source specifies a list of tags to apply to the component. This allows you to only vendor the components that have the specified tags by executing a commandatmos vendor pull --tags <tag1>,<tag2>importsThe
importssection defines the additional vendoring manifests that are merged into the main manifest. Hierarchical imports are supported at many levels (one vendoring manifest can import another, which in turn can import other manifests, etc.). Atmos processes all imports and all sources in the imported manifests in the order they are defined.noteThe imported file extensions are optional. Imports that do not include file extensions will default to the
.yamlextension.vendor.yaml
warningThe
globlibrary that Atmos uses to download remote artifacts does not treat the double-star**as including sub-folders. If the component's folder has sub-folders, and you need to vendor them, they have to be explicitly defined as in the following example.
Template Parameters
The vendor manifest supports basic template parameters, which is useful for versioning and other dynamic values. The following template parameters are supported:
{{ .Component }}Refers to the
componentattribute in the current section. Thecomponentattribute is used to specify the component name. This is useful to vendor components into folders by the same name.targets:
- "components/terraform/{{ .Component }}"{{ .Version }}Refers to the
versionattribute the current section. Theversionattribute is used to specify the version of the artifact to download. This is useful to version components into different folders.targets:
- "components/terraform/{{ .Component }}/{{ .Version }}"When stacks need to pin to different versions of the same component, the
{{ .Version }}template parameter can be used to ensure the components are vendored into different folders.
You can also use any of the hundreds of go-template functions. For example, to extract the major and minor version from the {{ .Version }} attribute, use the following template:
targets:
- "components/terraform/{{ .Component }}/{{ (first 2 (splitList \".\" .Version)) | join \".\" }}"
For accessing environment variables in templates, the most common use case is authentication to private Git repositories. We strongly recommend using Atmos's automatic token injection (see Authenticating to Private Git Repositories above) instead of manually injecting credentials with {{getenv}}. Automatic injection is simpler, more secure, and works without YAML quoting complications.
Hierarchical Imports in Vendoring Manifests
Use imports to split the main vendor.yaml manifest into smaller files for maintainability, or by their roles in the infrastructure.
For example, import separate manifests for networking, security, data management, CI/CD, and other layers:
imports:
- "layers/networking"
- "layers/security"
- "layers/data"
- "layers/analytics"
- "layers/firewalls"
- "layers/cicd"
Hierarchical imports are supported at many levels. For example, consider the following vendoring configurations:
vendor.yaml
vendor/vendor2.yaml
vendor/vendor4.yaml
When you execute the atmos vendor pull command, Atmos processes the import chain and the sources in the imported manifests in the order they
are defined:
- First, the main
vendor.yamlfile is read based on search paths - The
vendor/vendor2andvendor/vendor3manifests (defined in the mainvendor.yamlfile) are imported - The
vendor/vendor2file is processed, and thevendor/vendor4manifest is imported - The
vendor/vendor4file is processed, and thevendor/vendor5manifest is imported - Etc.
- Then all the sources from all the imported manifests are processed and the artifacts are downloaded into the paths defined by the
targets
Vendoring Multiple Versions of Components
Atmos supports vendoring multiple versions of the same component. This is useful when you need to pin stacks to different versions of the same component.
When vendoring multiple versions of the same component, use the {{ .Version }} template parameter in the targets attribute to ensure the components are vendored into different folders. Then update the stack configuration to point to the correct version of the component, and ensure the backend.s3.workspace_key_prefix is defined without the version to ensure you can seamlessly upgrade between versions of a component without losing the state. By default the workspace_key_prefix incorporates the component relative path, which will include the version if it's included in the path.
components:
terraform:
# `vpc` is the Atmos component name
vpc:
# Backend configuration for the component
backend:
s3:
# Ensure the path in the bucket is stable across versions
# IMPORTANT: If not explicitly set, the `workspace_key_prefix` will include the version
# This will cause the state to be lost when upgrading between versions.
workspace_key_prefix: vpc
metadata:
# Point to the Terraform component on the filesystem
component: vpc/1.2.3
If not using the S3 backend, use the appropriate parameter for your backend to ensure the workspace is stable across versions of the component deployed.
Vendoring from OCI Registries
Atmos supports vendoring from OCI registries.
To specify a repository in an OCI registry, use the oci://<registry>/<repository>:tag scheme.
Artifacts from OCI repositories are downloaded as Docker image tarballs, then all the layers are processed, un-tarred and un-compressed,
and the files are written into the directories specified by the targets attribute of each source.
OCI Authentication
Atmos uses the following precedence order for OCI registry authentication:
- Docker credentials (highest precedence) - Credentials from
docker loginstored in~/.docker/config.json - Environment variables - For GitHub Container Registry (ghcr.io):
- Token:
ATMOS_GITHUB_TOKENorGITHUB_TOKEN - Username:
ATMOS_GITHUB_USERNAME,GITHUB_ACTOR, orGITHUB_USERNAME
- Token:
- Anonymous - Fallback for public images
GitHub Container Registry (ghcr.io) Requirements
Unlike Git repository authentication, GitHub Container Registry requires both a username and a token:
- Username must be your actual GitHub username (not
x-access-token) - Token should be a GitHub Personal Access Token (PAT) or
GITHUB_TOKEN - Without a username, authentication will fail even if the token is set
Authentication methods:
# Method 1: Use docker login (recommended for local development)
docker login ghcr.io -u YOUR_USERNAME
# Enter your GitHub token when prompted
# Method 2: Use environment variables
export GITHUB_TOKEN=ghp_your_token
export GITHUB_ACTOR=your_username # Or ATMOS_GITHUB_USERNAME
# Method 3: Configure in atmos.yaml
# settings:
# github_username: "your_username"
GitHub Actions setup:
In GitHub Actions, both GITHUB_TOKEN and GITHUB_ACTOR are automatically available:
- name: Vendor from GHCR
run: atmos vendor pull -c vpc
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
# GITHUB_ACTOR is automatically set by GitHub Actions
For example, to vendor the vpc component from the public.ecr.aws/cloudposse/components/terraform/stable/aws/vpc
AWS public ECR registry, use the following source:
vendor.yaml
To vendor the vpc component, execute the following command:
Vendoring with Globs
In Atmos, glob patterns define which files and directories are included or excluded during vendoring. These patterns go beyond simple wildcard characters like *—they follow specific rules that dictate how paths are matched. Understanding the difference between greedy (**) and non-greedy (*) patterns, along with other advanced glob syntax, ensures precise control over vendoring behavior.
Understanding Wildcards, Ranges, and Recursion
Glob patterns in Atmos provide flexible and powerful matching, that's simpler to understand than regular expressions:
*(single asterisk)- Matches any sequence of characters within a single path segment.
- Example:
vendor/*.yamlmatchesvendor/config.yamlbut notvendor/subdir/config.yaml. **(double asterisk, also known as a "greedy glob")- Matches across multiple path segments recursively.
- Example:
vendor/**/*.yamlmatchesvendor/config.yaml,vendor/subdir/config.yaml, andvendor/deep/nested/config.yaml. ?(question mark)- Matches exactly one character in a path segment.
- Example:
file?.txtmatchesfile1.txtandfileA.txtbut notfile10.txt. [abc](character class)- Matches any single character inside the brackets.
- Example:
file[123].txtmatchesfile1.txt,file2.txt, andfile3.txt, but notfile4.txtorfile12.txt. [a-z](character range)- Matches any single character within the specified range.
- Example:
file[a-c].txtmatchesfilea.txt,fileb.txt, andfilec.txt. {a,b,c}(brace expansion)- Matches any of the comma-separated patterns.
- Example:
*.{jpg,png,gif}matchesimage.jpg,image.png, andimage.gif.
This distinction is important when excluding specific directories or files while vendoring.
Example: Excluding a Subdirectory
Consider the following configuration:
included_paths:
- "**/demo-library/**"
excluded_paths:
- "**/demo-library/**/stargazers/**"
How it works:
- The
included_pathsrule**/demo-library/**ensures all files insidedemo-library(at any depth) are vendored. - The
excluded_pathsrule**/demo-library/**/stargazers/**prevents any files insidestargazerssubdirectories from being vendored.
This means:
- All files within
demo-libraryexcept those inside anystargazerssubdirectory are vendored. - Any other files outside
stargazersare unaffected by this exclusion.
Example: A Non-Recursive Pattern That Doesn't Work
included_paths:
- "**/demo-library/*"
excluded_paths:
- "**/demo-library/**/stargazers/**"
In this case:
**/demo-library/*only matches immediate children ofdemo-library, not nested files or subdirectories.- This means
stargazers/itself could be matched, but its contents might not be explicitly excluded. - To correctly capture all subdirectories and files while still excluding stargazers, use
**/demo-library/**/*.
Using {...} for Multiple Extensions or Patterns
Curly braces {...} allow for expanding multiple patterns into separate glob matches. This is useful when selecting multiple file types or directories within a single glob pattern.
Example: Matching Multiple File Extensions
included_paths:
- "**/demo-library/**/*.{tf,md}"
This is equivalent to writing:
included_paths:
- "**/demo-library/**/*.tf"
- "**/demo-library/**/*.md"
The {tf,md} part expands to both *.tf and *.md, making the rule more concise.
Example: Excluding Multiple Directories
excluded_paths:
- "**/demo-library/**/{stargazers,archive}/**"
This excludes both:
**/demo-library/**/stargazers/****/demo-library/**/archive/**
Using {...} here prevents the need to write two separate exclusion rules.
Key Takeaways
- Use
**/for recursive matching to include everything inside a directory. - Use
*for single-segment matches, which won't include deeper subdirectories. - Use
{...}to match multiple extensions or directories within a single pattern. - Exclusion rules must match nested paths explicitly when trying to exclude deep directories.
By carefully combining included_paths, excluded_paths, and {...} expansion, you can precisely control which files are vendored while ensuring unwanted directories are omitted.