Template Data Sources

Advanced

Data sources in Atmos refer to external locations from which Atmos can fetch configuration data. Atmos supports all data sources supported by Gomplate. For example, you can use data sources to fetch JSON metadata from API endpoints or read from various backends like S3 Buckets, AWS SSM Parameter Store, HashiCorp Vault, and many others.

Data sources

Currently, Atmos supports all the Gomplate Datasources. More data sources will be added in the future (and this doc will be updated). All datasource configurations are defined in the templates.settings.gomplate.datasources section in atmos.yaml CLI config file or in the settings.templates.settings.gomplate.datasources section of any Atmos stack manifests.

The gomplate.datasources section is a map of Gomplate Datasource definitions.

The keys of the map are the data source names (aliases) that you will use to refer to them. For example, if you define a data source called foobar which has a property called tag, you could refer to it like this in a stack manifest: {{ (datasource "foobar").tag }}.

For example:

stack.yaml

terraform:
  vars:
    tags:
      provisioned_by_ip: '{{ (datasource "ip").ip }}'
      config1_tag: '{{ (datasource "config-1").tag }}'
      config2_service_name: '{{ (datasource "config-2").service.name }}'

The values in the map are data source definitions following this schema:

url

All data sources are defined as a URL.

As a refresher, a Gomplate Data Source URL is made up of the following components:

scheme://user@host.com:8080/path?query=string#fragment

headers

A map of HTTP request headers for the http data source. The keys of the map are the header names. The values of the map are lists of values for the header.

The following configuration will result in the accept: application/json HTTP header being sent with the HTTP request to the data source:

headers:
  accept:
    - "application/json"

Types of Data Sources

The following are the types of data sources are supported by Atmos via Gomplate.

aws+smp://: AWS Systems Manager Parameter Store is a key/value store that supports encryption and versioning.
aws+sm://: AWS Secrets Manager lets you store and retrieve secrets.
s3://: Amazon S3 provides object storage, which is convenient for stashing shared configurations.
consul://, consul+http://, consul+https://: Use HashiCorp Consul provides as a backend key/value store
env://: Environment variables can be used as data sources, although template functions might make more sense.
file://: Files can be read in any of the supported formats (JSON, YAML). Directories are also supported, just end the URL path with a /.
git://, git+file://, git+http://, git+https://, git+ssh://: Files can be read from a local or remote git repository, at specific branches or tags. Directory semantics are also supported.
gs://: Google Cloud Storage is the object storage service that is similar to AWS S3.
http://, https://: Retrieve data from HTTP/HTTPS endpoints. Custom HTTP headers can also be passed.
merge://: Merge two or more data sources together to produce the final value - useful for resolving defaults. Uses coll.Merge for merging.
stdin://: Read configuration data from standard input.
vault://, vault+http://, vault+https://: HashiCorp Vault is a popular open-source secret management platform.

Environment Variables

Some data sources might need environment variables that are different from the environment variables in Stack configuration. Environment variables may be passed to data sources when processing and executing templates by defining env map. It's supported in both the templates.settings section in atmos.yaml CLI config file and in the settings.templates.settings section in Atmos stack manifests.

For example:

atmos.yaml

settings:
  templates:
    settings:
      # Environment variables passed to `datasources` when evaluating templates
      # https://docs.gomplate.ca/datasources/#using-awssmp-datasources
      # https://docs.gomplate.ca/functions/aws/#configuring-aws
      # https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html
      env:
        AWS_PROFILE: "<AWS profile>"
        AWS_TIMEOUT: 2000

This is useful when executing data sources that need to authenticate to cloud APIs.

For more details, refer to:

Configuring Data Sources

For example, let's define the following Gomplate datasources in the global settings section (this will apply to all components in all stacks in the infrastructure).

First, enable Go templates and gomplate datasources in the atmos.yaml CLI config file:

atmos.yaml

templates:
  settings:
    # Enable `Go` templates in Atmos stack manifests
    enabled: true
    gomplate:
      # Enable Gomplate functions and data sources in `Go` templates in Atmos stack manifests
      enabled: true

Then, define the following data sources in the global settings section in an Atmos stack manifest:

stacks/orgs/acme/_defaults.yaml

settings:
  templates:
    settings:
      gomplate:
        # Timeout in seconds to execute the data sources
        timeout: 5
        # https://docs.gomplate.ca/datasources
        datasources:
          # 'http' data source
          # https://docs.gomplate.ca/datasources/#using-file-datasources
          ip:
            url: "https://api.ipify.org?format=json"
            # https://docs.gomplate.ca/datasources/#sending-http-headers
            # https://docs.gomplate.ca/usage/#--datasource-header-h
            headers:
              accept:
                - "application/json"
          # 'file' data sources
          # https://docs.gomplate.ca/datasources/#using-file-datasources
          config-1:
            url: "./config1.json"
          config-2:
            url: "file:///config2.json"
          # `aws+smp` AWS Systems Manager Parameter Store data source
          # https://docs.gomplate.ca/datasources/#using-awssmp-datasources
          secret-1:
            url: "aws+smp:///path/to/secret"
          # `aws+sm` AWS Secrets Manager datasource
          # https://docs.gomplate.ca/datasources/#using-awssm-data source
          secret-2:
            url: "aws+sm:///path/to/secret"
          # `s3` datasource
          # https://docs.gomplate.ca/datasources/#using-s3-data sources
          s3-config:
            url: "s3://mybucket/config/config.json"

After the above data sources are defined, you can use them in Atmos stack manifests like this:

terraform:
 vars:
   tags:
     tag1: '{{ (datasource "config-1").tag }}'
     service_name2: '{{ (datasource "config-2").service.name }}'
     service_name3: '{{ (datasource "s3-config").config.service_name }}'

components:
  terraform:
    vpc-1:
      settings:
        provisioned_by_ip: '{{ (datasource "ip").ip }}'
        secret-1: '{{ (datasource "secret-1").secret1.value }}'
      vars:
        enabled: '{{ (datasource "config-2").config.enabled }}'

Using templates in the URLs of `datasources`

Advanced

Let's suppose that your company uses a centralized software catalog to consolidate all tags for tagging all the cloud resources. The tags can include tags per account, per team, per service, billing tags, etc.

note

An example of such a centralized software catalog could be Backstage.

Let's also suppose that you have a service to read the tags from the centralized catalog and write them into an S3 bucket in one of your accounts. The bucket serves as a cache to not hit the external system's API with too many requests and not to trigger rate limiting.

And finally, let's say that in the bucket, you have folders per account (dev, prod, staging). Each folder has a JSON file with all the tags defined for all the cloud resources in the accounts.

We can then use the Gomplate S3 datasource to read the JSON file with the tags for each account and assign the tags to all cloud resources.

In atmos.yaml, we figure two evaluations steps of template processing:

atmos.yaml

templates:
  settings:
    enabled: true
    # Number of evaluations to process `Go` templates
    evaluations: 2
    gomplate:
      enabled: true

In an Atmos stack manifest, we define the environment variables in the env section (AWS profile with permissions to access the S3 bucket), and the s3-tags Gomplate datasource.

In the terraform.vars.tags section, we define all the tags that are returned from the call to the S3 datasource.

import:
  # Import the default configuration for all VPCs in the infrastructure
  - catalog/vpc/defaults

# Global settings
settings:
  templates:
    settings:
      # Environment variables passed to data sources when evaluating templates
      # https://docs.gomplate.ca/functions/aws/#configuring-aws
      # https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html
      env:
        # AWS profile with permissions to access the S3 bucket
        AWS_PROFILE: "<AWS profile>"
      gomplate:
        # Timeout in seconds to execute the data sources
        timeout: 5
        # https://docs.gomplate.ca/datasources
        datasources:
          # `s3` datasource
          # https://docs.gomplate.ca/datasources/#using-s3-datasources
          s3-tags:
            # The `url` uses a `Go` template with the delimiters `${ }`,
            # which is processed as first step in the template processing pipeline
            url: "s3://mybucket/{{ .vars.stage }}/tags.json"

# Global Terraform config
terraform:
  # Global variables that are used by all Atmos components
  vars:
    tags:
      atmos_component: "{{ .atmos_component }}"
      atmos_stack: "{{ .atmos_stack }}"
      terraform_component: "{{ .component }}"
      terraform_workspace: "{{ .workspace }}"
      devops_team: '{{`{{ (datasource "s3-tags").tags.devops_team }}`}}'
      billing_team: '{{`{{ (datasource "s3-tags").tags.billing_team }}`}}'
      service: '{{`{{ (datasource "s3-tags").tags.service }}`}}'

# Atmos component configurations
components:
  terraform:
    vpc/1:
      metadata:
        component: vpc  # Point to the Terraform component in `components/terraform/vpc` folder
        inherits:
          # Inherit from the `vpc/defaults` base Atmos component, which defines the default
          # configuration for all VPCs in the infrastructure.
          # The `vpc/defaults` base component is defined in the `catalog/vpc/defaults`
          # manifest (which is imported above).
          # This inheritance makes the `vpc/1` Atmos component config DRY.
          - "vpc/defaults"
      vars:
        name: "vpc-1"

When executing an Atmos command like atmos terraform apply vpc/1 -s plat-ue2-dev, the above template will be processed in two evaluation steps:

Evaluation 1:

datasources.s3-tags.url is set to s3://mybucket/dev/tags.json

the tags that use the datasource templates are set to the following:

devops_team: '{{ (datasource "s3-tags").tags.devops_team }}'
billing_team: '{{ (datasource "s3-tags").tags.billing_team }}'
service: '{{ (datasource "s3-tags").tags.service }}'

Evaluation 2:
- all s3-tags datasources get executed, the JSON file s3://mybucket/dev/tags.json with the tags for the dev account is downloaded from the S3 bucket, and the tags are parsed and assigned in the terraform.vars.tags section

After executing the two evaluation steps, the resulting tags for the Atmos component vpc/1 in the stack plat-ue2-dev would look like this:

atmos_component: vpc/1
atmos_stack: plat-ue2-dev
terraform_component: vpc
terraform_workspace: plat-ue2-dev-vpc-1
devops_team: dev_networking
billing_team: billing_net
service: net

The tags will be added to all the AWS resources provisioned by the vpc Terraform component in the plat-ue2-dev stack.

Data sources​

stack.yaml

Types of Data Sources​

Environment Variables​

atmos.yaml

Configuring Data Sources​

atmos.yaml

stacks/orgs/acme/_defaults.yaml

Using templates in the URLs of datasources​

atmos.yaml

Data sources

Types of Data Sources

Environment Variables

Configuring Data Sources

Using templates in the URLs of `datasources`