# Template Data Sources

_Advanced_

Data sources in Atmos refer to external locations from which Atmos can fetch configuration data.
Atmos supports all data sources supported by [Gomplate](https://docs.gomplate.ca/datasources).
For example, you can use data sources to fetch JSON metadata from API endpoints or read from various backends like S3 Buckets, AWS SSM Parameter Store, HashiCorp Vault, and many others.

## Data sources

Currently, Atmos supports all the [Gomplate Datasources](https://docs.gomplate.ca/datasources).
More data sources will be added in the future (and this doc will be updated).
All datasource configurations are defined in the `templates.settings.gomplate.datasources` section in `atmos.yaml` [CLI config file](/cli/configuration)
or in the `settings.templates.settings.gomplate.datasources` section of any [Atmos stack manifests](/learn/stacks).

The `gomplate.datasources` section is a map of [Gomplate Datasource](https://docs.gomplate.ca/datasources) definitions.

The keys of the map are the data source names (aliases) that you will use to refer to them. For example,
if you define a data source called `foobar` which has a property called `tag`, you could refer to it like this in a
stack manifest: `{{ (datasource "foobar").tag }}`.

For example:

**File:** `stack.yaml`

```yaml
terraform:
  vars:
    tags:
      provisioned_by_ip: '{{ (datasource "ip").ip }}'
      config1_tag: '{{ (datasource "config-1").tag }}'
      config2_service_name: '{{ (datasource "config-2").service.name }}'
```

The values in the map are data source definitions following this schema:

- **`url`**

  All data sources are defined as a [URL](https://docs.gomplate.ca/datasources/#url-format).

  As a refresher, a Gomplate Data Source URL is made up of the following components:
  ```plaintext
  scheme://user@host.com:8080/path?query=string#fragment
  ```
- **`headers`**

  A map of [HTTP request headers](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers) for
  the [`http` data source](https://docs.gomplate.ca/datasources/#sending-http-headers).
  The keys of the map are the header names. The values of the map are lists of values for the header.

  The following configuration will result in the
  [`accept: application/json`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept) HTTP header
  being sent with the HTTP request to the data source:
  ```yaml
  headers:
    accept:
      - "application/json"
  ```

## Types of Data Sources

The following are the types of data sources are supported by Atmos via [Gomplate](https://docs.gomplate.ca/datasources/#url-format).

- **`aws+smp://`**

  AWS Systems Manager Parameter Store is a key/value store that supports encryption and versioning.
- **`aws+sm://`**

  AWS Secrets Manager lets you store and retrieve secrets.
- **`s3://`**
  Amazon S3 provides object storage, which is convenient for stashing shared configurations.
- **`consul://`, `consul+http://`, `consul+https://`**
  Use HashiCorp Consul provides as a backend key/value store
- **`env://`**
  Environment variables can be used as data sources, although 
  [template functions](/functions/template)
   might make more sense.
- **`file://`**
  Files can be read in any of the supported formats (JSON, YAML). Directories are also supported, just end the URL path with a 
  `/`
  .
- **`git://`, `git+file://`, `git+http://`, `git+https://`, `git+ssh://`**

  Files can be read from a local or remote git repository, at specific branches or tags. Directory semantics are also supported.
- **`gs://`**

  Google Cloud Storage is the object storage service that is similar to AWS S3.
- **`http://`, `https://`**

  Retrieve data from HTTP/HTTPS endpoints. Custom HTTP headers can also be passed.
- **`merge://`**

  Merge two or more data sources together to produce the final value - useful for resolving defaults. Uses coll.Merge for merging.
- **`stdin://`**

  Read configuration data from standard input.
- **`vault://`, `vault+http://`, `vault+https://`**

  HashiCorp Vault is a popular open-source secret management platform.

## Environment Variables

Some data sources might need environment variables that are different from the environment variables in Stack configuration. Environment variables may be passed to data sources when processing and executing templates by defining `env` map.
It's supported in both the `templates.settings` section in `atmos.yaml` [CLI config file](/cli/configuration) and in the
`settings.templates.settings` section in [Atmos stack manifests](/learn/stacks).

For example:

**File:** `atmos.yaml`

```yaml
settings:
  templates:
    settings:
      # Environment variables passed to `datasources` when evaluating templates
      # https://docs.gomplate.ca/datasources/#using-awssmp-datasources
      # https://docs.gomplate.ca/functions/aws/#configuring-aws
      # https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html
      env:
        AWS_PROFILE: "<AWS profile>"
        AWS_TIMEOUT: 2000
```

This is useful when executing data sources that need to authenticate to cloud APIs.

For more details, refer to:

- [Configuring AWS](https://docs.gomplate.ca/functions/aws/#configuring-aws)
- [Configuring GCP](https://docs.gomplate.ca/functions/gcp/#configuring-gcp)

## Configuring Data Sources

For example, let's define the following Gomplate `datasources` in the global `settings` section (this will apply to all
components in all stacks in the infrastructure).

First, enable `Go` templates and `gomplate` datasources in the `atmos.yaml` CLI config file:

**File:** `atmos.yaml`

```yaml
templates:
  settings:
    # Enable `Go` templates in Atmos stack manifests
    enabled: true
    gomplate:
      # Enable Gomplate functions and data sources in `Go` templates in Atmos stack manifests
      enabled: true
```

Then, define the following data sources in the global `settings` section in an Atmos stack manifest:

**File:** `stacks/orgs/acme/_defaults.yaml`

```yaml
settings:
  templates:
    settings:
      gomplate:
        # Timeout in seconds to execute the data sources
        timeout: 5
        # https://docs.gomplate.ca/datasources
        datasources:
          # 'http' data source
          # https://docs.gomplate.ca/datasources/#using-file-datasources
          ip:
            url: "https://api.ipify.org?format=json"
            # https://docs.gomplate.ca/datasources/#sending-http-headers
            # https://docs.gomplate.ca/usage/#--datasource-header-h
            headers:
              accept:
                - "application/json"
          # 'file' data sources
          # https://docs.gomplate.ca/datasources/#using-file-datasources
          config-1:
            url: "./config1.json"
          config-2:
            url: "file:///config2.json"
          # `aws+smp` AWS Systems Manager Parameter Store data source
          # https://docs.gomplate.ca/datasources/#using-awssmp-datasources
          secret-1:
            url: "aws+smp:///path/to/secret"
          # `aws+sm` AWS Secrets Manager datasource
          # https://docs.gomplate.ca/datasources/#using-awssm-data source
          secret-2:
            url: "aws+sm:///path/to/secret"
          # `s3` datasource
          # https://docs.gomplate.ca/datasources/#using-s3-data sources
          s3-config:
            url: "s3://mybucket/config/config.json"
```

After the above data sources are defined, you can use them in Atmos stack manifests like this:

```
terraform:
 vars:
   tags:
     tag1: '{{ (datasource "config-1").tag }}'
     service_name2: '{{ (datasource "config-2").service.name }}'
     service_name3: '{{ (datasource "s3-config").config.service_name }}'

components:
  terraform:
    vpc-1:
      settings:
        provisioned_by_ip: '{{ (datasource "ip").ip }}'
        secret-1: '{{ (datasource "secret-1").secret1.value }}'
      vars:
        enabled: '{{ (datasource "config-2").config.enabled }}'
```

## Using templates in the URLs of `datasources`

_Advanced_

Let's suppose that your company uses a centralized software catalog to consolidate all tags for tagging all the cloud
resources. The tags can include tags per account, per team, per service, billing tags, etc.

:::note
An example of such a centralized software catalog could be [Backstage](https://backstage.io).
:::

Let's also suppose that you have a service to read the tags from the centralized catalog and write them into an S3
bucket in one of your accounts. The bucket serves as a cache to not hit the external system's API with too many requests
and not to trigger rate limiting.

And finally, let's say that in the bucket, you have folders per account (`dev`, `prod`, `staging`). Each folder has a JSON
file with all the tags defined for all the cloud resources in the accounts.

We can then use the [Gomplate S3 datasource](https://docs.gomplate.ca/datasources/#using-s3-datasources) to read the JSON
file with the tags for each account and assign the tags to all cloud resources.

In `atmos.yaml`, we figure two evaluations steps of template processing:

**File:** `atmos.yaml`

```yaml
templates:
  settings:
    enabled: true
    # Number of evaluations to process `Go` templates
    evaluations: 2
    gomplate:
      enabled: true
```

In an Atmos stack manifest, we define the environment variables in the `env` section (AWS profile with permissions to
access the S3 bucket), and the `s3-tags` Gomplate datasource.

In the `terraform.vars.tags` section, we define all the tags that are returned from the call to the S3 datasource.

```yaml
import:
  # Import the default configuration for all VPCs in the infrastructure
  - catalog/vpc/defaults

# Global settings
settings:
  templates:
    settings:
      # Environment variables passed to data sources when evaluating templates
      # https://docs.gomplate.ca/functions/aws/#configuring-aws
      # https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html
      env:
        # AWS profile with permissions to access the S3 bucket
        AWS_PROFILE: "<AWS profile>"
      gomplate:
        # Timeout in seconds to execute the data sources
        timeout: 5
        # https://docs.gomplate.ca/datasources
        datasources:
          # `s3` datasource
          # https://docs.gomplate.ca/datasources/#using-s3-datasources
          s3-tags:
            # The `url` uses a `Go` template with the delimiters `${ }`,
            # which is processed as first step in the template processing pipeline
            url: "s3://mybucket/{{ .vars.stage }}/tags.json"

# Global Terraform config
terraform:
  # Global variables that are used by all Atmos components
  vars:
    tags:
      atmos_component: "{{ .atmos_component }}"
      atmos_stack: "{{ .atmos_stack }}"
      terraform_component: "{{ .component }}"
      terraform_workspace: "{{ .workspace }}"
      devops_team: '{{`{{ (datasource "s3-tags").tags.devops_team }}`}}'
      billing_team: '{{`{{ (datasource "s3-tags").tags.billing_team }}`}}'
      service: '{{`{{ (datasource "s3-tags").tags.service }}`}}'

# Atmos component configurations
components:
  terraform:
    vpc/1:
      metadata:
        component: vpc  # Point to the Terraform component in `components/terraform/vpc` folder
        inherits:
          # Inherit from the `vpc/defaults` base Atmos component, which defines the default
          # configuration for all VPCs in the infrastructure.
          # The `vpc/defaults` base component is defined in the `catalog/vpc/defaults`
          # manifest (which is imported above).
          # This inheritance makes the `vpc/1` Atmos component config DRY.
          - "vpc/defaults"
      vars:
        name: "vpc-1"
```

When executing an Atmos command like `atmos terraform apply vpc/1 -s plat-ue2-dev`, the above template will be processed
in two evaluation steps:

- Evaluation 1:

  - `datasources.s3-tags.url` is set to `s3://mybucket/dev/tags.json`

  - the tags that use the `datasource` templates are set to the following:

    ```yaml
    devops_team: '{{ (datasource "s3-tags").tags.devops_team }}'
    billing_team: '{{ (datasource "s3-tags").tags.billing_team }}'
    service: '{{ (datasource "s3-tags").tags.service }}'
    ```

- Evaluation 2:
  - all `s3-tags` datasources get executed, the JSON file `s3://mybucket/dev/tags.json` with the tags
    for the `dev` account is downloaded from the S3 bucket, and the tags are parsed and assigned in the
    `terraform.vars.tags` section

After executing the two evaluation steps, the resulting tags for the Atmos component `vpc/1` in the stack `plat-ue2-dev`
would look like this:

```yaml
atmos_component: vpc/1
atmos_stack: plat-ue2-dev
terraform_component: vpc
terraform_workspace: plat-ue2-dev-vpc-1
devops_team: dev_networking
billing_team: billing_net
service: net
```

The tags will be added to all the AWS resources provisioned by the `vpc` Terraform component in the `plat-ue2-dev` stack.