Brownfield Considerations

There are some considerations you should be aware of when adopting Atmos in a brownfield environment. Atmos works best when you adopt the Atmos mindset.

The term "brownfield" comes from urban planning and refers to the redevelopment of land that was previously used and may need cleaning or modification. As it relates to infrastructure, Brownfield development describes the development and deployment of new software systems in the presence of existing (legacy) software applications/systems. Anytime this happens, new software architectures must take into account and coexist with the existing software.

Atmos is not just a tool; it is a framework that provides a set of opinionated conventions, methodologies, design patterns, and best practices to ensure teams succeed with Terraform from the start. It can be hard to shoehorn existing systems that are not designed according to the Atmos mindset.

Decomposition: Not only do you have challenges around how to decompose your architecture, but also the difficulty of making changes to live systems.
Technical Debt: You may have significant technical debt that needs to be addressed
Knowledge Gaps: There may be gaps in knowledge within the team regarding Atmos conventions and methodologies.

By understanding these challenges, teams can better prepare for a smooth transition to using Atmos effectively.

Brownfield Development in Atmos

Atmos is easier for new organizations or "greenfield" environments because you need to architect Terraform according to our best practices to get all the benefits of Atmos. For example, when using our Terraform components, we frequently use Terraform Remote State to retrieve the outputs from other components.

This works well when you use our components but less so when you operate in a "brownfield" environment, for example, with an existing VPC, S3 bucket, or IAM role.

When you approach brownfield development with Atmos, begin by designing what your architecture could look like if you break it down into various pieces. Then devise a plan to decompose those pieces into components you implement as Terraform "root modules".

The process of configuring Atmos components and stacks for the existing, already provisioned resources, will depend on how easy or hard this decomposition will be. Working on and updating existing infrastructure rather than creating a new one from scratch, known as "greenfield" development, will always be more difficult.

The process needs to respect the existing systems' constraints while progressively introducing improvements and modern practices. This will ultimately lead to more robust, flexible, and efficient systems.

Remote State in Brownfield Development

So what happens when infrastructure wasn't provisioned by Atmos or predates your infrastructure? Then there's no way to retrieve that state in Terraform.

For this reason, we support something we refer to as the static remote state backend. Using the static remote state backend, you can populate a virtual state backend with the outputs as though it had been provisioned with Terraform. You can use this technique anytime you want to use the remote state functionality in Atmos, but when the remote state was provisioned elsewhere.

Hacking Remote State with `static` Backends

Atmos supports brownfield configuration by using the remote state of type static.

Suppose that we need to provision the vpc Terraform component and, instead of provisioning an S3 bucket for VPC Flow Logs, we want to use an existing bucket.

The vpc Terraform component needs the outputs from the vpc-flow-logs-bucket Terraform component to configure VPC Flow Logs.

Let's redesign the example with the vpc and vpc-flow-logs-bucket components described in Terraform Component Remote State and configure the static remote state for the vpc-flow-logs-bucket component to use an existing S3 bucket.

Examples

Configure the `vpc-flow-logs-bucket` Component

In the stacks/catalog/vpc-flow-logs-bucket.yaml file, add the following configuration for the vpc-flow-logs-bucket/defaults Atmos component:

stacks/catalog/vpc-flow-logs-bucket.yaml
components:
  terraform:
    vpc-flow-logs-bucket/defaults:
      metadata:
        type: abstract
      # Use `static` remote state to configure the attributes for an existing
      # S3 bucket for VPC Flow Logs
      remote_state_backend_type: static
      remote_state_backend:
        static:
          # ARN of the existing S3 bucket
          # `vpc_flow_logs_bucket_arn` is used as an input for the `vpc` component
          vpc_flow_logs_bucket_arn: "arn:aws:s3:::my-vpc-flow-logs-bucket"

In the stacks/ue2-dev.yaml stack config file, add the following config for the vpc-flow-logs-bucket-1 Atmos component in the ue2-dev Atmos stack:

stacks/ue2-dev.yaml
# Import the base Atmos component configuration from the `catalog`.
# `import` supports POSIX-style Globs for file names/paths (double-star `**` is supported).
# File extensions are optional (if not specified, `.yaml` is used by default).
import:
  - catalog/vpc-flow-logs-bucket

components:
  terraform:
    vpc-flow-logs-bucket-1:
      metadata:
        # Point to the Terraform component in `components/terraform` folder
        component: infra/vpc-flow-logs-bucket
        inherits:
          # Inherit all settings and variables from the
          # `vpc-flow-logs-bucket/defaults` base Atmos component
          - vpc-flow-logs-bucket/defaults

Configure and Provision the `vpc` Component

In the components/terraform/infra/vpc/remote-state.tf file, configure the remote-state Terraform module to obtain the remote state for the vpc-flow-logs-bucket-1 Atmos component:

components/terraform/infra/vpc/remote-state.tf
module "vpc_flow_logs_bucket" {
  count = local.vpc_flow_logs_enabled ? 1 : 0

  source  = "cloudposse/stack-config/yaml//modules/remote-state"
  version = "1.5.0"

  # Specify the Atmos component name (defined in YAML stack config files)
  # for which to get the remote state outputs
  component = var.vpc_flow_logs_bucket_component_name

  # `context` input is a way to provide the information about the stack (using the context
  # variables `namespace`, `tenant`, `environment`, `stage` defined in the stack config)
  context = module.this.context
}

In the components/terraform/infra/vpc/vpc-flow-logs.tf file, configure the aws_flow_log resource for the vpc Terraform component to use the remote state output vpc_flow_logs_bucket_arn from the vpc-flow-logs-bucket-1 Atmos component:

components/terraform/infra/vpc/vpc-flow-logs.tf
locals {
  enabled               = module.this.enabled
  vpc_flow_logs_enabled = local.enabled && var.vpc_flow_logs_enabled
}

resource "aws_flow_log" "default" {
  count = local.vpc_flow_logs_enabled ? 1 : 0

  # Use the remote state output `vpc_flow_logs_bucket_arn` of the `vpc_flow_logs_bucket` component
  log_destination = module.vpc_flow_logs_bucket[0].outputs.vpc_flow_logs_bucket_arn

  log_destination_type = var.vpc_flow_logs_log_destination_type
  traffic_type         = var.vpc_flow_logs_traffic_type
  vpc_id               = module.vpc.vpc_id

  tags = module.this.tags
}

In the stacks/catalog/vpc.yaml file, add the following default config for the vpc/defaults Atmos component:

stacks/catalog/vpc.yaml
components:
  terraform:
    vpc/defaults:
      metadata:
        # `metadata.type: abstract` makes the component `abstract`,
        # explicitly prohibiting the component from being deployed.
        # `atmos terraform apply` will fail with an error.
        # If `metadata.type` attribute is not specified, it defaults to `real`.
        # `real` components can be provisioned by `atmos` and CI/CD like Spacelift and Atlantis.
        type: abstract
      # Default variables, which will be inherited and can be overridden in the derived components
      vars:
        public_subnets_enabled: false
        nat_gateway_enabled: false
        nat_instance_enabled: false
        max_subnet_count: 3
        vpc_flow_logs_enabled: false
        vpc_flow_logs_log_destination_type: s3
        vpc_flow_logs_traffic_type: "ALL"

In the stacks/ue2-dev.yaml stack config file, add the following config for the vpc/1 Atmos component in the ue2-dev stack:

stacks/ue2-dev.yaml
# Import the base component configuration from the `catalog`.
# `import` supports POSIX-style Globs for file names/paths (double-star `**` is supported).
# File extensions are optional (if not specified, `.yaml` is used by default).
import:
  - catalog/vpc

components:
  terraform:
    vpc/1:
      metadata:
        # Point to the Terraform component in `components/terraform` folder
        component: infra/vpc
        inherits:
          # Inherit all settings and variables from the `vpc/defaults` base Atmos component
          - vpc/defaults
      vars:
        # Define variables that are specific for this component
        # and are not set in the base component
        name: vpc-1
        ipv4_primary_cidr_block: 10.8.0.0/18
        # Override the default variables from the base component
        vpc_flow_logs_enabled: true
        vpc_flow_logs_traffic_type: "REJECT"

        # Specify the name of the Atmos component that provides configuration
        # for the `infra/vpc-flow-logs-bucket` Terraform component
        vpc_flow_logs_bucket_component_name: vpc-flow-logs-bucket-1

Having the stacks configured as shown above, we can now provision the vpc/1 Atmos component in the ue2-dev stack by executing the following Atmos commands:

atmos terraform plan vpc/1 -s ue2-dev
atmos terraform apply vpc/1 -s ue2-dev

When the commands are executed, the vpc_flow_logs_bucket remote-state module detects that the vpc-flow-logs-bucket-1 component has the static remote state configured, and instead of reading its remote state from the S3 state bucket, it just returns the static values from the remote_state_backend.static section. The vpc_flow_logs_bucket_arn is then used as an input for the vpc component.

Brownfield Development in Atmos​

Remote State in Brownfield Development​

Hacking Remote State with static Backends​

Examples​

Configure the vpc-flow-logs-bucket Component​

Configure and Provision the vpc Component​