# Scale Terraform Workflows with Atmos Today I want to share with you a little bit about how you can scale terraform workflows using a tool we wrote called Atmos. But first. I want to tell a story. It's about business and what differentiates great businesses from the rest. In order to understand Atmos, we first need to understand the problem it solves. The success of every business hinges on how well it's able to capture its processes and improve them over time. Whether it's lead generation, sales, marketing, or engineering - it doesn't matter. The best companies have the best processes. (without needing to be superheroes) Those processes are captured by workflows. These effective, proven workflows are what give teams super powers, without needing to be super heroes to use them. The best workflows can be run by anyone, anywhere - both locally and in CI. They are easily understood by the team. They ensure fewer mistakes are made. They are written down and documented. And they are constantly iterated on and optimized over time. Atmos is the tool that DevOps teams use to capture their workflows. It's designed specifically for infrastructure automation and helps teams codify their processes in a repeatable, shareable way. It's the differentiator for businesses who succeed at implementing DevOps from those who don't. There are many tools out there, but Atmos was built from real-world experience helping hundreds of companies scale their infrastructure. Cloud Posse is a DevOps Accelerator that helps enterprise and funded startups build and own their AWS infrastructure. We work alongside their teams, building infrastructure together and teaching them best practices along the way. In our quest for scalable infrastructure solutions, we built Atmos to solve our own problems. We needed to support many customer teams concurrently. We needed to automate our deliverables so teams could own them after we left. We needed something extensible that could handle unknown requirements. And we needed it to be easy to use and stable - which means testable. Atmos solves the problem of terraliths - those massive, monolithic Terraform configurations that become unmaintainable. Stacks are defined in YAML, which enables hierarchical configurations that reduce complexity through inheritance. Teams can import DRY configurations from catalogs. You can run workflows anywhere you can run a command. And it's a single binary, cross-compiled for every architecture. So why did we choose YAML? It's a universal language that everyone knows. It's a simple, extensible markup language perfect for defining a declarative DSL. It easily supports imports, inheritance, deep merging, and policies. Unlike HCL, it's tool agnostic - not tied to Terraform. Every language can read it, even remotely via HTTP. Every language can write it, so you can generate configurations. And YAML anchors provide additional reusability. Configuration at Scale Requires Policies & Validation Use OPA to lay down the law and enforce it Examples: We need guardrails for large-scale configuration. Policies and validation are essential when you have many teams and environments. Use OPA to lay down the law and enforce it. For example: never allow certain values in production, never allow two conflicting things in staging, restrict which teams can deploy which components, and ensure production never uses unstable component versions. Inheritance in configuration brings immense power. You can make configuration hierarchical with imports and deep merging. Organize it any way that makes sense for your organization. Define logical groups of configuration. Define something once and reuse it everywhere. And most importantly, you can know exactly where configuration differs between environments. However, a general tool will always suffer for a specialized use case, even Atmos and its custom workflows. So that's why we wrote it in Golang - for performance and portability. And we implemented opinionated workflows for the tools we use like Terraform and Helmfile. We also acknowledged that not everyone knows Go, so we made it extensible through YAML configuration. Terraform, on its own, falls short in scaling for enterprise needs. This is a controversial statement, but it's true. Terraform is an amazing tool, but it wasn't designed for the scale and complexity that modern organizations require. Let me enumerate the specific challenges with Terraform. GitOps and CI/CD for teams is non-trivial to set up correctly. Configuration is not DRY - you end up copying and pasting everywhere. Terraliths are an anti-pattern, yet teams keep creating them. Root modules aren't easily shared. There's no native support for multiple root modules or dependency ordering. Backend configurations can't be parameterized. And Terraform lacks built-in guardrails and policy controls that enterprises need. Terraform plus Atmos is the answer to these challenges. Atmos provides opinionated workflows for Terraform. It establishes conventions and patterns to manage Terraform at scale. It helps you break terraliths into smaller, manageable pieces called components. It enables team collaboration through catalogs and libraries. It enforces guardrails with OPA and JSON Schema validation. And there's zero vendor lock-in - it's all open source. Before I go on, let's define our terminology. Components are opinionated Terraform root modules - the deployable units of infrastructure. Child modules are modules called by other modules, like the building blocks that components use. Stacks are the combination of components into an architecture - they represent an environment or deployment target. And catalogs are collections of reusable configurations that teams can import and customize. With Atmos, we supercharge Terraform in several key ways. You can write components once and reuse them any number of times. It's vanilla Terraform HCL without customizations - no proprietary syntax. Use our Terraform provider to read YAML configurations from anywhere. Easily deploy across any number of regions without touching a line of HCL. Duplicate entire stacks by just copying a single YAML file. And you can generate YAML configurations programmatically. So now let's go a little bit deeper. Here's an example filesystem layout for organizing stacks. The catalog folder contains reusable imports with default configurations. Mixins set defaults for things like regions. You can break configurations down by team. And the orgs folder contains the actual stack definitions for each environment - like production and staging. Here's how the components, also known as Terraform root modules, can be organized. We recommend separating stable components from edge components. Each component is a self-contained Terraform root module with its own main.tf, variables.tf, outputs.tf, and README. You can also have components for other tools like Helmfile and Packer. This is what it looks like. Imports allow us to pull in configuration from catalogs - keeping things DRY. The configurations are 100% declarative and support deep merging with multiple inheritance. Then you define your components and can easily override settings at any level. Here we're overriding just the CIDR block for the VPC. Atmos also supports validation at the component level. You can define validation rules using JSON Schema or OPA policies. All validation steps must succeed before the component can be deployed. This is how you enforce those guardrails we talked about earlier - right in the stack configuration. Remember those workflows we talked about at the beginning? Here's what they look like in Atmos. You define a workflow with a description and a series of steps. Each step is a command that Atmos will execute in order. This example brings up an EKS cluster by first applying the VPC, then EKS, then the ALB controller. Users run it with a simple command. You can also extend Atmos with custom subcommands. This example adds a command to generate a PNG graph of Terraform resources. You define the command name, description, arguments, and flags - all in YAML. This is how teams can capture their unique workflows and tooling within Atmos, making it available to everyone on the team. Businesses must encapsulate their processes into efficient workflows to succeed. Terraform alone isn't the silver bullet for infrastructure at scale - it needs help. Terraform configuration becomes difficult to manage as you grow. Atmos provides the conventions, patterns, and tooling to make managing those configurations easier and more maintainable. With Atmos, you're not alone. You're backed by a passionate community of DevOps practitioners. It's open source under the Apache 2 license with no strings attached. It's under active development with hundreds of GitHub stars. We've built an ecosystem of tools and modules that have been downloaded tens of millions of times. Visit atmos.tools to learn more, and join us on Slack at slack.cloudposse.com. We invite you to join us!