Skip to main content

Azure Blob Storage Support for !terraform.state Function

· 6 min read
PePe Amengual
PePe Amengual
Principal @ Slalom Build

Atmos now supports Azure Blob Storage backends in the !terraform.state YAML function. Read Terraform outputs directly from Azure-backed state files without initializing Terraform—bringing the same blazing-fast performance to Azure that S3 users already enjoy.

What's New​

The !terraform.state YAML function now supports Azure Blob Storage (azurerm) backends, joining existing support for S3 and local backends. This means you can retrieve Terraform outputs from Azure-backed state files at lightning speed—without the overhead of Terraform initialization.

Why This Matters​

Before this feature, if you were using Azure Blob Storage as your Terraform backend, you had two options for reading remote state:

  1. !terraform.output - Slow but reliable. Requires full Terraform initialization, provider downloads, and varfile generation.
  2. !store - Fast but requires extra setup. You had to manually configure external secret stores.

Now you can use !terraform.state with Azure backends—getting 10-100x faster performance compared to !terraform.output by reading directly from blob storage.

How It Works​

Backend Configuration​

Configure your Terraform component with an azurerm backend:

components:
  terraform:
    vpc:
      backend_type: azurerm
      backend:
        azurerm:
          storage_account_name: "mystorageaccount"
          container_name: "tfstate"
          key: "vpc.terraform.tfstate"

Reading State​

Use the !terraform.state function to read outputs:

components:
  terraform:
    eks-cluster:
      vars:
        # Get vpc_id output from vpc component in current stack
        vpc_id: !terraform.state vpc vpc_id

        # Get private subnet IDs
        subnet_ids: !terraform.state vpc private_subnet_ids

        # Get first subnet using YQ expression
        subnet_id: !terraform.state vpc .private_subnet_ids[0]

Cross-Stack References​

Reference components from different stacks:

components:
  terraform:
    tgw:
      vars:
        # Get VPC ID from production stack
        vpc_id: !terraform.state vpc plat-ue2-prod vpc_id

        # Use template for dynamic stack names
        vpc_id: !terraform.state vpc {{ printf "net-%s-%s" .vars.environment .vars.stage }} vpc_id

Authentication​

The Azure Blob Storage integration uses Azure DefaultAzureCredential, which supports multiple authentication methods automatically:

  1. Environment variables - AZURE_TENANT_ID, AZURE_CLIENT_ID, AZURE_CLIENT_SECRET
  2. Managed Identity - When running in Azure (AKS, VMs, Functions)
  3. Azure CLI credentials - az login
  4. Visual Studio Code credentials - Authenticated VS Code sessions

No additional configuration needed—just authenticate using your preferred method.

Workspace Handling​

Azure Blob Storage uses a specific naming convention for workspaces:

  • Default workspace: Uses the key as-is (e.g., terraform.tfstate)
  • Non-default workspaces: Appends workspace as suffix (e.g., terraform.tfstateenv:dev)

Atmos handles this automatically—you don't need to worry about the naming convention.

Example​

If you have:

  • Key: apimanagement.terraform.tfstate
  • Workspace: dev-wus3-apimanagement-be

Atmos will look for: apimanagement.terraform.tfstateenv:dev-wus3-apimanagement-be

Performance Benefits​

Before: Using !terraform.output​

$ time atmos terraform plan eks-cluster -s plat-ue2-dev

# Must initialize Terraform for each dependency
Initializing vpc component...
Downloading providers...
Generating backend config...
Generating varfiles...
Reading outputs...

real    2m34.521s

After: Using !terraform.state​

$ time atmos terraform plan eks-cluster -s plat-ue2-dev

# Direct blob storage access
Reading state from Azure Blob Storage...

real    0m3.142s

~50x faster in this example—and the speedup grows with infrastructure complexity.

Advanced Features​

YQ Expressions​

Use YQ expressions for complex data extraction:

vars:
  # Get nested map values
  db_endpoint: !terraform.state database .config_map.endpoint

  # String concatenation
  jdbc_url: !terraform.state 'postgres .master_hostname | "jdbc:postgresql://" + . + ":5432/events"'

  # Default values for unprovisioned components
  username: !terraform.state config .username // "default-user"

Caching​

Results are cached in memory per CLI execution:

vars:
  # All three calls use the same cached result
  sg_id_1: !terraform.state vpc security_group_id
  sg_id_2: !terraform.state vpc security_group_id
  sg_id_3: !terraform.state vpc {{ .stack }} security_group_id

The first call reads from Azure; subsequent calls return cached data instantly.

Error Handling​

  • Blob not found (404): Returns null (component not provisioned yet)
  • Permission denied (403): Returns clear error message
  • Network errors: Automatically retries up to 2 times with exponential backoff

Technical Details​

Implementation Highlights​

  • Azure SDK for Go - Uses official github.com/Azure/azure-sdk-for-go/sdk/storage/azblob package
  • Client caching - Azure Blob clients are cached per storage account/container
  • Retry logic - Automatic retry with exponential backoff for transient failures
  • Nil safety - Robust error handling prevents panics
  • Test coverage - Comprehensive unit tests with mocked Azure SDK
  • Cross-platform - Works on Linux, macOS, and Windows

Backend Configuration Options​

All standard Azure backend options are supported:

backend:
  azurerm:
    storage_account_name: "mystorageaccount"  # Required
    container_name: "tfstate"                  # Required
    key: "terraform.tfstate"                   # Optional (default: terraform.tfstate)
    # Authentication happens via DefaultAzureCredential

Migration Guide​

From !terraform.output to !terraform.state​

The syntax is identical—just replace !terraform.output with !terraform.state:

# Before
vpc_id: !terraform.output vpc vpc_id

# After
vpc_id: !terraform.state vpc vpc_id

From !store to !terraform.state​

Simplify your configuration by removing store setup:

# Before: Required store configuration
vpc_id: !store azurekeyvault plat-ue2-dev vpc vpc_id

# After: Direct state access
vpc_id: !terraform.state vpc vpc_id

Examples​

Basic Usage​

components:
  terraform:
    app:
      vars:
        # String output
        security_group_id: !terraform.state security-group id

        # List output
        subnet_ids: !terraform.state vpc private_subnet_ids

        # Map output
        config: !terraform.state config config_map

Cross-Region References​

components:
  terraform:
    replication:
      vars:
        # Reference component from different region
        primary_db: !terraform.state database {{ printf "%s-use1-%s" .vars.tenant .vars.stage }} endpoint

Disaster Recovery Scenarios​

components:
  terraform:
    failover:
      vars:
        # Primary region
        primary_vpc: !terraform.state vpc plat-ue2-prod vpc_id

        # DR region with default fallback
        dr_vpc: !terraform.state vpc plat-uw2-prod .vpc_id // "vpc-mock-dr"

Considerations​

  • Secrets exposure: Using !terraform.state with secrets will expose them in atmos describe output
  • Permission scoping: Ensure your Azure credentials have access to all referenced storage accounts
  • Cross-region access: Consider latency when reading state across regions
  • Cold starts: Components not yet provisioned return null (use YQ default values to handle this)

Try It Now​

Upgrade to the latest Atmos release and start using Azure Blob Storage backends:

# Check your version
atmos version

# Describe a component using Azure backend
atmos describe component vpc -s plat-ue2-dev

# Use !terraform.state in your stack configs
# (See examples above)

Documentation​

Get Involved​

We're building Atmos in the open and welcome your feedback:

Next up: Google Cloud Storage (GCS) backend support for !terraform.state. Stay tuned!