Skip to main content

Performance Profiling

Atmos includes built-in support for performance profiling using Go's standard pprof tool. Profiling helps identify performance bottlenecks, memory usage patterns, and CPU hotspots to optimize Atmos operations.

What is pprof?​

pprof is Go's built-in profiling tool that collects runtime performance data from Go applications. It can capture several types of profiles:

  • CPU Profile: Shows where your program spends CPU time
  • Heap Profile: Shows current heap memory allocation patterns
  • Allocs Profile: Shows all memory allocations since program start
  • Goroutine Profile: Shows active goroutines and their call stacks
  • Block Profile: Shows operations that led to blocking on synchronization primitives
  • Mutex Profile: Shows lock contention patterns
  • Thread Create Profile: Shows stack traces that led to thread creation
  • Trace Profile: Shows detailed execution traces for performance analysis

Atmos Profiling Integration​

Atmos integrates pprof in two modes:

File-Based Profiling​

File-based profiling captures profiles directly to a file, which is ideal for CLI tools that exit quickly. This mode automatically enables profiling when a file path is specified. You can specify the type of profile to collect using the --profile-type flag.

# CPU profiling (default)
atmos terraform plan vpc -s plat-ue2-dev --profile-file=cpu.prof

# Memory heap profiling
atmos terraform plan vpc -s plat-ue2-dev --profile-file=heap.prof --profile-type=heap

# Execution trace profiling
atmos terraform plan vpc -s plat-ue2-dev --profile-file=trace.out --profile-type=trace

# Goroutine profiling
atmos terraform plan vpc -s plat-ue2-dev --profile-file=goroutine.prof --profile-type=goroutine

File-based profiling output

INFO Profiling started type=cpu file=cpu.prof
INFO Profiling completed type=cpu file=cpu.prof

Server-Based Profiling​

Server-based profiling starts an HTTP server that serves pprof endpoints. This mode is useful for long-running operations or when you need access to multiple profile types.

atmos terraform plan vpc -s plat-ue2-dev --profiler-enabled

Server-based profiling output

INFO Profiler server available at: url=http://localhost:6060/debug/pprof/

Configuration​

CLI Flags​

--profile-file
Write profiling data to the specified file (enables profiling automatically)
--profile-type
Type of profile to collect when using --profile-file. Options: cpu, heap, allocs, goroutine, block, mutex, threadcreate, trace (default: cpu)
--profiler-enabled
Enable pprof profiling server (default: false)
--profiler-host
Host address for profiling server (default: localhost)
--profiler-port
Port for profiling server (default: 6060)

Environment Variables​

All profiling flags can be set using environment variables for easier CI/CD integration:

ATMOS_PROFILER_ENABLED
Enable pprof profiling server (equivalent to --profiler-enabled)
ATMOS_PROFILER_HOST
Host address for profiling server (equivalent to --profiler-host)
ATMOS_PROFILER_PORT
Port for profiling server (equivalent to --profiler-port)
ATMOS_PROFILE_FILE
File path for file-based profiling (equivalent to --profile-file)
ATMOS_PROFILE_TYPE
Profile type for file-based profiling (equivalent to --profile-type)

Examples:

# File-based profiling via environment variables
export ATMOS_PROFILE_FILE=cpu.prof
export ATMOS_PROFILE_TYPE=cpu
atmos terraform plan vpc -s prod

# Server-based profiling via environment variables
export ATMOS_PROFILER_ENABLED=true
export ATMOS_PROFILER_PORT=8080
atmos terraform apply vpc -s prod

# Combined environment and CLI usage
export ATMOS_PROFILER_HOST=0.0.0.0
atmos terraform plan vpc -s prod --profiler-enabled --profiler-port=9090

Configuration File​

You can also enable profiling through your atmos.yaml configuration:

atmos.yaml

profiler:
  enabled: true
  host: "localhost"
  port: 6060
  file: "profile.out"           # Optional: file-based profiling
  profile_type: "cpu"           # Optional: profile type for file-based profiling

Configuration Precedence:

  1. Command-line flags (highest priority)
  2. Environment variables
  3. Configuration file (atmos.yaml)
  4. Default values (lowest priority)

Analyzing Profile Data​

File-Based Profiles​

After capturing a profile file, you can analyze it using Go's pprof tool. The analysis method depends on the profile type:

CPU and Memory Profiles:

# Interactive text mode
go tool pprof cpu.prof
go tool pprof heap.prof

# Web interface (requires Graphviz: brew install graphviz)
go tool pprof -http=:8080 cpu.prof
go tool pprof -http=:8080 heap.prof

# Direct text output
go tool pprof -top cpu.prof
go tool pprof -top heap.prof

Trace Profiles:

# Use go tool trace for execution traces
go tool trace trace.out

# This opens a web interface showing:
# - Timeline view of goroutines
# - Network blocking profile
# - Synchronization blocking profile
# - System call blocking profile

Sample pprof output

(pprof) top
Showing nodes accounting for 230ms, 95.83% of 240ms total
Dropped 15 nodes (cum <= 1.20ms)
flat flat% sum% cum cum%
80ms 33.33% 33.33% 80ms 33.33% github.com/cloudposse/atmos/internal/exec.processStackConfig
60ms 25.00% 58.33% 60ms 25.00% gopkg.in/yaml.v3.(*Decoder).Decode
40ms 16.67% 75.00% 40ms 16.67% github.com/cloudposse/atmos/pkg/utils.ProcessTmplWithDatasources
30ms 12.50% 87.50% 30ms 12.50% encoding/json.(*Decoder).Decode
20ms 8.33% 95.83% 20ms 8.33% github.com/cloudposse/atmos/pkg/stack.ProcessStackConfig

Server-Based Profiles​

When using server mode, you can access different profile types through HTTP endpoints:

# CPU profile (30-second sample)
go tool pprof http://localhost:6060/debug/pprof/profile

# Memory allocation profile
go tool pprof http://localhost:6060/debug/pprof/heap

# Goroutine profile
go tool pprof http://localhost:6060/debug/pprof/goroutine

# Mutex contention profile
go tool pprof http://localhost:6060/debug/pprof/mutex

You can also access the web interface directly:

open http://localhost:6060/debug/pprof/

Common Profiling Scenarios​

Performance Optimization​

Profile slow Atmos operations to identify bottlenecks:

# Profile CPU usage in a slow terraform plan operation
atmos terraform plan large-component -s prod --profile-file=slow-plan.prof --profile-type=cpu

# Profile memory usage
atmos terraform plan large-component -s prod --profile-file=memory.prof --profile-type=heap

# Profile detailed execution trace
atmos terraform plan large-component -s prod --profile-file=trace.out --profile-type=trace

# Analyze the results
go tool pprof -http=:8080 slow-plan.prof
go tool pprof -http=:8080 memory.prof
go tool trace trace.out

Memory Usage Analysis​

Identify memory-intensive operations using different approaches:

File-based memory profiling:

# Capture heap profile to file
atmos describe stacks --profile-file=heap.prof --profile-type=heap

# Capture allocation profile to file
atmos describe stacks --profile-file=allocs.prof --profile-type=allocs

# Analyze the results
go tool pprof -http=:8080 heap.prof
go tool pprof -http=:8080 allocs.prof

Server-based memory profiling:

# Start server mode for live memory profiling
atmos describe stacks --profiler-enabled

# In another terminal, capture memory profile
go tool pprof http://localhost:6060/debug/pprof/heap
go tool pprof http://localhost:6060/debug/pprof/allocs

Custom Server Configuration​

Run profiler on a different host/port for security or networking requirements:

atmos terraform apply vpc -s prod \
  --profiler-enabled \
  --profiler-host=0.0.0.0 \
  --profiler-port=8060

Dependencies​

Graphviz (Optional)​

The pprof web interface requires Graphviz to generate visual graphs:

macOS:

brew install graphviz

Ubuntu/Debian:

sudo apt-get install graphviz

CentOS/RHEL:

sudo yum install graphviz

Without Graphviz, you can still use the text-based analysis modes.

Best Practices​

File-Based vs Server-Based​

  • Use file-based profiling for most CLI operations and performance analysis. Supports all profile types: cpu, heap, allocs, goroutine, block, mutex, threadcreate, trace
  • Use server-based profiling for long-running operations or when you need to collect multiple profile types interactively

Profile Regularly​

  • Profile before and after performance optimizations to measure impact
  • Establish baseline profiles for typical operations
  • Profile different stack sizes and complexity levels

Security Considerations​

  • Server-based profiling exposes runtime information through HTTP
  • Use localhost binding in production environments
  • Disable profiling in production unless actively debugging

Troubleshooting​

Common Issues​

Profile file creation errors:

# Ensure the directory exists
mkdir -p /path/to/profile/
atmos command --profile-file=/path/to/profile/cpu.prof --profile-type=cpu

Invalid profile type:

# Check supported profile types
echo "Supported types: cpu, heap, allocs, goroutine, block, mutex, threadcreate, trace"
atmos command --profile-file=profile.out --profile-type=heap

Graphviz not found:

# Use text-based analysis instead
go tool pprof -top cpu.prof
go tool pprof -list=functionName cpu.prof

Server already running:

# Use a different port
atmos command --profiler-enabled --profiler-port=7070

Environment variable usage:

# CI/CD environment setup for profiling
export ATMOS_PROFILE_FILE=/var/log/atmos-cpu.prof
export ATMOS_PROFILE_TYPE=cpu
atmos terraform apply --auto-approve

# Different profile types via environment variables
export ATMOS_PROFILE_TYPE=heap
export ATMOS_PROFILE_FILE=memory-analysis.prof
atmos describe stacks

# Server mode via environment variables
export ATMOS_PROFILER_ENABLED=true
export ATMOS_PROFILER_HOST=0.0.0.0
export ATMOS_PROFILER_PORT=6060
atmos terraform plan vpc -s prod

Understanding Profile Data​

  • flat: Time spent in the function itself
  • cum: Cumulative time spent in the function and its callees
  • flat%: Percentage of total execution time spent in the function
  • sum%: Cumulative percentage up to this function

Focus optimization efforts on functions with high flat time and high flat% percentages.