Self-service AWS infrastructure with GitOps, Terraform and Terragrunt

A platform team that creates every queue, bucket and role by hand becomes the queue everyone waits in. The opposite, where everyone has console access, gives you an estate nobody can explain six months later.

I run AWS infrastructure the GitOps way instead. Git is the only way to change anything. Teams build what they need by opening a pull request, the pipeline shows them exactly what will change, and the platform team reviews only the changes that carry real risk. This post covers how that works, the guardrails that make it safe, and why I put Terragrunt on top of Terraform instead of using the plain multi-folder layout.

The principles

Everything below follows from five rules:

1. Git is the only way in. No applies from laptops, no console changes. If it isn’t in Git, it shouldn’t exist.
2. A paved road. Teams compose versioned modules that already have encryption, tagging and naming built in. Doing the right thing is the easy option.
3. Small blast radius. One state file per stack, per environment, per account. A mistake in one stack can’t touch another.
4. Least privilege for pipelines. Each pipeline can only change what its team owns.
5. Plan on every pull request, apply in the merge queue, check for drift every night.

Two repositories

The platform team owns a modules repository: tested, versioned Terraform modules such as sqs-queue, s3-bucket and iam-workload-role, each released with a semantic version tag.

Everyone works in a live repository, where one folder is one stack with its own state. A CODEOWNERS file decides who approves what:

# .github/CODEOWNERS (the last matching line wins)
/teams/orders/       @org/team-orders
/teams/*/prod/       @org/platform-reviewers
/platform/           @org/platform
/policies/           @org/platform @org/security

A team owns its folders and approves its own non-production changes. Production, IAM and networking need a platform reviewer. Nobody files a ticket.

The workflow

← scroll sideways for the full diagram →

1. Pull request. A developer adds or changes a short module call in their team’s folder.
2. Checks. CI finds the stacks that changed and runs only those: terraform fmt and validate, tflint, and Checkov for insecure settings.
3. Plan on the pull request. CI runs the plan with a read-only role and posts it on the pull request, with creates, replaces and destroys called out at the top.
4. Review. CODEOWNERS decides who has to approve.
5. Apply in the merge queue. An approved pull request joins GitHub’s merge queue, and the apply runs there, before anything reaches main. The pipeline gets its deploy role through GitHub’s OIDC integration, so there are no stored AWS keys anywhere.
6. Merge. Only a pull request whose apply succeeded is merged, so main always matches what is running in AWS. If the apply fails, the pull request comes back to its author with the error.
7. Drift. A scheduled plan runs every night. If someone changed something by hand during an incident, the team hears about it and puts it right with a pull request.

The deploy role trusts one repository and one GitHub environment, nothing else:

"Condition": {
  "StringEquals": {
    "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
    "token.actions.githubusercontent.com:sub": "repo:org/infra-live:environment:orders-prod"
  }
}

Guardrails that hold

Pipeline checks are fast feedback, not a security boundary. Anyone who can get around the pipeline gets around the checks. So I layer them:

Layer	Catches
Static checks on every PR	Formatting, invalid code, insecure settings, committed secrets
Module validation and tests	A module that accepts unsafe inputs, such as a public bucket
Policy on the plan	Real computed values: missing encryption, missing tags, changes to resources a team doesn’t own, unexpected destroys
Account structure, SCPs, permission boundaries	Anything that slipped past the pipeline, from any source
Drift detection, AWS Config, Security Hub	Proof that the rules above actually held

The bottom three layers are where the real boundary is. A team pipeline that tries to change a VPC should get a clear message from the plan policy, and if the policy misses it, an AccessDenied from IAM.

Governance built into every module

Self-service only works if every change meets the same governance baseline, whoever wrote it. Ours covers things like:

• encryption with customer-managed KMS keys;
• mandatory tags: owner, cost-centre, data-classification and environment;
• nothing public unless it goes through an approved front door;
• deletion protection on production data stores;
• no wildcard actions in IAM policies.

Those rules are checked in three places, each catching what the one before can’t.

1. Inside the module

Each module enforces the rules that apply to it with variable validation blocks and lifecycle precondition checks. A developer who passes a bad input gets a clear error at plan time, in their own pull request, before any policy engine runs:

# terraform-modules/s3-bucket/variables.tf
variable "tags" {
  type = map(string)
  validation {
    condition = alltrue([
      for k in ["owner", "cost-centre", "data-classification", "environment"] :
      contains(keys(var.tags), k)
    ])
    error_message = "Tags must include owner, cost-centre, data-classification and environment."
  }
}

# terraform-modules/s3-bucket/main.tf
resource "aws_s3_bucket" "this" {
  bucket = var.name
  tags   = var.tags

  lifecycle {
    precondition {
      condition     = var.environment != "prod" || var.kms_key_arn != null
      error_message = "Production buckets must use a customer-managed KMS key."
    }
  }
}

The secure choice is also the default: encryption on, public access blocked, versioning on. Most teams never touch those inputs.

2. Module tests

Every module has terraform test files that prove the guardrails work, using a mocked AWS provider so they run in seconds without credentials. A module version can’t be released unless they pass:

# terraform-modules/s3-bucket/tests/governance.tftest.hcl
mock_provider "aws" {}

run "rejects_missing_tags" {
  command = plan
  variables {
    name        = "orders-exports"
    environment = "dev"
    tags        = { owner = "team-orders" }
  }
  expect_failures = [var.tags]
}

3. OPA policies on the plan

Module checks only cover people who use the module. Policy on the plan covers everything: raw resources, modules used in unexpected ways, and values that are only known once Terraform has computed them. The pipeline turns the plan into JSON with terraform show -json and evaluates it against OPA policies with conftest, on the pull request and again in the merge queue. The policies live in their own folder, owned by the platform and security teams:

package terraform.governance

import rego.v1

required_tags := {"owner", "cost-centre", "data-classification", "environment"}

deny contains msg if {
  some rc in input.resource_changes
  rc.mode == "managed"
  some action in rc.change.actions
  action in {"create", "update"}
  tags := rc.change.after.tags_all
  is_object(tags)
  missing := required_tags - {k | tags[k]}
  count(missing) > 0
  msg := sprintf("%s is missing required tags: %v", [rc.address, missing])
}

A new rule starts as a warning so teams can fix existing stacks, then becomes a deny that blocks the pull request.

Rule	Module validation	OPA on the plan	AWS backstop
Mandatory tags	validation on tags	Required tags on every taggable resource	SCPs that deny creation without the tag, where the service supports aws:RequestTag
KMS encryption	Secure defaults, precondition for production	Encryption set on every data store	SCPs on unencrypted resources, where a condition key exists
Nothing public	Public access blocked by default	No public buckets, security groups or endpoints	S3 Block Public Access at account level
No IAM wildcards	Module builds least-privilege policies	Parse policy documents and deny * actions	Permission boundaries

The module gives the fastest and friendliest feedback, OPA guarantees the rule for everything in the plan, and the AWS controls hold even for changes that never went through the pipeline.

Many developers at the same time

Dozens of people can work in the live repository at once, across teams and often in the same folders. Four things make that work:

• Each pull request plans only what it changed. CI works out which stacks a pull request touches (terragrunt run --all plan --filter-affected does this from the Git diff) and plans only those. Two teams working in different folders never wait for each other.
• Plans don’t take the state lock. Plans on pull requests run with a read-only role, so they run with -lock=false. Nothing is written, so any number of pull requests can plan the same stack at the same time. Only the apply in the merge queue takes the lock.
• New commits replace old plans. A concurrency group per pull request cancels a plan that’s still running when the developer pushes again. Apply jobs are never cancelled.
• Git conflicts are handled in the pull request, as in any repository: update the branch from main and push.

# .github/workflows/plan.yml
concurrency:
  group: plan-${{ github.event.pull_request.number }}
  cancel-in-progress: true

The harder problem isn’t Git conflicts. It’s stale plans. A plan is a snapshot of the code, the state and the real infrastructure at one moment. Two pull requests can touch different files in the same folder, merge cleanly in Git, and still collide in the single state that folder shares.

That’s why I apply in the merge queue instead of on main.

Most pipelines apply after the merge. When that apply fails, main already claims a change that AWS doesn’t have. Every pull request after it now plans against a broken baseline, and the fix has to come as a second pull request, sometimes a third.

GitHub’s merge queue changes the order. When a pull request enters the queue, GitHub builds a temporary branch containing main plus that pull request, and runs the required checks on it through the merge_group event. My apply job is one of those required checks:

• It plans the exact commit that will land. The plan on the pull request was for review. The queue plans again against the state as it is right now, runs the policy checks on that plan, and applies it in the same job.
• A failed apply goes back to the same pull request. A failing required check removes the pull request from the queue and leaves main untouched. The author fixes it in the same pull request and queues it again. If part of the apply had already run, the state records what was created, so the next plan picks up from there.
• One pull request at a time. I set the queue’s build concurrency to 1 and merge one pull request per group. When an entry fails, GitHub rebuilds the entries behind it without the failed change. With several entries in flight, one behind could already have applied changes from a pull request that never merges.
• Keep stacks small. If one team keeps colliding in the same folder, split it by lifecycle, for example network versus application, or stateful versus stateless.

Plan on the PR (parallel)Review, then waiting in the queuePlan + apply in the queueApply failed, back to the PR

← scroll sideways for the full diagram →

The cost is throughput: one apply at a time across the repository. Because each apply covers only the stacks its pull request changed, the queue keeps moving. If it ever became the bottleneck, the next step would be to split the live repository by area, giving each its own queue.

Why Terragrunt instead of plain folders

The usual plain-Terraform layout is a folder per environment per stack, each calling versioned modules. I like that layout, and the isolation comes from it, not from any tool. The problem is what every folder has to repeat.

Plain Terraform · folder per stack

• staging/
  • network/vpc/backend.tfprovider.tfversions.tfmain.tf
  • security/kms/backend.tfprovider.tfversions.tfmain.tf
  • orders/sqs/backend.tfprovider.tfversions.tfmain.tf
• prod/
  • network/vpc/backend.tfprovider.tfversions.tfmain.tf
  • security/kms/backend.tfprovider.tfversions.tfmain.tf
  • orders/sqs/backend.tfprovider.tfversions.tfmain.tf

18 near-identical files to keep in step

Same folders · Terragrunt

• root.hcl backend · provider · versions, once
• staging/
  • account.hcl account ID · deploy role
  • network/vpc/terragrunt.hcl
  • security/kms/terragrunt.hcl
  • orders/sqs/terragrunt.hcl
• prod/
  • account.hcl account ID · deploy role
  • network/vpc/terragrunt.hcl
  • security/kms/terragrunt.hcl
  • orders/sqs/terragrunt.hcl

Each stack is one file: a module version and its inputs

Every plain stack carries its own backend block, provider block and version pins. With a handful of stacks that’s fine. With dozens of stacks across many accounts and regions, it goes wrong in predictable ways:

• Copy-pasted backends. Someone copies a folder and forgets to change the state key. Two stacks now share one state file, and the next apply tries to delete the other stack’s resources.
• The wrong account. The provider block names the role to assume. Copy it from a production folder into a staging one and you’ve pointed staging code at production.
• Values repeated everywhere. Account IDs, regions, tags and environment names get typed again in every folder, and slowly drift apart.
• Wiring between stacks. The EKS stack needs the VPC ID. Plain Terraform reaches into the other stack’s state with terraform_remote_state, and nothing records the order the stacks must run in.

Terragrunt fixes those four without changing the layout. The backend is defined once, and the state key comes from the folder path, so two folders can never share a state:

# live/root.hcl
locals {
  account = read_terragrunt_config(find_in_parent_folders("account.hcl"))
  region  = read_terragrunt_config(find_in_parent_folders("region.hcl"))
}

remote_state {
  backend = "s3"
  config = {
    bucket       = "org-tfstate"
    key          = "${path_relative_to_include()}/terraform.tfstate"
    region       = "ap-southeast-2"
    encrypt      = true
    use_lockfile = true
  }
}

generate "provider" {
  path      = "provider.tf"
  if_exists = "overwrite_terragrunt"
  contents  = <<EOF
provider "aws" {
  region = "${local.region.locals.aws_region}"
  assume_role {
    role_arn = "arn:aws:iam::${local.account.locals.account_id}:role/terraform-deploy"
  }
}
EOF
}

The provider is generated from the account.hcl above each stack, so code in the staging folder can only ever reach the staging account. A stack is then just a module version and its inputs:

# live/prod/ap-southeast-2/orders/sqs/terragrunt.hcl
include "root" {
  path = find_in_parent_folders("root.hcl")
}

terraform {
  source = "git::https://github.com/org/terraform-modules.git//sqs-queue?ref=sqs-queue/v1.4.0"
}

dependency "kms" {
  config_path  = "../../security/kms"
  mock_outputs = { key_arn = "arn:aws:kms:ap-southeast-2:111111111111:key/mock" }
  mock_outputs_allowed_terraform_commands = ["validate", "plan"]
}

inputs = {
  name        = "orders-events"
  kms_key_arn = dependency.kms.outputs.key_arn
}

The dependency block replaces terraform_remote_state. It passes outputs between stacks and records the order they have to run in. The mock outputs let a plan run before the upstream stack exists, and they’re limited to validate and plan, so an apply can never run against a fake value.

With the graph known, terragrunt run --all plan plans a whole folder in dependency order, and --filter-affected narrows it to the stacks that changed since the default branch. That’s what makes a fleet-wide change practical, such as bumping a module version across every account.

Promotion stays simple too. Staging and production pin their own module versions, so a new version goes to staging in one pull request and to production in the next.

When I wouldn’t use Terragrunt

Terragrunt is a tool for a particular kind of scale, not a default:

• A small estate. One team, a handful of components and environments of the same shape are simpler as one root module with a .tfvars file and backend config per environment.
• A team new to Terraform. It’s one more layer to learn and debug.
• Big run --all applies in production. That’s a large blast radius. I keep production applies per stack in CI and use run --all for plans.
• Mock outputs can hide a missing dependency until apply time. Keep them limited to validate and plan.
• A managed platform that already does this. If you’re on HCP Terraform or Spacelift, their stack and workspace features may cover the same ground.

Also, if you’re starting fresh, name the shared root file root.hcl. Terragrunt now treats a root terragrunt.hcl as an anti-pattern and warns about it.

What it adds up to

• Teams don’t wait for the platform team. They open a pull request and the plan appears on it.
• The platform team reviews risk, not routine. Routine changes are approved by the team that owns them; production, IAM and networking come to us.
• main matches AWS. Applies happen before the merge, so a failed apply never leaves main ahead of reality or needs a follow-up pull request.
• Every change has a record: who asked, what the plan was, who approved it and when it was applied.
• Adding an account is a new folder. A new account.hcl and the stacks under it inherit everything else.

The tooling matters less than the shape. Git as the only way in, small states, owners decided by folder, plans on every pull request and guardrails that hold even when someone goes around the pipeline. Terragrunt is how I keep that shape tidy once the estate grows past a few dozen stacks.