Skip to content

Troubleshooting

A job fails at startup with an OIDC or environment error

The OIDC subject binds the repository and the environment name. Only the Azure jobs (the plan matrix, apply, unlock, drift-plan, and drift's lock-sweep) bind an environment and request an id-token; the trusted jobs (resolve, gate, drift's report, and the PR caller's dispatch) request no token. The reusable workflow already keeps this split, so the caller does not need to. If a login fails, confirm the federated credential subject matches the repository and environment. Repos created after GitHub's immutable-id rollout use the repo:org@id/repo@id:environment:env form; the bootstrap builds subjects in that format by default and only uses the legacy name-only form when oidc_subject_uses_repo_ids is set false.

The engine checkout fails or resolve cannot read the engine

The reusable workflow checks the private engine out at engine_ref using a token from the GitHub App. If the App's installation on the NRIT organization does not include nrit-tf-pr-ops, that checkout fails. NRIT grants the App read access to nrit-tf-pr-ops (ask for it if the App is yours), and confirm ENGINE_APP_CLIENT_ID (variable) and ENGINE_APP_PRIVATE_KEY (secret) are set.

The run uses the wrong engine version

Each caller pins the engine in two places: the uses: ref and the engine_ref input. They must match. If they skew, new workflow YAML runs against old scripts (or the reverse). The resolve job logs the requested engine_ref and the engine commit it checked out, at the top of the run. Bump both pins together.

A manual dispatch is refused without a PR number

Dispatching the ops caller (tf-pr-ops.yml) by hand without the pr_number input is refused at parse, with a message saying a full plan of the default branch belongs to drift.yml instead. Every plan, apply, and unlock drives a pull request; set pr_number to the PR the dispatch should act on.

A pull request from a fork fails at resolve

The engine rejects fork pull requests explicitly (engine v3.0.5 and later): resolve fails with "fork pull requests are not supported". Earlier versions diffed a fork against a same-named branch in the base repository, which could silently produce the wrong changed-file set. Push the branch to the repository itself instead; these are private single-team repositories, so a fork adds no value the engine can use.

A plan reports "Unit locked by another PR"

  • Another open pull request planned the same unit first and owns it until it merges or closes. The report comment names the owning PR.
  • Normal path: wait for the owning PR, or get it merged or closed. Its locks release automatically and the next plan on your PR acquires the unit.
  • If the owning PR is parked, comment /unlock on that PR (not yours) to force-release its locks.
  • A lock naming a PR that is already closed or merged clears itself two ways: the next plan that wants the unit reclaims it (engine v1.7.0 and later), and the daily drift run's lock sweep releases every lock whose holder is closed. If closed PRs keep leaving locks behind, the unlock caller is the thing to check: .github/workflows/tf-pr-ops-unlock.yml must exist and trigger on pull_request: types: [closed], per the canonical caller.
  • See Cross-PR unit locks.

The merge gate never turns green

  • Make sure tf-pr-ops / merge-gate is a required status check and that the name matches exactly.
  • The gate is red while a unit has unapplied changes. Run /apply and let it finish.
  • If a plan or apply errored, the gate is red until the error is fixed and the unit re-planned or re-applied.
  • If the PR changed Terraform but no unit was selected, the gate blocks on purpose. Comment /plan to see what is discovered, and confirm the changed unit is a real Terragrunt unit on the branch.
  • Deleting a whole unit directory does not block. The gate reports N unit(s) removed; destroy by hand, see the PR comment and passes. It still blocks if the same PR also adds a unit, because the removal then does not account for every changed Terraform path. See Removed units.

/apply says it is blocked

Read which of the two messages you got.

  • "required reviews not met". The PR needs approving reviews it does not have. Get the approvals the branch protection or ruleset requires, then comment /apply again.
  • "no reviews required by this repository". The repository requires no approving reviews at all, so GitHub reports no review decision and the engine refuses. There is nothing to approve here, so approvals cannot clear it. Either set a required approving review count on the main branch, or, if the repository requires none on purpose, set the TFPR_ALLOW_UNREVIEWED_APPLY repository variable to true. See Secrets and variables.

The same distinction shows in the tf-pr-ops / approval check run's description.

An apply is refused with "Plan changed since review"

The reviewed-plan guard (since engine v3.3.0). The plan taken at apply time does not match the plan that was reviewed, or the current commit has no plan report at all, so the engine refuses to ship something nobody read. The report comment shows the difference at resource-action level. Two causes:

  • Something moved between review and apply: an out-of-band portal change, drift, or a data source resolving differently. Comment /plan, review the new plan, then /apply again.
  • The branch was pushed after the last plan. The automatic plan for the new commit may still be running, or was never read. Wait for it (or comment /plan), review, then /apply. If the push landed during an apply, the new commit's plan rows read Waiting for apply #N on <sha> until that apply finishes; the plan then runs by itself (since engine v6.2.0).

To apply the current plan without a re-review, comment /apply --force. See the reviewed-plan guard.

The gate is red with "Terraform path(s) changed outside the planned units"

The unattributed-path check (since engine v3.4.0). The PR changes a Terraform-pattern path (.tf, .hcl, .tfvars, and friends) that no selected unit accounts for, so merging would ship a change nothing planned. The gate title names the paths when they fit; the gate job's log always lists them. Three causes:

  • A shared file no unit reads changed (a local module tree, a template). The units consuming it were not selected: comment /plan -p <unit> to inspect one, and restructure so the file lives under a unit or is read by one, or accept it with a gate_ignore: entry.
  • A stray tooling file matches the Terraform pattern (.tflint.hcl, a vendored example). Add it to gate_ignore: in projects.yml; the reference repositories ship entries for the files they carry.
  • A path was renamed across units. The deletion side belongs to no current unit. The removed-unit report usually explains it; read that comment first.

A /plan or /apply comment is refused as unauthorized

The engine accepts commands only from a comment whose author_association is OWNER, MEMBER, or COLLABORATOR. Any other author gets a 😕 reaction and a reply naming the association it saw. Add the person as a repository collaborator or an organization member, or have someone who already qualifies run the command. See Who can run a command.

The cost gate shows "skipped"

infracost needs a key. Set the INFRACOST_API_KEY secret and add infracost to the TFPR_EXTRA_TOOLS variable. Use a static key from the Infracost dashboard (the legacy line), not the newer OAuth CLI flow.

checkov reports "not installed"

checkov is baked into the self-hosted runner image. If you run on a hosted runner without the baked image, set TFPR_EXTRA_TOOLS to include checkov on a Python-capable runner, or use the baked runner image.

conftest reports "no policies found"

Check which path the message names before assuming it is expected.

policy, and you have not written any rego yet. Expected. Put .rego files in policy/ (or set CONFTEST_POLICY_DIR). The hook passes with a note until then.

.tfpr-engine/policy, and you do have rego in policy/. A bug in engine versions before v1.6.1. The gate resolved its policy directory relative to its own script location, which in reusable mode is the engine checkout rather than your repository, so it never saw your policies. Every run reported the skip and passed, which means the policy gate was not running at all and the plan-time gates were checkov and infracost only. Fix it by moving both engine pins to v1.6.1 or later.

After that upgrade the gate evaluates for the first time, so treat the first few reports as a baseline rather than as new findings. A warn rule only adds notes to the run comment; a deny that had been passing silently will now fail the hook and block the merge gate.

A runner is OOM-killed and leaves a stale state lock

The heaviest ALZ policy-library plans need more than the AVM runner defaults (1 vCPU, 2Gi, 1800-second timeout). Undersized, the plan OOM-hangs and the replica timeout kills the runner mid-job, which can leave a stale state lock. The bootstrap sizes replicas at 2 vCPU, 4Gi, and a 7200-second timeout through container_app_container_cpu, container_app_container_memory, and container_app_replica_timeout. If you see this, confirm those values are applied. The runbook has the full procedure for clearing a stale lock safely.

The bootstrap plan always shows an ACR bypass revert

The bootstrap plan permanently shows azurerm_container_registry.this wanting to set network_rule_bypass_for_tasks_enabled back to false. Never apply it: the flag is owned by an azapi patch the runner-image build depends on, and reverting it breaks the next image build. The bootstrap README carries the exact restore command if it is ever applied by accident.

Plan cannot reach state

State sits behind a firewall that allows the runner's egress address. Confirm the job ran on the self-hosted runner (RUNNER_LABEL is set) and that the runner's egress address is still allowed on the state account firewall.

A version constraint fails

Pin Terraform and Terragrunt in mise.toml. The built-in fallback versions are only used when no version manager is present, and they may not satisfy a unit's constraint (for example a unit requiring >= 1.12).