Runbook¶

Day-2 procedures for operating a deployed landing zone. The rest of this section covers the happy path: Plan and apply, Drift detection, and The gates. This page covers everything else, including the cases where a run stops half way.

Symptom-level fixes for a failing job live in Troubleshooting. This page carries the full procedures.

Throughout, "unit" means a Terragrunt unit, a leaf folder with a terragrunt.hcl. The PR-ops engine (nrit-tf-pr-ops) calls the same thing a project, so its labels, checks, and -p selector use that word.

The change process¶

Every change is a pull request. There is no manual dispatch and no local apply.

Branch, edit the unit, open a PR. The engine plans every impacted unit and posts the plan as a comment, with the policy, security, and cost gate output.
Read the plan. The gate panel expands on its own when a gate has a finding.
Get the required approval, then comment /apply. The engine applies the changed units in dependency order and reacts to your comment as it goes.
tf-pr-ops / merge-gate goes green once the applied set matches what the PR changed. Merge.

The merge gate stays red if the PR touched Terraform but no unit was selected, so a new or removed unit cannot merge without being applied. If that happens, the unit path is wrong or missing from projects.yml.

Full detail is in Plan and apply.

Adding a workload¶

Work down the tree, one PR per step, so each plan stays readable.

Create the subscription folder under the target management group, for example live/landingzones/corp/<workload>/, holding subscription.hcl with the subscription id.
Add the lz-vending unit under <workload>/_global/, with its own co-located region.hcl. root.hcl reads a region unconditionally, so the file must exist even though the unit is region-agnostic. The unit places the subscription under the management group and creates nothing else.
Add the network unit under <workload>/<region>/network/. It creates the spoke virtual network and peers it to the hub. The connectivity hub and step 2 must be applied first.
Add the workload units beside it, each on the matching Azure/avm-res-* module at a pinned version.

Growing the tree has the module names, the hub onboarding steps, and the two settings the ALZ policies force: an NSG on every subnet, and no remote gateways when the hub has none.

Clearing a stale state lock¶

Symptom¶

A plan or apply fails with Error acquiring the state lock, and the run waited five minutes first. The engine passes -lock-timeout=5m on every plan and apply, so a lock that is merely busy resolves on its own inside that window. A failure after it means the lock is held by nothing.

The known cause is an OOM-killed runner. The heaviest ALZ policy-library plans need more than the AVM runner defaults, and an undersized replica is killed mid-job, which leaves the lease on the state blob. See A runner is OOM-killed and leaves a stale lock for the runner sizing that prevents it.

Confirm nothing is running first¶

Never force-unlock a live run

Breaking the lock under a running apply corrupts state. Confirm the unit is idle before you touch it.

There is no /unlock command. The engine does not clear locks, by design. Two checks, both of them:

Open the repository's Actions tab and confirm no terraform-pr-ops or drift run is in progress for this unit.
Check the concurrency group. Plans and drift plans for a unit serialize on tfpr-<label>, where <label> is the unit path. Applies serialize per PR on tfpr-apply-<pr-number>. A queued run in either group means work is still pending, so wait for it rather than unlocking.

Find the lock ID¶

The error block in the job log and the PR comment names it:

Error: Error acquiring the state lock

Lock Info:
  ID:        6a1f0c92-1a4e-4b39-8d7c-2f0b6e4a91c3
  Path:      tfstate/landingzones/corp/corp-workload/westeurope/network/terraform.tfstate
  Operation: OperationTypeApply
  Who:       runner@...
  Created:   ...

ID is the lease id you pass to force-unlock. Path is the state key, which is the unit's path under live/.

Force-unlock the unit¶

Run it locally against the unit, from a machine that can reach the state account and is signed in to Azure:

terragrunt --working-dir live/landingzones/corp/corp-workload/westeurope/network \
  run -- force-unlock 6a1f0c92-1a4e-4b39-8d7c-2f0b6e4a91c3

The state account is default-deny and allows the runner's egress address, so this only works from an allowlisted address. If you cannot reach it, add your address to the state account firewall for the duration, or run from the runner's network.

Last resort: break the blob lease¶

If force-unlock cannot run, break the lease on the state blob directly:

az storage blob lease break \
  --auth-mode login \
  --account-name <state-account> \
  --container-name tfstate \
  --blob-name landingzones/corp/corp-workload/westeurope/network/terraform.tfstate

This releases the lease but leaves Terraform's lock record behind, so the next run may still report a lock. Clear it by running force-unlock once the blob is writable again.

After either route, re-plan the unit through the PR and confirm the plan is what you expect before applying.

A half-failed dependency-ordered apply¶

What the engine does¶

/apply is one job that walks the impacted set in dependency order, one unit at a time. When a unit fails, every unit downstream of it is skipped rather than applied against stale dependency outputs. Each skipped unit gets its own report comment saying dependency <label> did not apply, and the job exits non-zero so the run is red and the merge gate stays closed.

So a partial apply is the intended outcome, not a broken state. The units before the failure are applied. The failed unit is not. Everything after it is untouched.

Recover¶

Read the failing unit's report comment. Fix the cause in the same PR.
Push the fix. The engine re-plans the impacted units.
Comment /apply again.

Re-applying the whole set is safe. Units that already applied re-plan as no changes and apply nothing. There is no need to select the remaining units by hand, though /apply -p <label> works if you want to narrow it.

If the failure is in a unit you cannot fix now, revert that unit's change in the PR so the impacted set shrinks, and land the rest.

Moving or renaming a unit¶

The backend key is ${path_relative_to_include()}/terraform.tfstate, so a unit's path under live/ is its state key. Move or rename the folder and the unit points at a new, empty state file. Terraform then plans to create everything the unit already manages, and to destroy nothing, because the old state is simply orphaned. See The live tree.

Move the state before the merge, not after

A plan that shows a full create for a unit you only renamed is the signal that state has not moved yet. Do not apply it.

Procedure:

Open the PR with the move. Let it plan. Confirm the plan shows the new path creating everything and, if the old path is still discovered, destroying it. That is the orphan, made visible.
Copy the state blob to the new key, before you apply or merge:

az storage blob copy start \
  --auth-mode login \
  --account-name <state-account> \
  --destination-container tfstate \
  --destination-blob landingzones/corp/corp-workload/westeurope/network/terraform.tfstate \
  --source-container tfstate \
  --source-blob landingzones/corp/corp-workload/westeurope/net/terraform.tfstate

Copy, do not move. The old blob stays as the rollback until you are done. 3. Comment /plan again. The unit must come back with no changes. That is the verification: same resources, new key. 4. Apply if there are real changes, then merge. 5. Delete the old blob once the new key has applied cleanly at least once. Blob versioning and seven-day soft delete are on for the container, so a mistaken delete is recoverable within that window.

terragrunt run -- state mv moves resources inside one state file. It does not change the backend key, so it is not the tool for a folder move. Reach for it only when you are also splitting or merging units, and do it as a separate, deliberate step.

AMBA remediation¶

Applying the foundation assigns the AMBA DeployIfNotExists policies but does not deploy the action group or the alerts. DeployIfNotExists never touches resources that already exist until a remediation task runs.

Trigger a compliance scan per subscription first:

az policy state trigger-scan --subscription <subscription-id>

Then create a remediation task for each Deploy-AMBA-* assignment:

az policy remediation create \
  --name remediate-amba \
  --management-group <management-group-id> \
  --policy-assignment Deploy-AMBA-<name>

--resource-discovery-mode ReEvaluateCompliance is supported at subscription scope and below, not at management-group scope. That is why the order is scan the subscriptions, then remediate.

The remediation runs as the assignment's managed identity, which must already hold the RBAC role the policy needs. The foundation grants it.

Drift triage¶

The drift sweep opens one issue per drifted unit. See Drift detection for what runs and how issues open and close.

Triage by cause, not by re-applying blindly.

Cause	How it looks	What to do
A policy `deployIfNotExists` remediation changed the resource	Tags, diagnostic settings, AMBA resources appearing that no unit declares	Match the code to what the policy deploys, or stop managing the attribute here. The AMBA resource group is the standing example: it is left AMBA-owned for exactly this reason.
Someone changed it in the portal	A plan that reverts a hand edit	Re-apply through a PR to put it back, and raise the access that allowed the edit.
A provider or module upgrade changed the plan	Drift appears across many units at once, after a version bump	Pin work belongs in its own PR. Do not mix it with a functional change.

Never fix drift outside the PR flow

A drift issue is not closed by applying by hand. Open a PR that touches the drifted unit, let the plan confirm the fix, and /apply it. The next sweep closes the issue.

Rotating credentials¶

The deploy identities are Entra workload-identity OIDC federated credentials. There is no secret to rotate, and they are created and changed by the bootstrap (nrit-alz-bootstrap), not in the customer repository.

Three items are real secrets and do rotate.

Secret	What it is	How to rotate
`CATALOG_APP_PRIVATE_KEY`	The GitHub App private key the engine checkout uses to read the private engine repository. The name is historical: the App authenticates the checkout, not a module catalog.	Generate a new private key on the App, update the repository secret, then delete the old key on the App.
`INFRACOST_API_KEY`	The cost gate's key.	Replace the secret with a new static key from the Infracost dashboard, not the OAuth CLI flow.
The GitHub PAT used by the bootstrap	Only on the `self_hosted_private` posture, where runners register with GitHub. Supplied to the bootstrap as `TF_VAR_github_runners_token`, falling back to `github_token`. Never in a tfvars file.	Issue a new PAT, export it, and re-apply the bootstrap so the runners re-register.

A bad INFRACOST_API_KEY rotation degrades to no cost output rather than a failed plan: the cost gate notes the missing key and skips. See The cost gate shows "skipped".

After any rotation, dispatch the drift workflow to confirm the engine still authenticates, rather than finding out on the next customer PR.

Teardown and decommissioning¶

Order matters, and getting it wrong leaves resources that cannot be deleted.

Policy before management groups¶

Delete policy assignments, then any custom set-definitions, then custom definitions, then the management groups, bottom-up.

Deleting a management group with a live assignment orphans it

The assignment becomes an un-deletable ghost. If the management group is later recreated, the foundation apply hangs indefinitely trying to recreate that assignment on the inconsistent scope.

Full order¶

Workload units, leaf first. Destroy each through the repository, in reverse dependency order: workload resources, then the spoke network, then lz-vending.
Platform units, the connectivity hub last of those.
Foundation units: amba, then landing-zones, then management-resources. This is the reverse of the apply order.
Policy and management groups, as above: assignments, set-definitions, definitions, management groups bottom-up.
Bootstrap resources, last, and from the bootstrap repository, not the customer repository. The bootstrap owns the runners, the identities, the federated credentials, and the state account.

The state account goes last¶

One storage account holds the state for every unit. Destroy it and every remaining unit loses the record of what it manages, which leaves live Azure resources with no way to destroy them from code. So the state account is removed only after every unit has been destroyed and its state is genuinely empty.

The bootstrap's own state lives in a separate, IP-locked account, which is why the bootstrap can still destroy itself after the customer state account is gone.