Drift detection¶

A scheduled workflow checks whether the live tenant still matches the code, and turns any drift into GitHub Issues. There is no PR to comment on, so the issue is the signal.

What runs¶

The drift workflow runs daily (early morning UTC) and on demand. It has four jobs:

1. resolve (hosted runner): selects every unit for the sweep. An optional drift: block in projects.yml can scope or disable it; by default all units are swept. Its per-job check appears as drift / engine / resolve: the caller workflow's name, then the caller's job id, then the engine's job name.

   drift:
     enabled: true          # false disables the sweep entirely
     include:               # optional, repo-relative globs
       - live/_foundation/**
     exclude:
       - live/sandbox/**
   

   2. lock-sweep (self-hosted runner, plan environment): releases every cross-PR unit lock whose holder pull request has closed or merged, so a leaked lock heals daily without waiting for the unit to be contended. 3. drift-plan (self-hosted runner, plan environment): plans each unit against the deployed tenant, read-only. It uses the same read-only Reader identity as PR plans, so it can never change the tenant. A non-empty plan is drift, which is expected, so it never fails the job. It runs the same gates as a PR plan. 4. report (hosted runner): turns the results into issues.

The issues¶

The report job opens one issue per drifted unit, titled Drift: <label> with a drift label. It:

• opens or refreshes the issue when a unit has changes or errored,
• comments and closes the issue when the unit comes back clean.

So the set of open drift issues is always the current list of units that no longer match their code.

An issue body is the same run report a PR plan posts: the plan output in a code fence, the change summary table, and the gate findings. One thing is left out. A PR report ends with an instructions block offering /apply and /unlock, and an issue does not get it, because those commands are only answered on a pull request. Typing one on a drift issue starts no run and posts no reply.

An issue for a unit that errored (rather than drifted) carries the error output instead of a plan, so read the issue heading: "Drift detected in <label>" means drift, "Drift check errored for <label>" means the plan itself failed and needs troubleshooting.

Acting on a drift issue¶

There is no command that remediates from the issue. Open a pull request that touches the unit and drive it from there, so the change is planned, reviewed and gated like any other.

Read the plan first. Drift is not always the tenant being wrong. When the plan proposes undoing a value Azure derives or defaults, the configuration is incomplete rather than the tenant drifted, and applying it will not converge: Azure re-derives the value and the same drift returns on the next sweep. Fix the configuration in that case, and the next plan comes back clean with nothing applied.

The drift check runs¶

Since engine v6.4.0 the report job also posts one check run per swept unit on the default branch commit it planned, named tf-pr-ops / drift (<label>) #<run number>. It carries the same run summary a PR check carries: the change table and the gate counts, never plan output.

Drift is neutral rather than a failure because no commit caused it; a red default branch every morning would read as a broken build. An earlier drift row for the same unit on the same commit is retitled Outdated: superseded by #N and keeps its summary.

The check runs are posted with the org's checks App, so they need TFPR_CHECKS_APP_CLIENT_ID and its private key, which the bootstrap sets. Without the App the report job logs a notice and posts none; the issues still carry the whole report. The issues stay the place to read and act on drift.

Why it is split across runners¶

Only the plan needs Terraform state, which sits behind a private endpoint, so it must run on the self-hosted runner. Issue management needs only the GitHub API, so it runs on a hosted runner. The drift plan shares a per-unit concurrency group with PR plans, so a drift plan and a PR plan for the same unit never contend for the state lock.

Acting on drift¶

Drift itself never opens a pull request. To remediate, an operator opens a PR that touches the drifted unit, which plans it the normal way. Then either apply to bring the tenant back in line, or update the code to match an intended change. Applying closes the drift issue on the next sweep.

The runbook has a triage guide by cause: portal changes, policy remediation side effects, provider upgrades, and AMBA tag stripping each look different in the plan.

The runner canary¶

Drift detection assumes the self-hosted runners actually pick up work. When runner scale-up breaks (a bad scaler credential, a broken scale rule, an image the job cannot pull), workflow runs do not fail: they sit queued forever and nothing alerts. A separate scheduled workflow, runner-canary.yml, runs three times a day on a GitHub-hosted runner, deliberately outside the machinery it checks, and opens an issue titled "Self-hosted runners are not picking up queued jobs" when any run has waited longer than 30 minutes; one pass runs shortly after the nightly drift dispatch, so a stuck sweep is caught the same morning. The issue body carries the triage order, and the canary closes it again on its first healthy pass. Dispatch it manually with a stale_minutes input to test.