Skip to content

Self-hosted runner and the engine GitHub App

Why a self-hosted runner

Terraform state lives in a storage account behind a firewall that allows only known addresses. The jobs that read and write the state account (plan, apply, the drift plan, the unlock, and the drift run's lock sweep) run on a self-hosted runner on a private network with access the state firewall allows.

The jobs that only talk to the GitHub API (event parsing, project selection, the merge gate, and drift issue reporting) run on hosted ubuntu-latest runners, because the self-hosted runner is network-scoped and does not carry gh.

Set the runner label with the RUNNER_LABEL repository variable. When it is not set, jobs fall back to ubuntu-latest, which is fine for a demo that does not need private state access.

The runner image

The runner image bakes in the tools the gates need so nothing is installed on the critical path. It is built, scanned, and cosign-signed once and published as ghcr.io/nrit-solutions/alz-runner; the bootstrap imports the digest pinned in the customer tfvars into an in-tenant registry, and the job pulls it from there over a private endpoint. Sovereignty-sensitive customers can instead build the same image from source inside their tenant (the bootstrap's runner_image_commit fallback). Either way:

  • conftest and checkov are baked into the image. checkov is a Python tool with no static binary, so baking it is the production path. On the self-hosted posture, set TFPR_EXTRA_TOOLS to just infracost.
  • infracost is a static binary that the workflow installs per run when TFPR_EXTRA_TOOLS includes infracost. It installs to a user-writable path because the runner runs as a non-root user.

On a GitHub-hosted posture there is no baked image, so TFPR_EXTRA_TOOLS installs all three per run: conftest checkov infracost.

The image does not carry Terraform, Terragrunt, or the engine binary on purpose: the repository's mise.toml and the engine pin decide those versions per run, so a toolchain bump never needs an image release.

Each release ships with a cosign keyless signature on the image digest and a signed SPDX SBOM as a release asset, so a customer can verify what runs in their tenant without trusting the transport.

Runner sizing

Runner replicas run at 2 vCPU and 4Gi with a 7200-second replica timeout. This is deliberate. The AVM module defaults (1 vCPU, 2Gi, 1800s) OOM-hang the heavy ALZ policy-library plans and kill the runner mid-job, which leaves stale state locks. The bootstrap sets the larger size through container_app_container_cpu, container_app_container_memory, and container_app_replica_timeout.

The KEDA scaler checks the GitHub job queue every 10 seconds (container_app_polling_interval_seconds). A queued job waits on average half that interval for the next check, then the container starts and the runner registers, which together take about 40 seconds. The AVM default of 30 seconds added about 15 seconds of wait to every self-hosted job. Polling is not billed by Azure, and at 10 seconds it uses a few hundred GitHub API requests per hour on the runners App, against a 5,000 per hour budget.

The runners GitHub App

Runner registration and the queue-watching KEDA scaler authenticate as a dedicated GitHub App (repository permissions: Administration read and write, Actions read; installed on the customer repository only). Every call runs on a short-lived installation token minted from the App key, so no expiring credential lives in the customer tenant; the previous model, a stored PAT, stopped all runner scale-up silently the day it expired. The bootstrap wires it through github_runners_app_id, github_runners_app_installation_id, and TF_VAR_github_runners_app_private_key; see Bootstrap reference.

The engine GitHub App

The callers reference the public entrypoint repository nrit-solutions/tf-pr-ops, which any organization can call. The engine core behind it, nrit-tf-pr-ops, is private: every job checks it out at the pinned version, and the dispatch action and the callers download the prebuilt tfpr binary from the pinned release instead of building it from source. A GitHub App mints the short-lived token for both. Those are the App's only jobs.

The App is owned by NRIT and installed on the NRIT organization, because that is where the core lives; the token is minted for that organization whichever organization the landing-zone repository is in. Inside nrit-solutions the App is nrit-engine-reader. Any other organization gets its own App from NRIT, one per organization, so access can be revoked per organization by uninstalling it. See Running the platform from your own organization.

Wire it with:

  • ENGINE_APP_CLIENT_ID (repository variable): the engine App client id. The bootstrap sets this from engine_app_client_id.
  • ENGINE_APP_PRIVATE_KEY (repository secret): the engine App private key. The bootstrap sets this from TF_VAR_engine_app_private_key.

The App needs Contents: read only, and its installation on the NRIT organization must include nrit-tf-pr-ops. Because reusable mode depends on the engine checkout, both values are required on every landing-zone repository; without them the first pull request fails at the dispatch step with a message naming the missing variable.

The private key is a real secret

Store ENGINE_APP_PRIVATE_KEY only as a GitHub Actions secret. Never put it in a tfvars file or commit it. Rotating it means updating the secret; the workflow reads it fresh on every run.

The checks App

A second, single-purpose GitHub App owns every engine-authored check run: the per-unit plan and apply rows, the required tf-pr-ops / merge-gate, and the informational tf-pr-ops / approval. Since engine v3 this is part of onboarding, not an option: without the App the rows are created with the workflow token and rendered under an arbitrary Actions heading, and the merge gate loses the engine identity the surface is built around.

The App is per-organization, because its private key is the credential that writes the checks and is never shared across orgs. App names are globally unique on GitHub and cosmetic here: NRIT's is named tf-pr-ops; a customer's can be tf-pr-ops-<org>. The row names are identical everywhere, tf-pr-ops / plan (<label>) #<run> and the fixed gate and approval names; the client id and key are the contract. The engine reads:

  • TFPR_CHECKS_APP_CLIENT_ID (repository variable)
  • TFPR_CHECKS_APP_PRIVATE_KEY (repository secret)

The App needs Checks: read and write only, webhook off, installed on the landing-zone repository. A configured client id whose App is not installed on the repository fails the run loudly rather than silently falling back.

Creating the App is the one manual step: GitHub has no API for it, so use the org's developer settings and install it on the organization with "Only select repositories". The bootstrap does the rest from three tfvars inputs plus the key in the environment:

Input What it does
checks_app_client_id Sets TFPR_CHECKS_APP_CLIENT_ID on the repository
TF_VAR_checks_app_private_key Sets TFPR_CHECKS_APP_PRIVATE_KEY. Same rule as the engine key: omit it on a re-run and the secret is destroyed
checks_app_installation_id Adds the repository to the App's installation, so no per-repo install by hand
checks_app_id Pins the required tf-pr-ops / merge-gate context in the ruleset to the App's numeric id, so a same-named check from a workflow or another App cannot satisfy the merge gate

Look the ids up with gh api orgs/<org>/installations --jq '.installations[] | {id, app_id, app_slug}'; id is the installation id and app_id the numeric App ID.

Cross-repo access

GitHub resolves a uses: reference with the target repository's visibility, never with a caller's token: a private reusable workflow is callable only from its own organization. That is why the engine is split in two. The reusable workflows, the dispatch action, and the caller examples are published to the public nrit-solutions/tf-pr-ops repository by the engine's release job, under the same tag as the private core, so a landing-zone repository in any organization can reference them. No organization setting is needed on either side. The GitHub App token described above is what authenticates the core checkout itself.

Units source public Azure Verified Modules only, so no private module fetch happens on the plan path and nothing else needs a cross-repo grant.

Next: Secrets and variables.