Skip to content

Bootstrap a customer

The bootstrap is a plain Terraform root module in nrit-alz-bootstrap. An operator runs it once per customer, from a workstation, against the customer tenant. There is no CI for it and no pipeline to trigger. It stands up everything the customer repository needs before any workflow can run: the state backend, the deploy identities and their OIDC federation, the repository itself, and, on the production posture, the self-hosted runners.

This page is the procedure. For the full list of what it creates and every input it takes, see Bootstrap reference.

Before you start

You need three things: an Azure session, GitHub tokens, and rights in both.

Azure

Sign in to the customer tenant and select the management subscription. That subscription holds the state, the identities, and the runners.

az login --tenant <customer-tenant-id>
az account set --subscription <management-subscription-id>

Your account needs:

  • rights to create resource groups and resources in the management subscription,
  • rights to create role assignments at the management group scope you are targeting (the plan identity gets Reader there, the apply identity gets Owner), which in practice means User Access Administrator or Owner at that scope,
  • rights to read your own directory object, which scripts/create-backend.sh uses to grant you Storage Blob Data Contributor on the state account.

GitHub

A personal access token with repo and admin:org scope in the organization that receives the customer repository. The admin:org scope is needed because the bootstrap creates the approver team and the ruleset there. Nothing is needed on the NRIT organization: the template is public and the engine is reached through the engine App.

export GITHUB_TOKEN=<pat>

GITHUB_TOKEN authenticates the github provider. Some paths need the token as a real Terraform value instead, so export TF_VAR_github_token when either applies:

Variable When you need it
GITHUB_TOKEN or TF_VAR_github_token Always. Authenticates the github provider.
TF_VAR_github_runners_app_private_key self_hosted_private with the recommended runners GitHub App (github_runners_app_id set in the tfvars). The scaler and the runner registration mint short-lived installation tokens from it, so nothing stored in the tenant expires.
TF_VAR_github_runners_token self_hosted_private PAT fallback, only when the runners App is not configured. Registers the runners; can be the same PAT; falls back to github_token. PATs expire, and an expired one silently stops all runner scale-up.
TF_VAR_github_token (not just GITHUB_TOKEN) Build-from-source runner image only (runner_image_commit set). The ACR build task clones the context repo with a real Terraform value. TF_VAR_runner_image_context_token overrides it. The default prebuilt-image import needs no token at all.
TF_VAR_engine_app_private_key Whenever the tfvars file sets engine_app_client_id. Becomes the ENGINE_APP_PRIVATE_KEY secret the engine checkout uses.
TF_VAR_checks_app_private_key Whenever the tfvars file sets checks_app_client_id. Becomes the TFPR_CHECKS_APP_PRIVATE_KEY secret the engine signs its check rows with.

The GitHub App private keys

The engine App private key is what authenticates the private engine checkout for the PR-ops engine (nrit-tf-pr-ops). See Secrets and variables for what the engine does with it.

export TF_VAR_engine_app_private_key="$(cat /path/to/engine-app-private-key.pem)"
export TF_VAR_checks_app_private_key="$(cat /path/to/checks-app-private-key.pem)"

The second key belongs to the checks App, which writes the engine's check rows. It follows the same rules as the first.

Omitting the key destroys the secret

engine_app_private_key defaults to null, and the ENGINE_APP_PRIVATE_KEY repository secret is managed only when it is set. Run the bootstrap without it on a customer that has the secret, and Terraform destroys the secret. The engine checkout then fails on the next run. Always export it when engine_app_client_id is set in the tfvars file. checks_app_private_key and the TFPR_CHECKS_APP_PRIVATE_KEY secret behave the same way when checks_app_client_id is set.

Secrets go in the environment, never in tfvars

The tfvars files are committed. The PAT, the runners PAT, and the App private key are only ever supplied through environment variables. There are no long-lived Azure secrets at all: storage account keys are disabled and all pipeline auth is OIDC.

Run it

1. Create the customer var file

cd nrit-alz-bootstrap
cp customers/_example.tfvars customers/<customer>.tfvars

Fill in the tenant id, the management subscription id, the repository name, the approvers, and the network posture. Set network_posture = "self_hosted_private" for production. Set required_status_checks = ["tf-pr-ops / merge-gate"] unless you have a reason not to: the default is empty, which lets a pull request merge with unapplied changes. Every input is listed in Bootstrap reference.

2. Create the state backend

The bootstrap's own state lives in a dedicated storage account that create-backend.sh owns out of band. A config must not manage the account that stores its own state, so this account is not in the Terraform.

scripts/create-backend.sh -f customers/<customer>.tfvars

The script is idempotent. Each run:

  • ensures the resource group rg-cicd-bootstrap-<location_short> and the storage account (name derived deterministically from the subscription id, so no shared state is needed to compute it),
  • disables shared keys, enforces TLS 1.2, and sets the account default-deny,
  • enables blob versioning and seven-day soft delete,
  • resets the IP firewall to your current public IP, removing every other entry,
  • ensures the tfstate container through the ARM control plane,
  • grants you Storage Blob Data Contributor,
  • prints the exact terraform init command to run next.

Re-run the script whenever your public IP changes

The firewall allows exactly one operator IP. Change network, or get a new address from your ISP, and every terraform command fails on the backend. Re-run scripts/create-backend.sh before any terraform command; it resets the allowlist to your current IP.

3. Initialize Terraform

The backend block carries no values, because backend blocks cannot use variables. Use the -backend-config lines the script printed.

terraform init \
  -backend-config="resource_group_name=rg-cicd-bootstrap-<loc>" \
  -backend-config="storage_account_name=<printed by the script>" \
  -backend-config="container_name=tfstate" \
  -backend-config="key=bootstrap.tfstate"

First run only

The script grants you Storage Blob Data Contributor on the account it just created. That role assignment can take about a minute to propagate. If terraform init fails with an authorization error on the very first run, wait and retry.

4. Plan and apply

terraform plan  -var-file=customers/<customer>.tfvars
terraform apply -var-file=customers/<customer>.tfvars

Read the plan. On the production posture it is large, because the AVM runner module creates a network, a registry, and the runner compute. Check the known permadiff below before you apply.

Re-running the bootstrap

Re-running is normal. You re-run to add a capability to an existing customer, to change a decision variable, or to recover from a partial failure.

  1. Export the same environment variables as the first run, including TF_VAR_engine_app_private_key and TF_VAR_checks_app_private_key. Without them the ENGINE_APP_PRIVATE_KEY and TFPR_CHECKS_APP_PRIVATE_KEY secrets are destroyed.
  2. Re-run scripts/create-backend.sh -f customers/<customer>.tfvars, in case your IP changed.
  3. terraform plan and terraform apply with the same var file.

State is read from the remote backend, so a re-run does not depend on a local terraform.tfstate file or on the machine that did the first run.

The remote state must be preserved

A clean apply against empty state tries to create a GitHub repository that already exists, and fails. If bootstrap-managed resources were deleted out of band, terraform apply treats that as drift and recreates them, while the existing repository and its Action variables are updated in place.

The runner image

By default the bootstrap imports the prebuilt, signed runner image (ghcr.io/nrit-solutions/alz-runner: the AVM runner base plus the checkov and conftest the engine's gates need) into a small bootstrap-owned container registry, pinned by digest, and the Container App Job pulls it from there over a private endpoint. The import is the server-side importImage action: anonymous, no docker, no token, and the manifest digest survives the copy bit-identical, so the job runs exactly the signed artifact.

The pin is two values in the customer tfvars, both taken from the same image release, which is also the record of which image version the customer runs:

# in customers/<customer>.tfvars
runner_image_version = "X.Y.Z"
runner_image_digest  = "sha256:..."

Bump a release by changing both values and running:

terraform apply -var-file=customers/<customer>.tfvars \
  -target='module.runners' -target='azapi_resource_action.import_runner_image'

The job pulls by digest, so the digest change is what redeploys it.

Build-from-source fallback

For customers who must not pull NRIT-published artifacts, set runner_image_commit to a commit SHA of nrit-alz-bootstrap holding runner-image/github-runner-aca. A registry owned by the runner module then builds the image inside the tenant with an ACR build task instead, and the prebuilt-image pins are ignored.

export TF_VAR_github_token=<pat>   # the ACR build task needs a real Terraform value
terraform apply -target='module.runners' -var-file=customers/<customer>.tfvars

Use a commit SHA, not a branch. The image tag is content-addressed by the SHA (github-runner:nrit-<first 7 chars>), so a Dockerfile change produces a new tag and the Container App Job redeploys. A branch ref produces a stable tag and nothing rolls out. This path stores the clone PAT in the tenant's build task; that trade is the reason the prebuilt import is the default.

Known permadiff: the ACR bypass revert

Never apply the ACR bypass revert

Every plan on the self_hosted_private posture shows azurerm_container_registry.this wanting to set network_rule_bypass_for_tasks_enabled = true -> false. Do not apply it.

The flag is owned by an azapi_update_resource patch inside the AVM runners module. The wrapped azurerm resource does not know about the patch and tries to revert it. Applying the revert breaks the next runner-image build, because the registry is private and the build task needs the bypass. The azapi patch does not self-heal on drift.

If it is applied by accident, restore it:

az rest --method patch \
  --url "https://management.azure.com/<acr-resource-id>?api-version=2025-05-01-preview" \
  --body '{"properties":{"networkRuleBypassAllowedForTasks":true}}'

Use -target for runner changes, or read the plan carefully, so the revert never slips in with an unrelated apply.

What the bootstrap does not do

Five things are manual after the apply. None of them are managed by the bootstrap, so a re-run will not create or restore them.

Step Where Why it is manual
Create the checks App, once per organization, and install it on the organization with "Only select repositories" GitHub org settings GitHub has no API to create an App. Everything after that is the bootstrap's: with checks_app_client_id, checks_app_installation_id, and checks_app_id set it adds the repository to the installation, sets the variable and secret, and pins the merge gate to the App. See the checks App.
Set the INFRACOST_API_KEY secret on the customer repository Repository secrets The key is an account-level credential, not a per-customer value. Until it is set, the cost gate notes it and skips.
Set TFPR_ALLOW_UNREVIEWED_APPLY to true Repository variables Only for a single-writer organization that set required_approving_review_count = 0. GitHub then returns an empty review decision and the engine refuses to apply on it.
Point CODEOWNERS at a team in the organization Customer repository The bootstrap does no templating and writes no files. Everything in the generated repository is a verbatim copy of the template, whose CODEOWNERS is a commented example and whose LICENSE is Apache-2.0.

See Secrets and variables for the full list of what the repository reads and who sets each value.

After the bootstrap

The customer repository now has its backend, identities, environments, and merge gate wired. Next, pin the engine version in the four caller workflows and fill in the handful of placeholder values in the tree. The Quickstart walks that end to end.

If something fails, Troubleshooting covers the common bootstrap errors.

Next: Bootstrap reference.