Bootstrap a customer¶
The bootstrap is a plain Terraform root module in nrit-alz-bootstrap. An operator
runs it once per customer, from a workstation, against the customer tenant. There is
no CI for it and no pipeline to trigger. It stands up everything the customer
repository needs before any workflow can run: the state backend, the deploy
identities and their OIDC federation, the repository itself, and, on the production
posture, the self-hosted runners.
This page is the procedure. For the full list of what it creates and every input it takes, see Bootstrap reference.
Before you start¶
You need three things: an Azure session, GitHub tokens, and rights in both.
Azure¶
Sign in to the customer tenant and select the management subscription. That subscription holds the state, the identities, and the runners.
Your account needs:
- rights to create resource groups and resources in the management subscription,
- rights to create role assignments at the management group scope you are targeting (the plan identity gets Reader there, the apply identity gets Owner), which in practice means User Access Administrator or Owner at that scope,
- rights to read your own directory object, which
scripts/create-backend.shuses to grant you Storage Blob Data Contributor on the state account.
GitHub¶
A personal access token with repo and admin:org scope in the organization
that receives the customer repository. The admin:org scope is needed because
the bootstrap creates the approver team and the ruleset there. Nothing is needed
on the NRIT organization: the template is public and the engine is reached
through the engine App.
GITHUB_TOKEN authenticates the github provider. Some paths need the token as a
real Terraform value instead, so export TF_VAR_github_token when either applies:
| Variable | When you need it |
|---|---|
GITHUB_TOKEN or TF_VAR_github_token |
Always. Authenticates the github provider. |
TF_VAR_github_runners_app_private_key |
self_hosted_private with the recommended runners GitHub App (github_runners_app_id set in the tfvars). The scaler and the runner registration mint short-lived installation tokens from it, so nothing stored in the tenant expires. |
TF_VAR_github_runners_token |
self_hosted_private PAT fallback, only when the runners App is not configured. Registers the runners; can be the same PAT; falls back to github_token. PATs expire, and an expired one silently stops all runner scale-up. |
TF_VAR_github_token (not just GITHUB_TOKEN) |
Build-from-source runner image only (runner_image_commit set). The ACR build task clones the context repo with a real Terraform value. TF_VAR_runner_image_context_token overrides it. The default prebuilt-image import needs no token at all. |
TF_VAR_engine_app_private_key |
Whenever the tfvars file sets engine_app_client_id. Becomes the ENGINE_APP_PRIVATE_KEY secret the engine checkout uses. |
TF_VAR_checks_app_private_key |
Whenever the tfvars file sets checks_app_client_id. Becomes the TFPR_CHECKS_APP_PRIVATE_KEY secret the engine signs its check rows with. |
The GitHub App private keys¶
The engine App private key is what authenticates the private engine checkout for the
PR-ops engine (nrit-tf-pr-ops). See
Secrets and variables for what the engine does with it.
export TF_VAR_engine_app_private_key="$(cat /path/to/engine-app-private-key.pem)"
export TF_VAR_checks_app_private_key="$(cat /path/to/checks-app-private-key.pem)"
The second key belongs to the checks App, which writes the engine's check rows. It follows the same rules as the first.
Omitting the key destroys the secret
engine_app_private_key defaults to null, and the ENGINE_APP_PRIVATE_KEY
repository secret is managed only when it is set. Run the bootstrap without it on
a customer that has the secret, and Terraform destroys the secret. The engine
checkout then fails on the next run. Always export it when
engine_app_client_id is set in the tfvars file. checks_app_private_key
and the TFPR_CHECKS_APP_PRIVATE_KEY secret behave the same way when
checks_app_client_id is set.
Secrets go in the environment, never in tfvars
The tfvars files are committed. The PAT, the runners PAT, and the App private key are only ever supplied through environment variables. There are no long-lived Azure secrets at all: storage account keys are disabled and all pipeline auth is OIDC.
Run it¶
1. Create the customer var file¶
Fill in the tenant id, the management subscription id, the repository name, the
approvers, and the network posture. Set network_posture = "self_hosted_private" for
production. Set required_status_checks = ["tf-pr-ops / merge-gate"] unless you have
a reason not to: the default is empty, which lets a pull request merge with unapplied
changes. Every input is listed in
Bootstrap reference.
2. Create the state backend¶
The bootstrap's own state lives in a dedicated storage account that create-backend.sh
owns out of band. A config must not manage the account that stores its own state, so
this account is not in the Terraform.
The script is idempotent. Each run:
- ensures the resource group
rg-cicd-bootstrap-<location_short>and the storage account (name derived deterministically from the subscription id, so no shared state is needed to compute it), - disables shared keys, enforces TLS 1.2, and sets the account default-deny,
- enables blob versioning and seven-day soft delete,
- resets the IP firewall to your current public IP, removing every other entry,
- ensures the
tfstatecontainer through the ARM control plane, - grants you Storage Blob Data Contributor,
- prints the exact
terraform initcommand to run next.
Re-run the script whenever your public IP changes
The firewall allows exactly one operator IP. Change network, or get a new address
from your ISP, and every terraform command fails on the backend. Re-run
scripts/create-backend.sh before any terraform command; it resets the allowlist
to your current IP.
3. Initialize Terraform¶
The backend block carries no values, because backend blocks cannot use variables. Use
the -backend-config lines the script printed.
terraform init \
-backend-config="resource_group_name=rg-cicd-bootstrap-<loc>" \
-backend-config="storage_account_name=<printed by the script>" \
-backend-config="container_name=tfstate" \
-backend-config="key=bootstrap.tfstate"
First run only
The script grants you Storage Blob Data Contributor on the account it just created.
That role assignment can take about a minute to propagate. If terraform init
fails with an authorization error on the very first run, wait and retry.
4. Plan and apply¶
terraform plan -var-file=customers/<customer>.tfvars
terraform apply -var-file=customers/<customer>.tfvars
Read the plan. On the production posture it is large, because the AVM runner module creates a network, a registry, and the runner compute. Check the known permadiff below before you apply.
Re-running the bootstrap¶
Re-running is normal. You re-run to add a capability to an existing customer, to change a decision variable, or to recover from a partial failure.
- Export the same environment variables as the first run, including
TF_VAR_engine_app_private_keyandTF_VAR_checks_app_private_key. Without them theENGINE_APP_PRIVATE_KEYandTFPR_CHECKS_APP_PRIVATE_KEYsecrets are destroyed. - Re-run
scripts/create-backend.sh -f customers/<customer>.tfvars, in case your IP changed. terraform planandterraform applywith the same var file.
State is read from the remote backend, so a re-run does not depend on a local
terraform.tfstate file or on the machine that did the first run.
The remote state must be preserved
A clean apply against empty state tries to create a GitHub repository that already
exists, and fails. If bootstrap-managed resources were deleted out of band,
terraform apply treats that as drift and recreates them, while the existing
repository and its Action variables are updated in place.
The runner image¶
By default the bootstrap imports the prebuilt, signed runner image
(ghcr.io/nrit-solutions/alz-runner: the AVM runner base plus the checkov
and conftest the engine's gates need) into a small bootstrap-owned
container registry, pinned by digest, and the Container App Job pulls it
from there over a private endpoint. The import is the server-side
importImage action: anonymous, no docker, no token, and the manifest
digest survives the copy bit-identical, so the job runs exactly the signed
artifact.
The pin is two values in the customer tfvars, both taken from the same image release, which is also the record of which image version the customer runs:
Bump a release by changing both values and running:
terraform apply -var-file=customers/<customer>.tfvars \
-target='module.runners' -target='azapi_resource_action.import_runner_image'
The job pulls by digest, so the digest change is what redeploys it.
Build-from-source fallback¶
For customers who must not pull NRIT-published artifacts, set
runner_image_commit to a commit SHA of nrit-alz-bootstrap holding
runner-image/github-runner-aca. A registry owned by the runner module then
builds the image inside the tenant with an ACR build task instead, and the
prebuilt-image pins are ignored.
export TF_VAR_github_token=<pat> # the ACR build task needs a real Terraform value
terraform apply -target='module.runners' -var-file=customers/<customer>.tfvars
Use a commit SHA, not a branch. The image tag is content-addressed by the SHA
(github-runner:nrit-<first 7 chars>), so a Dockerfile change produces a new tag and
the Container App Job redeploys. A branch ref produces a stable tag and nothing rolls
out. This path stores the clone PAT in the tenant's build task; that trade
is the reason the prebuilt import is the default.
Known permadiff: the ACR bypass revert¶
Never apply the ACR bypass revert
Every plan on the self_hosted_private posture shows
azurerm_container_registry.this wanting to set
network_rule_bypass_for_tasks_enabled = true -> false. Do not apply it.
The flag is owned by an azapi_update_resource patch inside the AVM runners
module. The wrapped azurerm resource does not know about the patch and tries to
revert it. Applying the revert breaks the next runner-image build, because the
registry is private and the build task needs the bypass. The azapi patch does not
self-heal on drift.
If it is applied by accident, restore it:
Use -target for runner changes, or read the plan carefully, so the revert never
slips in with an unrelated apply.
What the bootstrap does not do¶
Five things are manual after the apply. None of them are managed by the bootstrap, so a re-run will not create or restore them.
| Step | Where | Why it is manual |
|---|---|---|
| Create the checks App, once per organization, and install it on the organization with "Only select repositories" | GitHub org settings | GitHub has no API to create an App. Everything after that is the bootstrap's: with checks_app_client_id, checks_app_installation_id, and checks_app_id set it adds the repository to the installation, sets the variable and secret, and pins the merge gate to the App. See the checks App. |
Set the INFRACOST_API_KEY secret on the customer repository |
Repository secrets | The key is an account-level credential, not a per-customer value. Until it is set, the cost gate notes it and skips. |
Set TFPR_ALLOW_UNREVIEWED_APPLY to true |
Repository variables | Only for a single-writer organization that set required_approving_review_count = 0. GitHub then returns an empty review decision and the engine refuses to apply on it. |
Point CODEOWNERS at a team in the organization |
Customer repository | The bootstrap does no templating and writes no files. Everything in the generated repository is a verbatim copy of the template, whose CODEOWNERS is a commented example and whose LICENSE is Apache-2.0. |
See Secrets and variables for the full list of what the repository reads and who sets each value.
After the bootstrap¶
The customer repository now has its backend, identities, environments, and merge gate wired. Next, pin the engine version in the four caller workflows and fill in the handful of placeholder values in the tree. The Quickstart walks that end to end.
If something fails, Troubleshooting covers the common bootstrap errors.
Next: Bootstrap reference.