Azure Production Deployment Plan

Updated: 2026-07-02

This plan starts from the accepted Azure staging baseline. No customer data migration is required because PeakPOS has not launched yet. Firebase stays in place. GCP Private CA stays in place for terminal certificate issuance unless a later security review replaces it.

Production Principles

• Keep production Terraform state separate from staging.
• Keep GCP production online as rollback until Azure production cutover is proven.
• Do not reuse APIM classic Developer in production.
• Do not move Firebase as part of this cutover.
• Do not add Google service-account private keys; use Azure IMDS plus Google WIF.
• Keep terminal-api behind an mTLS-only edge.
• Prefer private service-to-service paths inside Azure for cost and security.

Accepted Staging Baseline

Azure staging is accepted as the production-entry proof baseline as of 2026-07-01. The latest staging reconciliation was rechecked on 2026-07-02 at 15:45Z after the APIM custom-hostname drift repair and shared-services endpoint verification.

• GitHub Actions run: Azure Staging Smoke run 28602040658 on commit 232173c65dc41210cf6b18de35ba279ab1797c16.
• Deploy automation run: Build Container Images and Up run 28497605545 on commit de555cb4efbc1772a73aff4c7af37bef96598e57.
• Terrakube Azure staging apply: job 340, completed.
• Terrakube Azure staging reconciliation applies: job 526 completed after importing existing recovered Key Vault PostgreSQL secrets and the GitHub environment federated credential into state; job 531 completed after the production PostgreSQL HA guard was merged; job 550 completed after re-enabling the staging APIM custom hostname and importing the existing deploy Key Vault certificate role assignment; job 563 completed from main after the latest shared-services endpoint and edge reconciliation; job 585 completed as a no-op after the production-local shared notification private endpoint change was merged; job 596 completed as a no-op after the APIM edge marker policy PR was merged.
• Infrastructure smoke: scripts/azure-staging-smoke.sh --strict passed with 205 ok, 0 warnings, 0 failures; the 2026-07-02 post-merge staging smoke against https://staging-api.peakpos.co passed with 206 ok, 0 warnings, 0 failures and verified X-PeakPOS-Azure-Edge: apim-developer-1.
• Credentialed smoke: scripts/azure-staging-credentialed-smoke.sh --strict passed with 16 ok, 0 warnings, 0 skipped, 0 failures.
• Public health: https://staging-api.peakpos.co/status/health reports healthy service checks for the Azure-backed services.
• APIM staging hostname state: staging-api.peakpos.co is bound alongside the default peakpos-staging-apim-jx8zyn.azure-api.net gateway hostname.
• All monorepo Azure staging Container Apps are running the dev image tag and report healthy through https://staging-api.peakpos.co/status/health.
• Legacy GCP staging Cloud Run deploy/smoke is gated by GCP_STAGING_CLOUD_RUN_ENABLED=false and skipped on the post-merge deploy workflow.
• Azure PostgreSQL schema/init, grants, service roles, and DB-backed workflows are proven in staging.
• Azure runtime smoke verifies no SPANNER_* or PGADAPTER_* runtime variables on Azure Container Apps.
• Terminal onboarding issued a GCP Private CA certificate from Azure, and terminal-api accepted it through Azure Application Gateway mTLS using the forwarded SHA-1 fingerprint.
• Product image storage, merchant Prolog cache persistence, notification queue transport, schedulers, and alert resources are Azure-backed in staging.
• Safe-recipient receipt smoke reached the shared-services email worker path and observed recent provider success signal.
• Live resource-group inspection on 2026-07-02 found monorepo staging resources tagged and owned in peakpos-staging-rg, with shared notification workers and shared platform resources in peakpos-shared-services-rg.
• Terrakube variable inspection on 2026-07-02 verified staging notification queue/blob variables point at the shared-services storage account endpoints, while the staging workspace state and monorepo resources remain separate.

If staging changes after the accepted commit above, rerun the same full Azure Staging Smoke workflow before production cutover. The lighter public health check is useful for drift checks but does not replace the strict smoke.

GCP-To-Azure Feature Mapping

Every non-1:1 mapping below is explicit because it needs either cutover smoke, runtime evidence, or a rollback decision before final launch readiness.

Existing Terraform feature	Azure production target	Mapping
Cloud Run services	Azure Container Apps	Close 1:1 for HTTP services; revision, scaling, ingress, and identity semantics differ.
Cloud Run service accounts and service IAM	User-assigned managed identities, Entra internal-auth app roles	Non-1:1; Entra tokens replace Cloud Run ID tokens for Azure service-to-service calls.
Cloud Scheduler	Azure Container Apps Jobs	Close 1:1 for scheduled invocations; job history and identity are Azure-specific.
Artifact Registry	Azure Container Registry	Close 1:1 for OCI image storage.
Secret Manager	Azure Key Vault with managed identity RBAC and private endpoint	Non-1:1; secret payloads stay out of Terraform state, and network/RBAC behavior is Azure-specific.
Cloud KMS for GCP-managed resources	Azure platform encryption plus Key Vault where the service needs keys	Non-1:1; Azure services use platform-managed encryption unless a service-specific customer-managed key path is added.
Cloud Logging	Log Analytics	Non-1:1 query, retention, and diagnostic-setting semantics.
Cloud Monitoring alerts and SLOs	Azure Monitor metric and scheduled-query alerts	Non-1:1; alert thresholds need tuning from Azure traffic and SLO resources are not a direct Terraform equivalent.
Cloud Trace	Application Insights and Container Apps OpenTelemetry	Non-1:1; OpenTelemetry is enabled in Azure, but trace sampling/export behavior differs from GCP.
GCS product images and Prolog cache buckets	Azure Blob Storage containers	Close for object storage; private containers and managed identity replace GCS IAM.
GCS backend bucket/CDN/signed-url product-image delivery	APIM product-image proxy to Azure Blob Storage	Non-1:1; APIM policy and managed identity replace Cloud CDN/backend-bucket behavior.
Pub/Sub notification topics	Azure Storage Queue plus Blob pointer envelopes	Non-1:1; shared-services workers consume queues, dereference pointers, and own poison handling. During the production VNet transition, production uses local storage private DNS zones plus producer-local private endpoints for the shared notification storage account.
HTTPS load balancer, URL maps, serverless NEGs, and Cloud Armor	Cloudflare plus APIM plus private Container Apps origins	Non-1:1; preserve auth headers, rate limiting, private origins, logs, and cost controls.
Terminal mTLS load balancer	Azure Application Gateway WAF_v2 plus retained GCP Private CA	Non-1:1; App Gateway forwards SHA-1 cert fingerprints for terminal lookup.
VPC, subnets, firewall, NAT, private Google access	Azure VNet, delegated subnets, private endpoints, private DNS links	Non-1:1; production is in a documented VNet transition until the legacy 10.90.* ranges are removed.
Cloud Spanner PostgreSQL interface	Azure PostgreSQL Flexible Server	Non-1:1 data system replacement; no data migration is required pre-launch.
Firebase Auth/App Check and Identity Platform tenants	Firebase retained	Not migrated. Use Azure managed identity plus Google WIF, never Google private keys.
GCP Private CA	GCP Private CA retained	Not migrated for now; Azure terminal-onboarding requests certs through Google WIF.
GCP Workload Identity Federation for GitHub/Terrakube	GitHub OIDC plus Azure federated credentials and Terrakube variables	Non-1:1; Azure deploy identity and Terrakube workspace variables own the Azure apply/deploy path.

Production Prep Status

The first production Azure origin deployment has already been applied. Treat this section as the prep checklist for continuing toward customer-facing cutover, not as proof that final production readiness is complete.

1. Complete: use the existing infra/tf/azure root for both staging and production, with separate Terrakube workspaces, states, and variable sets.
2. Complete: create monorepo-azure-prod with production-only variables and state.
3. Complete for origin deployment: create production Azure resource naming and DNS output values for api.peakpos.co and terminal.peakpos.co without moving customer-facing Cloudflare DNS.
4. Complete for origin deployment: create production Key Vault, Azure managed identities, and Google WIF trust for Firebase/App Check and GCP Private CA access.
5. Complete for production readiness: create production Azure PostgreSQL, service roles, grants, repo-owned seed/init data, geo-redundant backup, ZoneRedundant HA, and backup/restore drill evidence.
6. Complete for origin deployment: create production ACR, Blob Storage, notification queues, APIM Standard v2, Log Analytics, Application Insights, Azure Monitor alerts, and the terminal Application Gateway certificate/client CA lane.
7. Complete for the current workflow shape: GitHub Actions production deploy values are passed as workflow variables/secrets and arguments rather than inline production YAML constants.
8. Complete for current Azure public proof: production public health, credentialed merchant/support smoke, notification smoke, and terminal mTLS smoke pass through api.peakpos.co and terminal.peakpos.co.
9. Still required: keep GCP production healthy and ready for rollback until the temporary dev production image tag is replaced with a release tag and the final post-release cutover smoke passes.

Phase 1: Production Terraform Shape

Use the same Azure Terraform entry point for staging and production, with environment separation handled by Terrakube workspace variables and state.

Recommended path:

• Keep infra/tf/azure as the shared Azure root.
• Create a Terrakube workspace such as monorepo-azure-prod.
• Set environment = "prod" in the production workspace.
• Keep production state, variables, secrets, DNS, and applies independent from monorepo-azure-staging.
• Keep production deploy approval controls stricter than staging even though the Terraform folder is shared.
• Use the checked-in backend selector for the production workspace and prove the first monorepo-azure-prod Terrakube plan before any apply. The shared root is no longer pinned to staging, but the production workspace must still prove it is not reusing staging state or staging variables.

Current Production Workspace Prep

Last verified: 2026-07-02.

Use Terrakube as the source of truth for raw workspace IDs, subscription IDs, tenant IDs, Firebase project identifiers, and endpoint values. The production workspace must satisfy these invariants before any plan:

Area	Required production invariant
Terrakube	Workspace is monorepo-azure-prod, root is infra/tf/azure, default template is plan-only, and remote apply remains disabled until approval.
Backend	Production state selection uses backend-prod.hcl or Terrakube workspace selection and is proven without staging-state reuse.
Environment	environment=prod, production hostnames are set, and no variable value still points at a staging hostname.
Images	Final cutover must use a production image tag, not dev. The current dev tag is an explicit temporary exception for code propagation proof.
Firebase retained services	Production Firebase project and both production Firebase tenant IDs are set.
Google WIF	Production Azure IMDS plus Google WIF credential secret IDs are set for Firebase/App Check and GCP Private CA access.
Shared services	Production shared-services notification blob and queue endpoints are set.
Edge	apim_sku_name is an approved production SKU, not APIM Consumption or classic Developer.
Certificates	Production APIM and terminal Application Gateway certificate secret IDs are set before public hostname cutover.

Current Production Deployment Evidence

As of 2026-07-02, the monorepo-azure-prod workspace has applied the production Azure service rollout from main.

• Terrakube plan job 497 completed against workspace monorepo-azure-prod with 85 to add, 4 to change, 0 to destroy.
• Terrakube apply job 498 completed with 85 added, 4 changed, 0 destroyed.
• Terrakube plan job 500 completed the APIM Standard v2 blue/green plan with 83 to add, 7 to change, 0 to destroy. The 7 in-place changes were unchanged Container App environment values losing Terraform's sensitive-state marker; no replacements or destroys were planned.
• Terrakube apply job 501 completed the APIM Standard v2 blue/green apply from commit 4a657825555fb56bf6d39f87deb40815cee78222.
• Terrakube apply job 503 attempted the Premium removal but failed during classic APIM child API deletion with an Azure 412 Precondition Failed on the terminalonboarding API. The retry was required because APIM child resource deletion is eventually consistent.
• Terrakube apply job 504 completed the Premium removal retry with 0 added, 8 changed, 11 destroyed. The active Terraform gateway output is now https://peakpos-prod-apim-v2-uw504n.azure-api.net.
• Terrakube apply job 524 completed after importing the staging dev images into production ACR and setting the explicit temporary production override allow_prod_dev_microservice_image_tag=true.
• Terrakube apply job 530 recreated the intentionally empty production PostgreSQL server as Standard_D2ds_v5 with 35-day retention, geo-redundant backup enabled, and ZoneRedundant HA healthy. The preceding failed attempts showed that enabling geo-redundant backup forces PostgreSQL server replacement and that Azure rejects HA on burstable PostgreSQL SKUs.
• Terrakube apply jobs 564 and 566 completed against monorepo-azure-prod on 2026-07-02 after the latest production reconciliation. The workspace remains unlocked, points at main, uses Terraform 1.15.6, and uses folder infra/tf/azure.
• Terrakube apply job 587 completed on 2026-07-02 after the production-local shared notification private endpoint fix merged in PR 2189. The apply added production-owned privatelink.queue.core.windows.net and privatelink.blob.core.windows.net private DNS zones in peakpos-prod-rg, moved the production blob/queue VNet links off the central shared-services zones, and created producer-local private endpoints for ppsharedfsor6vst queue/blob access in peakpos-prod-rg.
• Shared-services Terrakube apply job 588 completed on 2026-07-02 with 2 to add, 0 to change, 0 to destroy, granting the production monorepo runtime identity Storage Queue Data Contributor and Storage Blob Data Contributor on the shared notification storage account.
• Terrakube staging apply job 590 completed as a no-op after PR 2190; staging reported No changes. Your infrastructure matches the configuration.
• Terrakube production apply job 591 completed after PR 2190 with 0 added, 1 changed, 0 destroyed, updating only the terminal Application Gateway mTLS SSL profile.
• Terrakube production apply job 595 completed after PR 2192 with 0 added, 10 changed, 0 destroyed, updating APIM Standard v2 API policies to expose the Azure edge marker response header.
• Terrakube staging apply job 596 completed after PR 2192 as a no-op; staging reported No changes. Your infrastructure matches the configuration.
• Terrakube staging apply job 620 completed after PR 2197 from commit da227ed7174d7eee1ed3b6a5f14d3e1c01345f98, proving the shared GCP root still applies cleanly with staging runtime, load balancing, Spanner, Scheduler, and product storage disabled.
• Terrakube production apply job 623 completed after PR 2197 from commit da227ed7174d7eee1ed3b6a5f14d3e1c01345f98, completing the approved GCP production wind-down. Earlier wind-down applies imported the existing pinpointpos-invoice-view-token-signing-key Secret Manager container into prod state, disabled Cloud Run and Spanner deletion protection, enabled GCS force_destroy for the app buckets, and removed unmanaged Spanner backups that otherwise blocked instance deletion. Live verification on 2026-07-02 found no prod pinpointpos-* Cloud Run services, no pinpointpos-spanner instance, no pinpointpos-product-images-* or pp-prolog-cache-* buckets, no prod GCP forwarding rules, and no prod Cloud Scheduler jobs.
• Firebase and GCP Private CA were intentionally retained after the GCP runtime wind-down. Terraform state still owns the production Firebase project, Firebase Android/web apps, Identity Platform tenants, and Firebase/Identity Toolkit project services. Live API checks still show Firebase services enabled and the production Private CA pool/subordinate CA present for Azure terminal issuance through Google WIF.
• Production Azure resource group is peakpos-prod-rg; it is separate from staging and shared-services resource groups. Live resource-group inspection on 2026-07-02 found monorepo production resources tagged and owned in peakpos-prod-rg, shared notification workers and platform resources in peakpos-shared-services-rg, and Terrakube/IaC bootstrap resources in rg-peakpos-iac-use1.
• Production APIM Standard v2 gateway is https://api.peakpos.co. It is StandardV2 capacity 1 in peakpos-prod-rg, integrated with snet-apim-v2 (10.100.8.0/24) on peakpos-prod-vnet, and has the production custom hostname api.peakpos.co bound through Key Vault-managed certificate material. The active Terraform gateway selection is apim_active_gateway = "standard_v2" and enable_classic_apim = false.
• The classic Premium APIM instance peakpos-prod-apim-uw504n has been destroyed from peakpos-prod-rg.
• Production API hostname output is api.peakpos.co; customer-facing Cloudflare now routes API traffic to the Azure APIM Standard v2 edge.
• Production terminal hostname output is terminal.peakpos.co; positive and negative Azure Application Gateway mTLS smoke are proven.
• All ten production Container Apps are temporarily running image tag dev from production ACR so the latest code changes can be proven before release-tag cutover: ai-api, auth, customer-api, kitchen-api, management-api, merchant-api, status, terminal-api, terminal-onboarding, and tx-bundler.
• https://peakpos-prod-apim-v2-uw504n.azure-api.net/auth/health returned 200 OK with {"status":"UP","service":"auth"}.
• https://peakpos-prod-apim-v2-uw504n.azure-api.net/status/health returned 200 OK and reported all ten services healthy.
• https://api.peakpos.co/status/health returned 200 OK on 2026-07-02 and reported all production Azure services healthy while production remains in pre-cutover/dev-image proof mode.
• Live private DNS inspection after Terrakube job 587 found ppsharedfsor6vst.queue.core.windows.net resolving in peakpos-prod-rg to 10.90.6.9 and ppsharedfsor6vst.blob.core.windows.net resolving to 10.90.6.10. The central shared-services queue private DNS zone no longer has a production VNet link. This prevents production Container Apps from resolving the shared notification queue hostname to the production Key Vault private endpoint IP during the overlapping 10.90.0.0/16 transition.
• Production PostgreSQL was intentionally cleared because there is no launched customer data for POS. Temporary reset job execution peakpos-prod-pg-reset-4dglj7n truncated public tables with RESTART IDENTITY CASCADE, then normal bootstrap execution peakpos-prod-pg-bootstrap-p414036 succeeded. The temporary reset job was deleted and the bootstrap timeout was restored to 300 seconds.
• After the HA/geo-backup replacement, bootstrap execution peakpos-prod-pg-bootstrap-d0asmu9 succeeded against the recreated database.
• Production PostgreSQL backup/restore drill completed on 2026-07-02: customer-initiated backup peakpos-prod-restore-drill-20260702 completed at 2026-07-02T13:52:29.102490Z; point-in-time restore target peakpos-prod-pg-restore-0702 restored to Ready in peakpos-prod-rg with public network access disabled, the production PostgreSQL subnet, and the production PostgreSQL private DNS zone; restored catalog included peakpos; one-shot validation execution peakpos-prod-pg-restore-check-hq8hzjp connected from the production Container Apps environment through Key Vault secret wiring and returned public_table_count=208 and service_role_count=4; temporary validation job peakpos-prod-pg-restore-check and temporary restored server peakpos-prod-pg-restore-0702 were deleted after evidence capture.
• Production Container App env inspection found no SPANNER_* or PGADAPTER_* variables on any service; all services expose PostgreSQL connection variables.
• Terrakube variable inspection on 2026-07-02 verified production uses environment=prod, api_subdomain=api, terminal_subdomain=terminal, apim_active_gateway=standard_v2, enable_classic_apim=false, PostgreSQL HA/geo-backup production settings, and shared-services notification queue/blob endpoints. It also verified the explicit temporary exceptions: microservice_image_tag=dev, allow_prod_dev_microservice_image_tag=true, additional_virtual_network_cidrs=["10.90.0.0/16"], mixed network_cidrs, and enable_iac_vnet_peering=false.
• Azure Log Analytics console error query over the first post-apply window returned no application error rows. Transient ProbeFailed system logs were observed during revision startup, then the last five-minute system error query was clean.
• Production scheduled Container Apps Jobs were created and provisioned, including support cleanup, stale cleanup, invite expiry, orphan reconciliation, integration sync, expiry alert, loyalty sweep, invoice reminders, appointment jobs, diagnostics cleanup, low-stock sweep, campaign send, and marketing trigger evaluation.
• Production origin-mode smoke now defaults to the active Standard v2 gateway: AZURE_ENVIRONMENT=prod ./scripts/azure-staging-smoke.sh --allow-prod returned 203 ok, 5 warnings, 0 failures on 2026-07-02 after Terrakube job 544, and again at 2026-07-02T13:41:24Z. Those earlier warnings were expected pre-cutover items before the api.peakpos.co APIM custom hostname was bound.
• Production public API health through Cloudflare returned 200 OK from https://api.peakpos.co/status/health after Terrakube job 587, with all ten Azure services reported healthy.
• Production credentialed smoke preflight against the Standard v2 origin at 2026-07-02T13:41:24Z found the runtime runnable but missing operator inputs for the credentialed probes: MERCHANT_TOKEN, MANAGEMENT_TOKEN, ORG_ID, STORE_ID, TERMINAL_ID, and AZURE_SAFE_EMAIL|AZURE_SAFE_PHONE.
• Production strict public smoke passed in GitHub Actions run 28624538554 against https://api.peakpos.co and https://terminal.peakpos.co after Terrakube job 591: infrastructure smoke and credentialed smoke both passed. Credentialed smoke reported 16 ok, 0 warnings, 0 skipped, 0 failures.
• The Azure edge bootstrap workflow now supports an explicit prod target and uses the production resource group, deploy identity, APIM Standard v2 name, and monorepo-azure-prod Terrakube workspace variables. The repository variables AZURE_PROD_APIM_NAME and AZURE_PROD_APIM_BASE_URL were corrected to the Standard v2 gateway on 2026-07-02.
• Terminal issuance still uses the GCP Private CA subordinate pinpointpos-subroot-ca in pinpoint-payments/us-east1/pinpointpos-subroot-ca-pool. Azure Application Gateway requires exactly one self-signed root CA in trusted_client_certificate.data, so production Terrakube stores the DER-base64 encoded PinpointPOS Root CA in terminal_trusted_client_ca_base64. The accepted root SHA-256 fingerprint is FC:D7:FB:A8:68:E2:AB:5F:88:44:13:61:7C:83:D2:7D:07:4B:E4:75:92:DA:77:FD:0A:91:C4:49:DD:3B:95:BD. Do not replace it with the subordinate or with a certificate chain; Azure rejects those as containing zero root CAs.
• Production terminal negative mTLS smoke completed on 2026-07-02 against the Azure Application Gateway public IP 52.173.53.187 with SNI/Host terminal.peakpos.co: a no-client-certificate request to https://terminal.peakpos.co/terminal/health returned 400 No required SSL certificate was sent from Microsoft-Azure-Application-Gateway/v2. This proves the production Azure listener requests/requires a client certificate before forwarding traffic.
• Production credentialed smoke run 28624021021 isolated the terminal mTLS blocker: terminal-onboarding issued a GCP Private CA terminal certificate from Azure, but the Application Gateway returned HTTP 400 before forwarding to terminal-api because the gateway was still configured to verify the client certificate immediate issuer DN. The uploaded trusted certificate is the private root while the leaf is issued by the retained GCP Private CA subordinate. PR 2190 left chain validation enabled but disabled the extra issuer-DN match so terminal-api can enforce terminal identity through the forwarded SHA-1 fingerprint and database lookup.
• Production credentialed smoke run 28624538554 then passed. It minted a merchant Firebase token for a bootstrapped demo context, passed merchant and support auth route probes, proved payment config/gateway binding/product create-update workflows, issued a terminal certificate with SHA-1 fingerprint 4177c60548389486dc400923df29350498a5337e, proved terminal mTLS health and terminal config through Azure Application Gateway, accepted a safe-recipient receipt, and observed a recent shared-services email provider success log.
• Production smoke run 28625392199 passed from main after PR 2192. Infrastructure smoke reported 207 ok, 4 warnings, 0 failures and verified X-PeakPOS-Azure-Edge: apim-standard-v2. Credentialed smoke again reported 16 ok, 0 warnings, 0 skipped, 0 failures, including terminal onboarding, terminal-api mTLS acceptance, and shared-services email provider success.

Current Checked-In Gaps

These gaps are from the checked-in infra/tf/azure root and live production evidence as of 2026-07-02. Items marked as proven no longer block the initial Azure origin deployment, but unresolved items still block customer-facing DNS cutover or a final production-readiness claim.

Area	Current state	Production prep gap
Terrakube backend	Proven. monorepo-azure-prod planned and applied from infra/tf/azure on main with production state.	Keep future production applies in the production workspace; do not reuse staging state or staging variables.
Workspace isolation	Proven for initial origin deployment. The production workspace applied environment=prod and created peakpos-prod-* resources.	Keep default behavior plan-first and require explicit production apply approval.
Resource group	Proven for initial origin deployment and rechecked live on 2026-07-02. Production resources are in peakpos-prod-rg; staging resources are in peakpos-staging-rg; shared notification workers and their shared platform resources are in peakpos-shared-services-rg; Terrakube/shared private DNS bootstrap resources are in rg-peakpos-iac-use1. Azure-managed ME_* Container Apps resource groups exist per environment.	Finish the VNet transition below before claiming final network steady state.
Shared private DNS	Proven for initial origin deployment. Shared private DNS zones live in rg-peakpos-iac-use1; production uses central links for ACR and Key Vault, but uses production-owned storage private DNS zones in peakpos-prod-rg for blob and queue while the production VNet still overlaps shared-services on 10.90.0.0/16. Terrakube job 587 moved the production blob/queue VNet links to those local zones and created production-local private endpoints to the shared notification storage account. Staging continues to use the central shared-services storage private DNS path. While production still carries the overlapping transition range, shared-services Terrakube bootstrap owns a temporary IaC-subnet private endpoint plus AKS CoreDNS override for peakposproduw504nkv.vault.azure.net; Terrakube job 541 applied it and executor pods resolved the vault to 10.70.17.6.	Keep the links environment-specific and do not link the same prod VNet to both central and local storage private DNS zones. After the VNet transition, re-enable IaC peering and remove the temporary CoreDNS override/private endpoint once Terrakube reaches production private endpoints through normal routing.
APIM	Proven for initial origin deployment, Standard v2 cost cutover, production custom hostname binding, header preservation, and credentialed public smoke. Standard v2 is live at https://api.peakpos.co; aggregate service health checks are green through Cloudflare. Classic Premium APIM has been destroyed. Standard v2 capacity 1 is approximately $700/month plus overage after the included request allowance.	Keep Cloudflare/API rollback records ready while production is still using the temporary dev image tag.
APIM headers	Proven. APIM policies emit the environment-neutral X-PeakPOS-Azure-Edge marker, and production smoke run 28625392199 verified apim-standard-v2 through https://api.peakpos.co.	Keep this assertion in public smoke so logs and downstream diagnostics identify Azure edge traffic without staging labels.
Terminal edge	Positive and negative mTLS are proven for the Azure production edge. enable_terminal_application_gateway=true is applied in production; peakpos-prod-terminal-appgw and peakpos-prod-terminal-appgw-pip exist, provisioned successfully in Terrakube job 544, and prod smoke sees the terminal Application Gateway and public IP. The listener trusts the DER-base64 encoded PinpointPOS Root CA while terminal issuance remains on the GCP Private CA subordinate. No-client-certificate rejection was proven on 2026-07-02 against the Azure Application Gateway IP. Production run 28624538554 proved terminal-onboarding issuance, Application Gateway mTLS admission, and terminal-api cert-hash lookup through terminal.peakpos.co.	Keep monitoring App Gateway failed requests and terminal-api auth failures after production traffic moves.
Firebase	Firebase is intentionally retained after the GCP runtime wind-down. Terraform state still owns the production Firebase project, Android/web apps, Identity Platform tenants, and Firebase/Identity Toolkit project services; live API checks still show Firebase services enabled. Production Firebase/App Check inputs are supplied through Terrakube/Key Vault/Google WIF rather than checked-in values.	Confirm Firebase authorized domains and credentialed production auth flows after Cloudflare cutover.
GCP Private CA	Private CA is intentionally retained for terminal issuance after the GCP runtime wind-down. Live checks on 2026-07-02 found the production Private CA pool/subordinate CA still present. Azure terminal-onboarding uses Google WIF rather than Google private keys. Production smoke run 28625392199 proved terminal onboarding plus terminal-api mTLS through Azure Application Gateway.	Keep monitoring terminal issuance and App Gateway client-certificate failures after production traffic moves.
PostgreSQL	Azure PostgreSQL is the POS production data store for the Azure deployment. Services are wired to PostgreSQL and no Spanner/PGAdapter env vars are present. Production tables were cleared and repo-owned bootstrap completed. Production is on Standard_D2ds_v5 with 35-day retention, geo-redundant backup enabled, ZoneRedundant HA healthy, and a completed 2026-07-02 backup/restore drill. Terraform validation rejects HA-disabled production plans and HA on burstable SKUs.	Keep recurring restore drills in the operations cadence; no longer blocking the initial Azure production readiness claim.
Spanner cutover	Proven and wound down. Production Azure services expose PostgreSQL env vars and no SPANNER_* or PGADAPTER_* env vars. Terrakube production GCP apply job 623 destroyed the prod Spanner database and instance after unmanaged Spanner backups were removed. Live verification on 2026-07-02 found no pinpointpos-spanner instance in pinpoint-payments.	No GCP Spanner rollback remains for POS. Rely on Azure PostgreSQL backups, the completed restore drill, and the rollback routing records that do not require Spanner.
Notification mapping	Pub/Sub is replaced by Azure Storage Queue plus Blob pointer envelopes and shared-services workers. Production Container Apps publish to the shared notification storage account over production-local private endpoints and production-owned storage private DNS zones while the VNet transition is incomplete. Shared-services job 588 grants the prod monorepo runtime identity queue/blob contributor access on the shared storage account. Production run 28624538554 accepted a safe-recipient receipt and observed a recent shared-services email provider success log.	Continue monitoring poison handling and alert coverage checks for the shared-services workers.
DNS and certificates	Staging has scripts for issuing/importing public certificates and configuring Cloudflare DNS. Production APIM has api.peakpos.co bound through Key Vault certificate material and Cloudflare reaches the Azure APIM Standard v2 edge. Production terminal listener certificate material is imported, no-client-certificate rejection is proven through the Application Gateway public IP, and positive terminal mTLS is proven through terminal.peakpos.co.	Keep Cloudflare rollback records and certificate rotation runbooks ready while production is still using the temporary dev image tag.

The Terraform root enforces the most dangerous defaults as validation errors: production plans fail if they still use staging hostnames, the staging Firebase project, the dev image tag without the explicit allow_prod_dev_microservice_image_tag break-glass flag, or non-production APIM SKUs. Remove the break-glass image flag before final production release cutover. Do not finalize the release cutover until the temporary dev image tag is replaced, origin smoke, terminal mTLS smoke, Firebase auth/App Check smoke, and rollback routing are proven against the release image.

Current Production Network Transition

The first production apply created several network-dependent resources while the production VNet still used 10.90.0.0/16, which overlaps the shared-services VNet already peered with the IaC VNet. The target steady-state production range is 10.100.0.0/16, but Azure does not allow delegated or allocated subnets to be moved in place once APIM, Container Apps environments, or private endpoints occupy them.

As of the 2026-07-01 production apply attempts, the production workspace uses a temporary mixed-network state:

• additional_virtual_network_cidrs = ["10.90.0.0/16"] so the VNet can hold both the old and new address spaces during reconciliation.
• network_cidrs keeps in-use subnets on their current 10.90.* prefixes while unused or movable subnets use 10.100.*.
• enable_iac_vnet_peering = false until 10.90.0.0/16 is removed from peakpos-prod-vnet; Azure rejects peering to the IaC VNet while the prod VNet still overlaps shared-services.

This is not the final production network design. The steady-state cleanup is:

The one intentional non-1:1 exception in this transition is shared notification storage private access. Staging can use central shared-services storage private DNS because its VNet does not overlap the shared-services address space. Prod cannot safely use that same DNS link while both prod and shared-services contain 10.90.0.0/16; the shared queue hostname previously resolved to the same numeric IP as the prod Key Vault private endpoint from inside the prod VNet. Until prod exits the overlapping range, Terraform keeps prod-local storage private DNS zones and prod-local private endpoints for the shared notification storage account in peakpos-prod-rg.

1. Bring up or recreate production resources on 10.100.* subnets without sharing staging resources.
2. Remove the active allocations that keep snet-aca-public, snet-aca-terminal, snet-private-endpoints, and snet-apim on 10.90.*.
3. Move those subnets to the 10.100.* prefixes.
4. Remove additional_virtual_network_cidrs.
5. Re-enable IaC VNet peering and verify private DNS/private endpoint access.

Production variable differences:

• environment = "prod".
• production hostnames:
  • api.peakpos.co
  • terminal.peakpos.co
• api_subdomain = "api" and terminal_subdomain = "terminal".
• production Firebase project/tenant IDs.
• production Key Vault and secret IDs.
• production Gateway URLs.
• production mKonnekt callback secret ID.
• production alert recipients.
• production PostgreSQL SKU, backups, retention, high availability, and PITR.

Phase 2: Edge And DNS

Choose and configure the production API edge before any DNS cutover.

Allowed production options:

• APIM Standard v2 or Premium with private backend reachability.
• Another explicitly approved design that preserves private backend access, rate limiting, auth header preservation, and operational visibility.

Not allowed without a new design review:

• APIM classic Developer.
• APIM Consumption pointed at public service origins.
• Public Container Apps origins protected only by obscurity.

APIM Cost Mitigation

The initial production origin deployment used classic APIM Premium capacity 1 because that preserved the private-backend edge behavior needed for the first production proof. This was expensive: the Central US retail price checked on 2026-07-02 was about $3.829/hour, or approximately $2,795/month at 730 hours.

The target cost-reduction path is APIM Standard v2. As of 2026-07-02, the Standard v2 instance is created, has private backend reachability, passes public API health through api.peakpos.co, and is the active Terraform gateway. The classic Premium APIM has been destroyed. Custom-domain and certificate binding are complete for api.peakpos.co through Key Vault-managed certificate material.

Completed APIM cost-reduction work:

1. Created a new APIM Standard v2 instance in peakpos-prod-rg, not in a staging or shared-services resource group.
2. Configured outbound VNet integration/private backend reachability to the production Container Apps origins.
3. Recreated APIM APIs, policies, rate limits, logging, and health probes.
4. Ran production origin health smoke against the Standard v2 gateway hostname.
5. Removed Premium APIM after the Standard v2 path was accepted. This is complete as of Terrakube job 504.
6. Bound api.peakpos.co to Standard v2 with Key Vault-managed certificate material and verified public API health through Cloudflare after Terrakube job 587.

Remaining APIM cutover work:

1. Keep Cloudflare/API rollback records ready until the final release-tag smoke passes.

Do not flip apim_sku_name on the existing Premium APIM directly. Azure does not support in-place upgrade from classic APIM tiers to v2 tiers, so an in-place Terraform SKU change risks replacing the active production edge. The expected steady-state APIM cost after the classic instance is removed is approximately $700/month plus request overage after the included Standard v2 request allowance.

DNS plan:

• Keep Cloudflare in front.
• Keep rollback records ready while Azure production is running the temporary dev image tag.
• Verify Cloudflare TLS mode, origin certificates, and CNAME/A records after each edge change.
• API DNS now routes api.peakpos.co through the Azure APIM Standard v2 edge.
• Terminal DNS now routes to the Azure Application Gateway mTLS edge; keep the proven positive and negative terminal smoke in the final release-tag cutover checklist.

Phase 3: Runtime And Data

Because there is no launched customer data, production setup can initialize Azure PostgreSQL from repo-owned schema and seed paths rather than migrating Spanner data.

Completed for the current origin deployment:

• Applied SQLDelight schema/init to Azure PostgreSQL production.
• Created service roles and grants.
• Seeded repo-owned bootstrap data after intentionally clearing the empty POS production database.
• Verified all Azure production services expose PostgreSQL variables and do not expose SPANNER_* or PGADAPTER_* variables.

Remaining before final launch readiness:

• Rerun credentialed merchant/support/customer smoke against the final production release image tag.
• Continue monitoring production notification delivery through the shared-services Azure worker path. The safe-recipient path is proven, but oversized payload pointer envelopes and poison-message handling remain non-1:1 operational checks to keep in the shared-services runbook.

Phase 4: Secrets And Identity

Completed for the current origin deployment:

• Created production Key Vault.
• Populated production runtime secrets out of band.
• Configured Azure managed identities for runtime, internal service auth, image pull, Key Vault access, and Google WIF.
• Configured Google WIF trust from production Azure managed identity to the production Google/Firebase service account.

Remaining before final launch readiness:

• Re-audit Google roles for Firebase Auth, App Check, and Private CA after the production smoke pass.
• Keep break-glass Google JSON credentials unset by default.

Phase 5: Deploy Automation

Completed for the current origin deployment:

• Added GitHub Actions variables/secrets for Azure production deploy.
• Imported the current dev images into production ACR for temporary propagation proof.
• Wired deploy workflow arguments rather than inline production values in YAML.
• Wired Terrakube runs from the production workspace.

Remaining before final launch readiness:

• Push and deploy a production release image tag to the production ACR.
• Remove allow_prod_dev_microservice_image_tag and set microservice_image_tag to the release tag or SHA.
• Confirm Terrakube auto-plan/auto-apply behavior and production approvals match the desired merge policy.
• Keep the production smoke workflow green in both pre-cutover origin mode and post-cutover public mode.

Required production workspace variables:

• environment = "prod".
• microservice_image_tag pinned to the production release tag or SHA for final cutover. The current dev tag is only allowed while allow_prod_dev_microservice_image_tag=true is set for temporary code propagation proof.
• container_app_min_replicas and container_app_max_replicas sized for production.
• postgresql_sku_name = "GP_Standard_D2ds_v5" or another non-burstable HA-capable SKU sized for production.
• postgresql_storage_mb, backup retention, HA, and PITR values sized for production.
• postgresql_backup_retention_days = 35.
• postgresql_geo_redundant_backup_enabled = true.
• postgresql_high_availability_mode = "ZoneRedundant".
• Leave postgresql_standby_availability_zone unset unless Terraform also manages the primary zone. Azure exchanges primary and standby zones during a failover, and Terraform preserves that Azure-managed zone pair.
• firebase_project_id, firebase_project_number, identity_user_tenant_id, and identity_admin_tenant_id pointed at retained production Firebase.
• private_ca_project_id, private_ca_location, private_ca_pool, and private_ca_certificate_authority pointed at retained production GCP Private CA.
• gateway_base_url, gateway_pay_url, gateway_auth_url, and gateway_card_present_url pointed at production Gateway until Gateway itself moves.
• merchant_gateway_webhook_url and management_gateway_webhook_url using https://api.peakpos.co/....
• mkonnekt_api_key_secret_id points at the production-specific Key Vault secret before mKonnekt is enabled.
• Temporary network transition variables only while the prod VNet still has live 10.90.* subnets:
  • additional_virtual_network_cidrs = ["10.90.0.0/16"]
  • mixed network_cidrs preserving in-use 10.90.* subnet prefixes
  • include api_management_v2 when the Standard v2 blue/green subnet is not using the steady-state default 10.100.8.0/24
  • enable_iac_vnet_peering = false
• APIM cost-reduction variables:
  • apim_active_gateway = "standard_v2"
  • enable_classic_apim = false
  • apim_standard_v2_capacity = 1 initially, then scale only from live gateway metrics
• shared_private_dns_zone_resource_group_name = "rg-peakpos-iac-use1", with production-managed private DNS VNet links for ACR, Blob, Key Vault, and Queue. Do not point production at the deprecated staging queue-link alias.
• key_vault_secret_ids, google_external_account_credentials_secret_ids_by_service, and optional Apple Wallet certificate secret IDs populated from production Key Vault.
• notification_storage_queue_endpoint, notification_storage_blob_endpoint, notification_email_queue, notification_sms_queue, and notification_payloads_container pointed at the production shared-services notification resources.

Production secret and WIF bootstrap order:

1. Apply a production Azure foundation plan with services still gated if the runtime managed identity does not exist yet.
2. Use the Azure outputs runtime_identity_principal_id and google_runtime_wif_audience to enable production Azure runtime WIF in the retained GCP project. The audience is a dedicated Entra application ID URI, not the runtime managed identity client ID:
   • azure_runtime_wif_enabled = true
   • azure_runtime_wif_pool_id = "azure-production"
   • azure_runtime_wif_provider_id = "runtime-managed-identity"
   • azure_runtime_wif_principal_id = <runtime_identity_principal_id>
   • azure_runtime_wif_audience = <google_runtime_wif_audience>
3. Apply the GCP production workspace so Firebase/Auth/App Check/Private CA grants exist for az-prod-* Google service accounts.
4. Run infra/scripts/bootstrap-azure-runtime-secrets.sh to write production secret payloads to Azure Key Vault and update Terrakube with only Key Vault secret IDs:
   • key_vault_secret_ids
   • google_external_account_credentials_secret_ids_by_service
5. Re-plan the monorepo Azure production workspace with services enabled. A clean plan should no longer fail the Container Apps secret-ID precondition.

Phase 6: Cutover Runbook

Pre-cutover:

• Freeze production deploys outside the cutover lane.
• Confirm GCP production rollback remains healthy.
• Confirm Azure production public API smoke against origin hostname.
• Confirm terminal mTLS positive and negative cases.
• Confirm Firebase authorized domains and app configuration.
• Confirm Gateway webhooks/callback URLs for POS production.

Cutover:

• Update Cloudflare DNS/API routing to Azure origin.
• Run production public smoke.
• Run terminal mTLS smoke.
• Run authenticated merchant/support/customer smoke.
• Watch APIM, Container Apps, PostgreSQL, Key Vault, and Application Gateway metrics/logs.

Rollback:

• Repoint Cloudflare DNS/API routing to GCP production.
• Leave Azure production running for investigation unless it is actively harmful.
• Record exact failed smoke, request IDs, and Azure correlation IDs.

Phase 7: Post-Cutover Hardening

• Confirm backups and restore drills for Azure PostgreSQL.
• Confirm alert routing to the right on-call destination.
• Confirm Key Vault public network access remains disabled.
• Confirm ACR, Storage, Key Vault, and PostgreSQL private endpoints are healthy.
• Confirm cost monitoring is enabled against Microsoft startup credits.
• Update operational docs and support runbooks to prefer Azure production.

Gateway Handoff Gate

Gateway Azure work starts after monorepo Azure staging has the required terminal, database, and runtime parity evidence. Gateway planning must preserve these constraints:

• Gateway keeps Firebase where it is part of the existing contract.
• Gateway continues to use AWS Payment Cryptography.
• Gateway starts with an Azure staging environment before production.
• Gateway production cutover remains independently rollbackable from monorepo production cutover.