Repo Operations

This is the single source for build/test commands.

Repo-wide

bazel build //...
bazel test //...
bazel build //apps/specifications/dafny:verify

scripts/local-validation.sh now creates /tmp/bazel-sandbox automatically for local Bazel sandbox compatibility.

Bazel remote cache vs. remote execution

Every build already uses the BuildBuddy remote cache (.bazelrc): a plain bazel build //... downloads any action CI already built, and cache-missed actions compile locally. It does not use the remote executors unless you ask with --config=rbe, which offloads cache-missed actions to the BuildBuddy Linux x86_64 pool.

To default remote execution on for your machine (Linux):

scripts/enable-rbe.sh   # appends to .bazelrc.local (gitignored; never affects CI)

On macOS, --config=rbe offloads JVM/web targets but cannot build Apple targets remotely. On any OS, Android targets must build locally for now (the pool has no Android SDK yet) — with rbe enabled, keep an Android build local by appending --remote_executor=:

bazel build --config=android_arm64 --remote_executor= //apps/android:android_app

Apps

bazel build //apps/...
bazel test //apps/...

Microservices

bazel build //apps/microservices/...
bazel test //apps/microservices/...

The auth Cloud Run service owns canonical /auth/merchant/*, /auth/support/*, and /auth/account/* routes without a load-balancer rewrite. Its change-password endpoint uses per-instance in-memory throttling (5 requests per client IP per 60s), so autoscaled instances enforce limits independently.

Staging Gateway + POS Smoke Readiness

Environments span four GCP projects: pinpoint-payments is POS production, peaksuite-staging is POS staging, pinpoint-gateway is Gateway production, and peakgateway-staging is Gateway staging. Staging Cloud Run services, schedulers, secrets, and the Spanner database (pinpointpos on instance pinpointpos-spanner) use the same names as production; the GCP project is the only environment discriminator. Container images still live only in pinpoint-payments' Artifact Registry and staging pulls them cross-project with the dev tag. Staging Identity Platform/Firebase is its own Firebase project inside peaksuite-staging. Because the staging move issued a new terminal CA chain, staging terminals must re-onboard, and staging data was rebuilt fresh rather than migrated.

After a staging Cloud Run deploy, run the operator smoke readiness script from the repo root:

scripts/staging-cloud-smoke-readiness.sh

The script defaults to PROJECT_ID=peaksuite-staging; override PROJECT_ID or GOOGLE_CLOUD_PROJECT only when pointing it at a non-default staging project.

The default run verifies staging Cloud Run readiness, public health endpoints, terminal-api mTLS health when a client cert/key is supplied, Gateway env/secret/image wiring, Gateway webhook URL wiring, recent webhook registration logs, and recent terminal-api transaction sync logs. It does not require card hardware.

The script treats these staging Cloud Run inputs as required for real-card readiness:

Area	Required value
Image tag	EXPECTED_STAGING_IMAGE_TAG, default dev, must match the active Cloud Run image tag for pinpointpos-merchant-api, pinpointpos-management-api, and pinpointpos-terminal-api in peaksuite-staging; pinpointpos-ai-api and pinpointpos-auth are checked when provisioned.
Gateway env	GATEWAY_ENVIRONMENT=STAGING, GATEWAY_BASE_URL=https://staging-api.peakgateway.co, GATEWAY_PAY_URL=https://staging-pay.peakgateway.co, GATEWAY_AUTH_URL=https://staging-api.peakgateway.co/auth, GATEWAY_CARD_PRESENT_URL=https://staging-api.peakgateway.co/card-present.
Webhook env	GATEWAY_WEBHOOK_AUTO_REGISTER=true and GATEWAY_WEBHOOK_URL set to the staging callback for merchant, management, and Peak webhooks.
Secret refs	GATEWAY_CLIENT_ID -> pinpointpos-gateway-client-id, GATEWAY_CLIENT_SECRET -> pinpointpos-gateway-client-secret, GATEWAY_WEBHOOK_SECRET -> pinpointpos-gateway-webhook-secret; staging secrets live in peaksuite-staging with the same names as production, and each must have an enabled latest version.
Inter-service env	TERMINAL_API_BASE_URL on merchant-api points at the staging terminal-api Cloud Run URL; MERCHANT_API_CLIENT_BASE_URL on management-api and terminal-api points at the staging merchant-api Cloud Run URL; ONBOARDING_SERVICE_BASE_URL on terminal-api points at the staging terminal-onboarding Cloud Run URL.

To add resource-level checks for a demo staging store and terminal:

export MANAGEMENT_TOKEN="<support/admin Firebase bearer token>"
export MERCHANT_TOKEN="<merchant Firebase bearer token>"
export ORG_ID="<demo org UUID>"
export STORE_ID="<demo store UUID>"
export TERMINAL_ID="<demo terminal UUID>"
export EXPECTED_STAGING_IMAGE_TAG=dev

scripts/staging-cloud-smoke-readiness.sh

Before real-card smoke, also confirm these device/store facts are known and match the support/onboarding records:

Fact	Source
Demo POS IDs	ORG_ID, STORE_ID, TERMINAL_ID, plus the active Gateway merchant/location/terminal/device binding for that store.
Payment config	gatewayName=PEAK_GATEWAY, device type N62 or N92, terminal IP, terminal port, processor name, receipt behavior, and paired device serial.
Android device inventory	DEVICE_ADB_SERIALS for connected USB/wireless ADB devices. Current demo candidates observed on 2026-05-28: N92 (N92, Wi-Fi 10.1.10.196) and 10.1.10.228:41905 (C20Pro). External N62 target is serial N620W310496 at 10.1.10.23.
Card permission	Explicit operator approval for test/real-card amount, processor-approved Nexgo state, and rollback path to legacy runtime.

To include wireless ADB inventory in the readiness output:

export DEVICE_ADB_SERIALS="N92 10.1.10.228:41905"

scripts/staging-cloud-smoke-readiness.sh

For a physical Android launch-only smoke that must not create onboarding context or run value movement:

bazel build --config=android_arm64 //apps/android:android_app

ANDROID_SERIAL=N92 \
PEAK_POS_AUTO_CREATE_DEMO_CONTEXT=0 \
PEAK_POS_ALLOW_ONBOARDING_SCREEN=1 \
PEAK_POS_ANDROID_APK_PATH=/home/jackn/IdeaProjects/monorepo/bazel-bin/apps/android/android_app.apk \
apps/android/app/src/androidTest/launch_smoke_test.sh

To include terminal activation without real card hardware, provide the terminal mTLS client certificate and key for the demo terminal:

export TERMINAL_CLIENT_CERT=/path/to/staging-terminal-client.crt
export TERMINAL_CLIENT_KEY=/path/to/staging-terminal-client.key
export ACTIVATION_PUBLIC_KEY_SPKI_B64="smoke-spki"

scripts/staging-cloud-smoke-readiness.sh

Receipt delivery is intentionally side-effect gated. Set one destination only when you want the script to send a test receipt:

export RECEIPT_EMAIL=ops-smoke@example.com
# or: export RECEIPT_PHONE="+15551234567"

scripts/staging-cloud-smoke-readiness.sh

Expected output: zero FAIL lines. WARN lines mean the script skipped an optional probe because credentials/resources were not supplied, or no recent logs were available in LOG_FRESHNESS (default 2h).

Per-service binaries, tests, and images (the monorepo has 9 Cloud Run services — auth, merchant-api, management-api, terminal-api, terminal-onboarding, tx-bundler, status, kitchen-api, ai-api; the examples below use the same :core / :core_test / :image / :image_tarball target pattern):

bazel build //apps/microservices/merchant-api:core
bazel test //apps/microservices/merchant-api:core_test
bazel test //apps/microservices/merchant-api:merchant_controller_endpoint_unit_coverage_test
bazel build //apps/microservices/merchant-api:image
bazel build //apps/microservices/merchant-api:image_tarball

bazel build //apps/microservices/management-api:core
bazel test //apps/microservices/management-api:management_controller_endpoint_unit_coverage_test
bazel test //apps/microservices/management-api:management_controller_endpoint_unit_coverage_test
bazel test //apps/microservices/management-api:health_controller_test
bazel test //apps/microservices/management-api:notification_service_test
bazel test //apps/microservices/management-api:terminal_service_smoke_test
bazel build //apps/microservices/management-api:image
bazel build //apps/microservices/management-api:image_tarball

bazel build //apps/microservices/status:core
bazel test //apps/microservices/status:core_test
bazel build //apps/microservices/status:image
bazel build //apps/microservices/status:image_tarball

bazel build //apps/microservices/terminal-api:core
bazel test //apps/microservices/terminal-api:core_test
bazel build //apps/microservices/terminal-api:image
bazel build //apps/microservices/terminal-api:image_tarball

bazel build //apps/microservices/terminal-onboarding:core
bazel test //apps/microservices/terminal-onboarding:core_test
bazel build //apps/microservices/terminal-onboarding:image
bazel build //apps/microservices/terminal-onboarding:image_tarball

bazel build //apps/microservices/tx-bundler:core
bazel test //apps/microservices/tx-bundler:core_test
bazel build //apps/microservices/tx-bundler:image
bazel build //apps/microservices/tx-bundler:image_tarball


### Status service operational notes

The status service (`apps/microservices/status`) is an internal health-check
aggregator deployed to Cloud Run as `pinpointpos-status`. It exposes two
identical endpoints (`GET /` and `GET /health`) that poll the Cloud Run Admin
API for the five other services and return a combined JSON status.

**Monitored services:** merchant-api, management-api, terminal-api,
terminal-onboarding, tx-bundler.

**Environment variables:**

| Variable               | Required | Default    | Description                      |
| ---------------------- | -------- | ---------- | -------------------------------- |
| `GOOGLE_CLOUD_PROJECT` | yes      | —          | GCP project ID for Cloud Run API |
| `GOOGLE_CLOUD_REGION`  | no       | `us-east1` | Region to query                  |

When `GOOGLE_CLOUD_PROJECT` is unset the service returns `"status": "healthy"`
with `"checks": "skipped"`. Authentication uses Application Default Credentials
(ADC) with `cloud-platform` scope.

**Response shape:**

```json
{
  "success": true,
  "data": {
    "status": "healthy | degraded",
    "services": {
      "merchant-api": "healthy | unhealthy",
      "management-api": "healthy | unhealthy",
      "terminal-api": "healthy | unhealthy",
      "terminal-onboarding": "healthy | unhealthy",
      "tx-bundler": "healthy | unhealthy"
    }
  }
}
```

The `pinpointpos-` prefix is stripped from service names in the response.

---

## Websites

Build/test commands are covered here, while dev/preview/serve details live in
`apps/websites/AGENTS.md` (customer, support, landing-page).

```bash
bazel build //apps/websites/...
bazel test //apps/websites/...
bazel test //apps/websites/portals/retail:lint
bazel test //apps/websites/portals/support:lint

Playwright behavior integration tests

Playwright behavior tests verify real end-user flows trigger successful backend calls and complete in the UI. These are a temporary exception to the Bazel-first flow until dedicated Bazel targets/wrappers are added for Playwright. Set E2E_EMAIL and E2E_PASSWORD when running them directly; the pre-commit hook auto-sets local dev defaults if these are unset.

cd apps/websites/portals/retail
pnpm exec playwright test --list
pnpm exec playwright test e2e/customers.spec.ts

cd ../support
pnpm exec playwright test --list
pnpm exec playwright test e2e/features.spec.ts e2e/terminals.spec.ts

For a current endpoint gap inventory, see ./playwright-behavior-coverage-gaps.md.

Android

bazel build --config=android_arm64 //apps/android:android_app
bazel build --config=android_x86_64 //apps/android:android_app
bazel test --config=android_x86_64 //apps/android/app/src/androidTest:android_app_instrumentation_test

Android build cache issues

If you see an error like:

.../rules_android+/tools/jdk/bootclasspath_android_only_system doesn't exist

clean Bazel's output tree and rebuild:

bazel clean --expunge
bazel build --config=android_arm64 //apps/android:android_app

This typically indicates a corrupted or missing generated bootclasspath in the Bazel cache.

IntelliJ / BazelBSP sync errors

If IntelliJ fails during sync with:

.../rules_android+/tools/jdk/bootclasspath_android_only_system doesn't exist

the IDE is usually using a different --output_base (its own Bazel cache).

Fix:

1. Find the output base from the error path (e.g. /home/you/.cache/bazel/_bazel_you/...).
2. Run:

bazel shutdown
bazel --output_base=/path/from/error clean --expunge
bazel --output_base=/path/from/error build @rules_android//tools/jdk:bootclasspath_android_only

3. Re-sync the Bazel project in IntelliJ.

Optional: Set a dedicated output base for IntelliJ in Bazel settings, e.g. --output_base=$HOME/.cache/bazel-intellij, and clean that path when needed.

Android Production Signing

This section documents the end-to-end Android production release signing workflow. The production keystore is custodied in GCP Secret Manager (see apps/android/docs/keystore-custody.md).

How CI signs releases

CI fetches the production keystore from Secret Manager on non-PR events. In .github/workflows/android.yml:

• Step: Fetch prod keystore (only when github.event_name != 'pull_request')
  • Uses: google-github-actions/secret-manager@v2
  • Fetches: projects/pinpointpos/secrets/android-prod-keystore/versions/latest
• Step: Write prod keystore file
  • Decodes the base64 secret payload and writes it to: apps/android/prod.keystore

For production release artifacts, build the production target (which is configured to use apps/android/prod.keystore for signing):

bazel build --config=android_arm64 //apps/android:android_app_prod

Manual signing override (hotfix / local release)

Prereqs:

• You must have access to Secret Manager secret android-prod-keystore in project pinpointpos.
• Follow the custody and two-person controls described in apps/android/docs/keystore-custody.md.

Fetch the keystore and write it to the expected path (do not print secret contents):

mkdir -p apps/android
gcloud secrets versions access latest \
  --project=pinpointpos \
  --secret=android-prod-keystore \
  | base64 -d > apps/android/prod.keystore
chmod 600 apps/android/prod.keystore

Then build the production artifact:

bazel build --config=android_arm64 //apps/android:android_app_prod

After the build, securely remove the file (and ensure it never ends up in shell history/log output).

Guardrail (non-negotiable)

• Never use apps/android/dev.keystore for production releases.
• Never commit apps/android/prod.keystore (it is intentionally gitignored).
• Keep the keystore on disk only for the shortest practical time (CI should treat it as a temporary build input).
• Never print the base64 secret payload (or any derived bytes) to logs.

Key generation (if a new keystore is required)

Generate a new keystore (alias must match what the build/signing expects):

# Bazel's android_binary(debug_key=...) expects alias 'androiddebugkey' and password 'android'
keytool -genkey -v -keystore apps/android/prod.keystore -alias androiddebugkey -storepass android -keypass android -keyalg RSA -keysize 4096 -validity 10000

Upload the resulting keystore into Secret Manager as a new version of android-prod-keystore (custody/format details in apps/android/docs/keystore-custody.md).

Rotation / compromise response

For rotation, rollback, and compromise response procedures (including IAM controls, versioning strategy, and incident steps), follow:

• apps/android/docs/keystore-custody.md

Key reference points:

• Keystore secret: android-prod-keystore (projects/pinpointpos/secrets/android-prod-keystore/versions/latest)
• CI access: github-actions-android@pinpointpos.iam.gserviceaccount.com (scoped roles/secretmanager.secretAccessor on the secret resource)

Libraries

bazel build //libs/...
bazel test //libs/...

Tools

bazel build //tools/...
bazel test //tools/...

Specs

bazel build //apps/specifications/...
bazel build //apps/specifications/dafny:verify

Device Management & Fleet Operations

Fleet monitoring

The support portal provides a fleet dashboard at /fleet that shows all terminals across an organization with live status, diagnostics, and geofence compliance. Data flows from the Android app via periodic heartbeats:

1. Android app sends POST /v1/terminal/heartbeat (via terminal-api, mTLS) every 60 seconds
2. terminal-api writes diagnostics to the terminal_diagnostics Spanner table
3. Fleet API endpoints (/api/v1/admin/fleet/{orgId}/devices) read from Spanner and return enriched device entries with the latest diagnostics snapshot

Remote restart

Two remote-restart operations are available from the support portal and management-api:

• Device reboot (POST .../devices/{terminalId}/restart): sends an FCM data message with action: REBOOT. The gateway-owned Peak Pay Android reference host calls DevicePolicyManager.reboot() via the gateway device-owner API.
• App restart (POST .../devices/{terminalId}/app-restart): sends an FCM data message with action: APP_RESTART. The gateway-owned Peak Pay Android reference host kills and restarts its own process.

Both operations require the terminal to be online and have a valid FCM token registered via POST /v1/devices.

Geofence setup and violation monitoring

Geofences define a circular boundary around a store location. When a terminal's reported GPS coordinates fall outside its assigned geofence radius, a violation record is created in the geofence_violations Spanner table.

• Create/update/delete geofences via /api/v1/admin/geofences endpoints
• List active violations via GET /api/v1/admin/geofences/violations/org/{orgId}
• Resolve violations manually via POST .../violations/{violationId}/resolve

The retail portal and support portal both display geofence violation alerts when terminals are detected outside their assigned boundaries.

Device diagnostics pipeline

The full diagnostics pipeline:

Android heartbeat (60s interval)
  -> POST /v1/terminal/heartbeat (terminal-api, mTLS)
  -> terminal_diagnostics table (Spanner)
  -> GET /api/v1/admin/fleet/{orgId}/devices (management-api)
  -> Support portal fleet dashboard

Diagnostics fields include: battery level, charging state, Wi-Fi SSID/strength, cellular signal, storage free/total, RAM free/total, CPU temperature, app version, OS version, and GPS coordinates.

Cloud Run Revision Cleanup

Use infra/scripts/cleanup-cloud-run-revisions.sh to prune old revisions while keeping:

• The newest N revisions per service (--keep, default 3)
• Any revisions currently serving traffic

The examples below target production (--project pinpoint-payments). Staging uses the same service names in project peaksuite-staging, so pass --project peaksuite-staging to clean up staging revisions.

Dry-run (no deletion):

infra/scripts/cleanup-cloud-run-revisions.sh \
  --project pinpoint-payments \
  --region us-east1 \
  --service pinpointpos-tx-bundler

Apply deletion for tx-bundler:

infra/scripts/cleanup-cloud-run-revisions.sh \
  --project pinpoint-payments \
  --region us-east1 \
  --service pinpointpos-tx-bundler \
  --keep 3 \
  --apply

Apply deletion for all Cloud Run services in region:

infra/scripts/cleanup-cloud-run-revisions.sh \
  --project pinpoint-payments \
  --region us-east1 \
  --all-services \
  --keep 3 \
  --apply