Deployment Runbook

The operational sequence for deploying and upgrading SOAP-2-REST by Specaria, and how to confirm exactly what is running afterward. Since v1.0.

The operational sequence for deploying and upgrading SOAP-2-REST by Specaria, and how to confirm exactly what is running afterward. Since v1.0.

This is the platform-agnostic procedure. For topology-specific install detail see GCP-native, Helm, or Docker Compose. For upgrades between releases, the upgrade guide is the canonical companion to this page.

Why deploy order matters

The four services share one PostgreSQL database, and admin-api owns the Flyway migrations. Migrations are applied on startup, forward-only. So the schema-owning service must reach the new schema version before the runtime and worker depend on it.

1. admin-api  → applies Flyway migrations, advances the schema version
2. runtime    → serves conversion traffic against the new schema
3. worker     → aggregations, discovery derivation, learned examples
4. relay/ingest (if used) → refreshed to match the new schema

Deploy in that order. On Kubernetes, deploy/upgrade admin-api first so the schema is ready before runtime and worker pods cycle; standard rolling-update ordering then handles the rest.

Standard deploy sequence

1. Build images

./scripts/build-images.ps1            # all services
./scripts/build-images.ps1 -Service admin-api,runtime   # only what changed

The build dispatches a per-service container build; it honors a -Service filter so you can rebuild only the services you touched, and -DryRun to print the commands without submitting.

2. Deploy services

./scripts/deploy-gcloud.ps1                 # all four services (admin-api first)
./scripts/deploy-gcloud.ps1 -Service admin-api   # single service
./scripts/deploy-gcloud.ps1 -DryRun         # walk the logic without deploying

The deploy script applies the correct service-account, secret, and provenance bindings, runs the Flyway migrations on first deploy (via admin-api), and verifies each revision reaches a healthy/Ready=True state before moving to the next service.

Use the script, not a manual gcloud run deploy. A manual deploy drops the operator-critical secret bindings and the provenance variables (see below).

3. Apply infrastructure (if changed)

./scripts/deploy-terraform.ps1

Run Terraform after gcloud. It aligns the load balancer, IAP backend, and supporting infrastructure. Only required when infrastructure changed.

For Helm, the deploy is helm upgrade --install against your values; for Compose, docker compose up -d (see the respective install pages).

Operator-critical environment variables — re-apply every deploy

On Cloud Run, a gcloud run deploy does not inherit env vars or secrets from the prior revision. Any value the services need at runtime must be re-applied on every deploy or the next deploy silently drops it. The deploy script handles this through a centralized secret-bindings list — do not apply operator-critical values inline via a one-off command, which is exactly the drift pattern that causes them to disappear.

The bindings the services need at runtime include:

S2R_DB_PASSWORD — database password (password-auth deployments).
S2R_CREDENTIAL_KEY — AES key for the pgcrypto-encrypted backend credentials (admin-api + runtime). Reuse the existing key value; regenerating it invalidates every encrypted credential row.
S2R_F5_VM_DB_PASSWORD — used by F5 relay-VM provisioning (admin-api + worker).

A dropped S2R_CREDENTIAL_KEY surfaces as S2R-ADM-0419 on the first backend-profile request after the deploy — see Troubleshooting. Full inventory in the Configuration reference.

Refresh the relay / ingest path after a worker change

If you run a traffic relay / ingest VM, it runs the same worker image as a local container and does not auto-update when the worker image is re-pushed. After any worker deploy that touches the database schema, refresh the relay so its embedded logic matches the new schema — otherwise it can silently stop writing ingest rows while everything else keeps working.

Refresh it with the persisted helper on the VM:

sudo /opt/update_relay.sh

There can be more than one ingest target on a given install (a public ingest service, an edge ingest service, and the relay-VM container). After a worker change, confirm they are all running the matching image digest. The detailed diagnosis steps are in Troubleshooting → "ingest stopped flowing".

Post-deploy verification

Run these after every deploy — treat "the deploy command returned 0" as incomplete until verified:

Provenance — confirm the new code is running (see below).
Services healthy / internal — each service Ready and (GCP-native) internal-ingress only.
Schema validated — admin-api came up cleanly and applied any pending migrations.
Golden-path call — run a representative service end to end.
Fresh rows landing — discovery/ingest is still writing. For relay/ingest:
```
SELECT MAX(created_at) FROM s2r_f5_request_log;
```
Dashboard loads — open the Operations Dashboard and confirm KPIs render.

Deploy provenance

Every revision the tooling deploys is stamped with immutable provenance variables — S2R_DEPLOY_COMMIT_SHA, S2R_DEPLOY_COMMIT_SHORT, S2R_DEPLOY_BRANCH, S2R_DEPLOY_TIMESTAMP, S2R_DEPLOY_IMAGE_TAG — and each service logs a single line on startup:

S2R-DEPLOY service=<svc> commit=<short> sha=<full> branch=<branch> timestamp=<iso> image-tag=<tag>

This is the authoritative answer to "which code is actually running?" — image tags and digests can drift; the env-var stamp is immutable per revision and lives alongside the running code. Inspect it via:

# Read the provenance block off a Cloud Run revision:
gcloud run revisions describe <revision> `
    --region=<region> `
    --format='value(spec.containers[0].env)' | Select-String S2R_DEPLOY

# Or grep the startup log line:
gcloud logging read 'resource.labels.revision_name="<revision>" "S2R-DEPLOY"' --freshness=30m

Rolling back

Application rollback: redeploy the previous image tags for admin-api, runtime, and worker. Mapping versions and generated contracts are preserved across upgrades, so a rollback restores prior behavior. (Helm: helm rollback s2r <revision> -n s2r.)
Schema rollback: migrations are forward-only. If a release introduces a schema change a rollback cannot tolerate, the release notes say so; otherwise restore from the pre-upgrade database backup.
Per-service mapping rollback is independent of any deploy — roll a single service's mapping back from the UI at any time.

See the upgrade guide for the full upgrade and rollback procedure.

All Specaria SOAP to REST docs

Loading…