Loading…
Loading…
A runbook for deploying and upgrading the platform's services, covering deployment order, verification, release cadence, and rollback procedures.
This document provides the standard operational sequence for deploying and upgrading the SOAP-2-REST platform services. It covers the required deployment order, post-deploy verification steps, release artifact cadence, and rollback procedures. Following this runbook ensures that database schema dependencies are respected and that each deployment can be verified.
This procedure is platform-agnostic. For environment-specific instructions, see the guides for GCP-native, Helm, or Docker Compose. This runbook should be used in conjunction with the Upgrade Guide when moving between releases.
The platform's services (admin-api, runtime, worker, relay/ingest) share a single PostgreSQL database. The admin-api service is responsible for managing and applying database schema migrations using Flyway. These migrations are applied automatically when admin-api starts.
Because admin-api owns the schema, it must be deployed and started before the other services. This ensures the database schema is at the correct version before services that depend on it (runtime, worker) are started.
The required deployment order is:
admin-api: Applies database migrations and updates the schema version.runtime: Serves conversion traffic against the potentially new schema.worker: Performs aggregations and other background tasks against the new schema.relay/ingest (if used): Refreshed to align with any schema changes.flowchart TD
subgraph Deployment Sequence
A[1. Deploy admin-api] --> B{Database Schema Migrated};
B --> C[2. Deploy runtime];
B --> D[3. Deploy worker];
B --> E[4. Deploy relay/ingest];
endNOTE
On Kubernetes, deploying admin-api first allows its pod to become ready and apply migrations before the standard rolling update process cycles the runtime and worker pods.
The platform follows semantic versioning (MAJOR.MINOR.PATCH). The release process for downloadable self-deploy bundles differs from that of individual container images.
1.6.0, 2.0.0), where the PATCH version is 0.1.6.1, 1.6.2), the downloadable bundles are not rebuilt. The release carries forward the artifacts from its minor version line. The download URL and sha256 checksum remain the same as the x.y.0 release. Only the container image digests for the individual services are updated.When verifying a download, associate it with its minor version line (e.g., "1.6.x") and use the sha256 checksum published for that line.
The following steps outline the standard deployment process using the provided tooling.
Build the container images for the services. You can build all images or filter for specific services that have changed.
# Build images for all services
./scripts/build-images.ps1
# Build images only for specified services
./scripts/build-images.ps1 -Service admin-api,runtime
The script supports a -DryRun flag to print the build commands without executing them.
Use the provided deployment script to deploy the services. This script ensures the correct deployment order and applies necessary runtime configuration.
# Deploy all four services, starting with admin-api
./scripts/deploy-gcloud.ps1
# Deploy a single service
./scripts/deploy-gcloud.ps1 -Service admin-api
The script also supports a -DryRun flag to review the deployment logic without applying changes.
IMPORTANT
Always use the deploy-gcloud.ps1 script for Cloud Run deployments. A manual gcloud run deploy command does not inherit environment variables or secrets from the previous revision. The script ensures that operator-critical bindings and provenance variables are correctly applied to every new revision.
If there are changes to the underlying infrastructure (e.g., load balancer, IAP), apply them using the Terraform script. This step should be run after the service deployment.
./scripts/deploy-terraform.ps1
For other environments:
helm upgrade --install with your values file.docker compose up -d.On Cloud Run, environment variables and secrets are not automatically inherited by new revisions. They must be explicitly re-applied with every deployment. The deploy-gcloud.ps1 script manages this using a centralized list of bindings.
The following variables are critical for service operation and must be present at runtime.
| Variable | Description | Services |
|---|---|---|
S2R_DB_PASSWORD | The password for the PostgreSQL database. Required for password-authenticated deployments. | admin-api, runtime, worker |
S2R_CREDENTIAL_KEY | The AES key used for pgcrypto encryption of backend credentials in the database. | admin-api, runtime |
S2R_F5_VM_DB_PASSWORD | The database password used by the F5 relay-VM provisioning process. | admin-api, worker |
CAUTION
The value for S2R_CREDENTIAL_KEY must be reused across deployments. Regenerating this key will render all existing encrypted backend credentials in the database unreadable. A missing or incorrect key will cause S2R-ADM-0419 errors.
If you operate a traffic relay or ingest VM, it runs a version of the worker image. This VM does not automatically update when a new worker image is deployed. After any deployment that changes the worker service, you must manually refresh the relay VM to ensure its embedded logic matches the current database schema. Failure to do so can cause data ingest to stop silently.
To refresh the relay, execute the following command on the VM:
sudo /opt/update_relay.sh
Confirm that a deployment is successful by running through the following checks. Do not consider a deployment complete just because the deploy command finished without error.
Ready=True) and configured correctly (e.g., internal-ingress only for GCP-native).admin-api started cleanly and applied any pending database migrations by checking its logs.runtime service to ensure core functionality is working.SELECT MAX(created_at) FROM s2r_f5_request_log;
Every service revision deployed by the tooling is stamped with a set of immutable environment variables that record its source code origin. These variables are the authoritative source for what code is running.
S2R_DEPLOY_COMMIT_SHAS2R_DEPLOY_COMMIT_SHORTS2R_DEPLOY_BRANCHS2R_DEPLOY_TIMESTAMPS2R_DEPLOY_IMAGE_TAGOn startup, each service logs a single line with this information:
S2R-DEPLOY service=<svc> commit=<short> sha=<full> branch=<branch> timestamp=<iso> image-tag=<tag>
You can inspect the provenance of a running revision using one of the following methods:
Inspect Environment Variables on Cloud Run:
gcloud run revisions describe <revision> `
--region=<region> `
--format='value(spec.containers[0].env)' | Select-String S2R_DEPLOY
Grep Startup Logs:
gcloud logging read 'resource.labels.revision_name="<revision>" "S2R-DEPLOY"' --freshness=30m
S2R-ADM-0419: This error, with the message "envelope key not bound", typically occurs on the first request to a backend profile after a deployment. It indicates that the S2R_CREDENTIAL_KEY environment variable was not correctly applied to the new admin-api or runtime revision. To fix, re-deploy the service using the deploy-gcloud.ps1 script.worker deployment, the relay is likely out of sync with the database schema. Refresh it by running sudo /opt/update_relay.sh on the VM.admin-api, runtime, and worker. Mapping versions and generated contracts are preserved across upgrades, so behavior will be restored. For Helm, use helm rollback s2r <revision> -n s2r.