Troubleshooting Common Operational Issues

This document provides a symptom-first index for diagnosing and resolving the most common operational issues encountered when running the platform. It is intended for operators responsible for maintaining the platform's health.

Each section describes a symptom, explains its underlying cause, and provides a step-by-step resolution. To effectively use this guide, you should have access to the platform's structured logs and deployment environment.

If the guidance provided here does not resolve an issue, escalate to your support contact with the structured log evidence requested in each section.

Error Code Prefixes

Every error response and structured log entry includes a machine-readable error code that identifies the component where the error originated. Understanding these prefixes helps narrow down the scope of an investigation.

Prefix	Surface	Meaning
S2R-ADM-####	Admin API	An error occurred in the administrative plane, such as issues with authentication, validation, RBAC, settings, or the credential key.
S2R-RUN-####	Runtime	An error occurred in the REST runtime, such as a request timeout, excessive payload size, data mapping failure, or an error from an upstream service.
S2R-LIC-####	License	A license validation error occurred, such as an expired license, exceeding a usage cap, or an invalid signature.
S2R-WRK-####	Worker	An error occurred in a background job, such as data aggregation or ingest derivation.
S2R-DEPLOY-####	Deploy / startup	An error occurred during service deployment or startup, such as a database migration failure, environment variable binding issue, or a failure to resolve a secret.

The most frequently encountered error codes include:

• S2R-RUN-0413: A synchronous request payload exceeded the 30 MB limit.
• S2R-ADM-0419: The credential encryption key (S2R_CREDENTIAL_KEY) is not available to the running service revision.

"Request rejected — payload too large" (S2R-RUN-0413)

Symptom: An API consumer receives an HTTP 413 Payload Too Large response, and logs show the error code S2R-RUN-0413.

Cause: A synchronous API request was sent with a body larger than the fixed 30 MB ceiling. The runtime rejects the request before processing it.

Resolution:

1. Investigate the client: Confirm if the calling application is sending an unexpectedly large payload. This can be due to a client-side bug, embedding large attachments, or batching too many operations into a single request.
2. Find the rejection log: Use the correlation ID from the client's response to find the corresponding log entry. The log will have result_code=rejected and errorCode=S2R-RUN-0413.
3. Adjust the integration pattern: For legitimate transfers of large data, the integration must be changed. Options include splitting the payload across multiple smaller requests or using an asynchronous transport mechanism instead of a single synchronous call. The 30 MB limit is fixed in v1.0.

"Backend timeout" or Upstream Errors

Symptom: API consumers experience timeouts, often manifesting as HTTP 504 Gateway Timeout errors.

Cause: The platform's request to a backend service exceeded its configured timeout. The default timeout is 100 ms with 2 retries. A slow or unhealthy backend service is the typical cause.

Resolution:

1. Verify backend health: Before adjusting platform settings, confirm that the target backend service is healthy and performing within its expected latency profile.
2. Tune operation settings: If the backend is known to be slow but is otherwise healthy, you can adjust the timeout and retry settings on a per-operation basis. These settings are operator-configurable in the service's runtime configuration. Increase the values to accommodate the backend's latency. Avoid raising timeouts globally without first identifying the slow backend.

"Credential (envelope) key not bound" (S2R-ADM-0419)

Symptom: The platform fails to connect to a backend service, and logs show the error S2R-ADM-0419. This often occurs on the first request to a backend profile after a new deployment.

Cause: The S2R_CREDENTIAL_KEY environment variable, which is used to decrypt pgcrypto-encrypted backend credentials stored in the database, is missing from the running service revision. This commonly happens on platforms like Cloud Run, which do not automatically inherit environment variables or secret bindings from the previously active revision.

Resolution:

1. Redeploy correctly: Redeploy the service using the standard deployment script. This script is designed to re-apply all necessary secret bindings to the new revision. Avoid manual deployments (e.g., a direct gcloud run deploy command) that might omit these bindings.

   IMPORTANT

   Reuse the existing key value. Regenerating the S2R_CREDENTIAL_KEY will render all previously encrypted credentials in the database unreadable, causing a widespread outage for backend integrations.

2. Verify the binding: After deploying, confirm that the secret is correctly bound to the new revision. For Cloud Run, you can use the following command:

   gcloud run revisions describe <revision> `
       --region=<region> `
       --format='value(spec.containers[0].env)' | Select-String S2R_CREDENTIAL_KEY
   

   This command should output the environment variable configuration, confirming S2R_CREDENTIAL_KEY is present.

For more details, see the deployment runbook and configuration reference.

Ingest Stopped Flowing

Symptom: The live traffic log is empty, and dashboards show no gateway-edge traffic for the last several minutes.

Cause: Data is no longer being received or processed by the ingest components. This can be caused by a version mismatch between components, a full disk on a relay VM, or a misconfiguration in the upstream data source.

Resolution:

1. Verify data arrival: First, check the database to determine if any data is being written. An up-to-date timestamp indicates that data is arriving, and the issue may be with dashboarding or alerting, not ingest itself.

   SELECT MAX(created_at) FROM s2r_f5_request_log;
   

2. Check for mismatched image versions: If the timestamp is stale, check that all ingest components are running the same, correct image version. An installation may have multiple ingest targets (e.g., public, edge, relay VM), and all must match.

   • Public ingest service (Cloud Run):

     gcloud run services describe s2r-worker --region=<region> --format='value(status.imageDigest)'
     

   • Edge ingest service (Cloud Run):

     gcloud run services describe s2r-f5-ingest --region=<region> --format='value(status.imageDigest)'
     

   Compare the output digest to the one reported by your last successful deployment. Redeploy any service with a stale image.

   • Relay VM container: SSH into the relay VM, check the local Docker image digest, and if it's stale, run the update helper script. This script pulls the correct image and restarts the container.

     sudo /opt/update_relay.sh
     

   IMPORTANT

   After any worker service deployment that includes a database schema change, you must refresh any relay VMs. A stale relay will use outdated SQL queries that are incompatible with the new schema, causing ingest to fail silently.

3. Check for other common causes:

   • Relay VM disk full: Container JSON logs can fill the root partition. Truncate the logs and restart the container.

     sudo find /var/lib/docker/containers -name "*-json.log" -exec truncate -s 0 {} +
     docker restart <container_id>
     

   • Gateway feed paused: Ensure the upstream gateway's log target is still configured to send data to the platform's ingest endpoint and that network connectivity is intact.

Schema Mismatch After an Upgrade

Symptom: After a platform upgrade, dashboards show incorrect data, services fail to start with schema-related errors, or worker logs contain repeated errors.

Cause: The platform's services are not aligned with the current database schema version. This typically occurs when the deployment order is incorrect or when an ingest component was not refreshed after a schema change.

Resolution:

1. Follow the correct deploy order: The admin-api service owns and applies database schema migrations (using Flyway). It must always be deployed first.
2. Deploy admin-api: Deploy the admin-api service and confirm it starts successfully, indicating that migrations have completed.
3. Deploy other services: Once the schema is updated, redeploy the runtime and worker services.
4. Refresh ingest targets: Refresh any relay VMs or other edge ingest targets to ensure their embedded SQL queries match the new schema.

Dashboard Shows Incorrect or Stale Data

Symptom: Dashboard KPIs are inconsistent, show impossible values, or the UI pages take tens of seconds to load.

Cause: The background aggregation process, managed by the worker service, is stalled or failing. Dashboards read from pre-computed aggregation tables, and if this process fails, they will display stale data.

Resolution:

1. Check worker logs: Examine the worker service's logs for ERROR entries related to aggregation. A common cause is a schema mismatch following an incomplete upgrade.
2. Redeploy the worker: Redeploying the worker service can resolve transient issues and ensure it is running against the correct schema.
3. Isolate the stale aggregation: If numbers are still incorrect, compare global KPIs with per-service detail views. These are often sourced from different aggregation tables, and a mismatch can help pinpoint which aggregation is failing.

Administrator Lockout from Admin UI

Symptom: After authenticating successfully with your identity provider (IdP), the Admin UI does not grant you administrative access.

Cause: Your user account is not associated with the admin role in the platform's RBAC database.

Resolution:

Use the bootstrap recovery mechanism to regain access.

1. Redeploy admin-api in recovery mode: Redeploy the admin-api service with the S2R_ADMIN_BOOTSTRAP_EMAILS environment variable set to your email address.
2. Log in to recover access: When the service starts with this variable, it enters a temporary recovery authentication mode. Log in with the specified email address. The platform will automatically grant you the admin role.
3. Confirm mode flip: After you are granted access, the platform should automatically flip back to protected mode. You can verify this under Settings → Auth Mode in the Admin UI.
4. Redeploy to secure: If the mode does not revert automatically, redeploy the admin-api service again, this time without the S2R_ADMIN_BOOTSTRAP_EMAILS variable set.

Service Fails to Start After an Update

Symptom: After an update is applied, a service fails to start, and the platform may initiate a rollback to the previous version.

Cause: The new version of a service has a startup failure, which could be due to a misconfiguration, a bad migration, or an environmental issue.

Resolution:

1. Identify the failing service: Use the Settings → Updates page in the Admin UI to identify which service failed to roll out.
2. Pin to a good revision: Manually pin traffic to the last-known-good revision to stabilize the platform.
3. Capture logs: Collect the startup logs from the failed service revision. These are critical for diagnosing the root cause.
4. Escalate: Provide the logs to your support contact.
5. Retry the update: Once the underlying issue is resolved, you can retry the update. For Helm-based deployments, you can perform a rollback using a command like:

   helm rollback s2r <revision> -n s2r
   

Cold-Start Latency Spikes

Symptom: The first API request after a period of inactivity takes significantly longer than subsequent requests.

Cause: The service has been scaled to zero instances by the underlying container orchestrator (e.g., Cloud Run). The latency spike is the time required to provision a new container instance and, for VPC-attached services, perform network initialization.

Resolution:

Configure a minimum number of warm instances to prevent scale-to-zero behavior.

• Cloud Run: Set min-instances >= 1 on the runtime service.
• Helm/Kubernetes: Set replicas >= 1 on the runtime deployment.

Locating and Analyzing Logs

All platform components produce structured JSON logs, which are essential for troubleshooting.

Structured Log Format

Every log line is a JSON object containing standard fields that provide context for the event.

• service: The name of the component that generated the log (e.g., s2r-admin-api, s2r-runtime).
• correlationId: A unique ID that tracks a single transaction across all services.
• userId: The ID of the user associated with the request.
• errorCode: The machine-readable error code (e.g., S2R-RUN-0413).
• message: A human-readable description of the event.
• context: A JSON object containing additional, event-specific details.

The correlationId is propagated across all services, allowing you to trace a single request from the Admin API through the runtime, worker, and database.

Log Queries (Cloud Logging)

For GCP-native deployments, use the following filters in Cloud Logging to find relevant logs.

Base filter for all warnings and errors from a specific service:

resource.type="cloud_run_revision"
resource.labels.service_name="s2r-admin-api"   # or s2r-runtime, s2r-worker
severity>=WARNING

Useful additions:

• Trace a single transaction: jsonPayload.correlationId="<uuid>"
• Filter for runtime errors: jsonPayload.errorCode=~"^S2R-RUN-"
• Find the running code version: textPayload:"S2R-DEPLOY" (This log line appears once per service on startup and authoritatively identifies the running code version).

Container-Based Deployments

For Docker Compose or Helm/Kubernetes deployments, the same structured JSON logs are sent to the container's standard output. You can view them with standard tooling:

• Docker Compose: docker compose logs -f <service>
• Kubernetes: kubectl -n s2r logs deploy/<service>

Escalating an Issue

If the steps in this guide do not resolve your issue, gather the following information before contacting your support representative:

• The correlationId of the failed transaction.
• The errorCode from the log or API response.
• The deploy-provenance log line (textPayload:"S2R-DEPLOY") for the affected service to identify the exact code version.

See Also

• Deployment Runbook
• Configuration Reference
• Backup and Restore Procedures
• Platform Upgrade Guide
• RBAC & Roles
• Contacting Support

---

All SOAP-to-REST docs