Troubleshooting Common API Platform Issues

Overview

This document provides a symptom-first index for troubleshooting common operational issues on the platform. It is intended for engineers responsible for operating and maintaining the platform.

Each section describes a common symptom, explains its likely causes, and provides a step-by-step recovery path. The instructions include specific commands for diagnosing the issue and gathering structured log evidence. If a provided solution does not resolve the issue, escalate to your account contact with the requested logs.

Error Code Prefixes

The platform includes operator-readable error codes in all error responses and structured logs. The prefix of the code indicates the component or surface where the error originated.

Prefix	Surface	Meaning
S2R-ADM-####	Admin API	Admin-plane errors (auth, validation, RBAC, settings)
S2R-RUN-####	Runtime	REST-runtime errors (timeout, payload size, mapping, upstream)
S2R-LIC-####	License	License-validation errors (expired, over-cap, signature invalid)
S2R-WRK-####	Worker	Background job errors (aggregation, ingest derivation)
S2R-DEPLOY-####	Deploy / startup	Migration, env-var binding, secret-resolution errors

Common Error Examples

• S2R-RUN-0413: The synchronous request payload exceeded 30MB. For large payloads, use the asynchronous transport mechanism.
• S2R-RUN-0408: The request to the upstream backend timed out. Adjust the timeout and retry settings under Services → Runtime settings.
• S2R-ADM-0419: The envelope key is not bound. This indicates the S2R_CREDENTIAL_KEY environment variable is missing on the current revision. Redeploy the service with the variable set.
• S2R-LIC-0001: The platform license has expired. Upload a renewal token under Settings → License.
• S2R-LIC-0002: The service-count limit for the Community Edition has been reached. Upgrade to a different edition or remove an existing service to proceed.

A complete catalog of error codes is available in the Admin API documentation. You can access it at https://s2r.your-domain.com/admin/v3/api-docs and search for errorCodes.

Symptoms and Recovery Paths

Admin UI Lockout

Symptom: You can log in via your Identity Provider (IdP), but the admin UI displays an "Access denied" message or does not show your administrative role.

This recovery path enables an operator to bootstrap a new administrator.

1. Access a shell on the host that runs the platform's deployment tooling.

2. Set the bootstrap environment variable on the admin-api service and trigger a redeployment.

   ./scripts/deploy-gcloud.ps1 `
       -Service admin-api `
       -ProjectId your-project-id `
       -Region your-region `
       -AdminBootstrapEmails first.admin@your-org.com
   

3. The admin-api service will start in a special recovery authentication mode. When the user with the specified email (first.admin@your-org.com) logs in, they will be granted the admin role, and the platform will automatically switch back to protected mode.

4. To confirm, navigate to Settings → Auth Mode and verify that the mode is protected.

If the mode does not automatically revert to protected, redeploy the admin-api service without the -AdminBootstrapEmails parameter.

F5 Ingest Has Stopped

Symptom: The dashboard shows no F5 traffic for the last 5-10 minutes, and the F5 Live Traffic Log is empty.

The platform uses three independent ingest targets. All three must be running the same service image. Verify each one.

1. Public Ingest Service: Check the image digest of the s2r-worker service.

   gcloud run services describe s2r-worker --region=your-region --format='value(status.imageDigest)'
   

   Compare this digest to the one reported during your last successful deployment. If it's stale, redeploy the worker service.

2. Edge Ingest Service: Check the image digest of the s2r-f5-ingest service.

   gcloud run services describe s2r-f5-ingest --region=your-region --format='value(status.imageDigest)'
   

   If the digest is stale, redeploy the service.

3. Relay VM Container: SSH into the relay VM and check the running container's image digest.

   sudo docker images --digests \
     your-region-docker.pkg.dev/your-project-id/your-repo/s2r-worker | head -3
   

   If the digest does not match the expected one, run the refresh helper script to pull the :dev tag and restart the container.

   sudo /opt/update_relay.sh
   

Other Potential Causes:

• Relay VM Disk Full: Check disk usage with df -h. If Docker logs (e.g., /var/lib/docker/containers/*-json.log) are consuming all space, truncate them and restart the container.

  sudo find /var/lib/docker/containers -name "*-json.log" -exec truncate -s 0 {} +
  

• Gateway Syslog Misconfiguration: Verify that your gateway's outbound syslog feed is still configured to point to the platform's ingest endpoint and that no network changes have interrupted the path.
• Alert vs. Reality: An alert may be based on a single signal. To verify authoritatively if traffic is flowing, query the database directly. If new rows are present, ingest is working.

  SELECT MAX(created_at) FROM s2r_f5_request_log;
  

License Expired or Community Edition Cap Hit

Symptom: A banner appears on every page of the UI, and onboarding new services is blocked.

Existing services will continue to run without interruption. Only the onboarding of new services above the license cap is blocked.

Recovery Workflow:

1. Contact your account representative to obtain a renewal token for a trial extension or an enterprise upgrade.
2. The token will be provided as a JWT string or a .token file.
3. In the admin UI, navigate to Settings → License → Upload renewal token. Paste the JWT or upload the file.
4. The platform validates the token's signature against the public key hosted at license.specaria.io, applies the new license atomically, and removes the banner.

If the validation fails with error S2R-LIC-0003 (invalid signature), the token may have been corrupted. Request a new token from your account contact.

Service Fails to Start After an Update

Symptom: After applying an update from Settings → Updates, a service fails its rollout, and the platform requires manual intervention.

Until full update orchestration is available, you must manually pin traffic to a last-known-good revision.

1. Identify the failed service in the Settings → Updates view.

2. List recent revisions for the failed service to find the last-known-good one.

   gcloud run revisions list `
       --service=s2r-admin-api `
       --region=your-region `
       --format='table(name,status.conditions[0].status,creationTimestamp)' `
       --limit=10
   

3. Pin 100% of traffic to the last-known-good revision. Replace s2r-admin-api-00xxx-yyy with the correct revision name.

   gcloud run services update-traffic s2r-admin-api `
       --region=your-region `
       --to-revisions=s2r-admin-api-00xxx-yyy=100
   

4. Capture logs from the failed rollout and provide them to your account contact.

   gcloud logging read 'resource.labels.service_name="s2r-admin-api" severity>=ERROR' --freshness=10m
   

5. After the underlying issue is resolved (e.g., via an image patch), you can retry applying the update.

For Helm-based installations, use helm rollback s2r <previous-revision-number> -n s2r to revert to a previous release.

Discovered Services Fail to Onboard

Symptom: The Discovery view shows traffic for a service, but the onboarding wizard fails.

Common Causes:

• WSDL Fetch Failure: The platform could not fetch the WSDL from the URL inferred from traffic (e.g., received a 4xx or 5xx response). In the Onboard wizard → Schema step, click Use observed payloads to bypass the WSDL fetch and build a schema from observed traffic instead.
• WSDL Schema Validation Failure: The provided WSDL is malformed or contains unresolvable XSD imports. Upload a corrected WSDL file using the Manual WSDL Upload feature.
• Incorrect Character Set: The WSDL file is not encoded in UTF-8. Re-save the file with UTF-8 encoding and upload it manually.
• Filtered Traffic: The discovered traffic may be from a consumer IP address that is listed in the Excluded Consumer IP Ranges in Settings. This can result in stale data. Temporarily remove the exclusion and wait for the next aggregation cycle.

Dashboard Inconsistencies or Slow Performance

Symptom: Dashboard KPIs are inconsistent, or the page takes more than 30 seconds to load.

Cause: This typically indicates that the background aggregation refresh job, AggregationRefreshService, is failing or stalled.

To diagnose, check the logs for the s2r-worker service.

gcloud logging read \
  'resource.labels.service_name="s2r-worker" "AggregationRefreshService"' \
  --freshness=1h --limit=50

Look for repeated ERROR entries. A common cause is a schema drift from an incomplete database migration. Redeploying the worker service will apply any pending Flyway migrations and should resolve the issue.

If the refresh job appears healthy but numbers are still incorrect, note that global KPIs and per-service counters are derived from different aggregations. An inconsistency between them suggests one of the aggregations is stale.

High Latency on First Request (Cold Starts)

Symptom: The first request to a service after a period of inactivity takes 30-60 seconds, while subsequent requests are fast.

Cause: By default, Cloud Run services scale to zero instances when idle. A new request triggers a "cold start," which involves provisioning a new container instance. For services attached to a VPC, this warm-up period can be 30-60 seconds.

Fix: Configure the service to maintain a minimum number of running instances. This eliminates cold starts at the cost of an always-on instance.

gcloud run services update s2r-runtime `
    --region=your-region `
    --min-instances=1

For Helm-based deployments, set replicas: 1 as the minimum on the runtime deployment.

Log Analysis

Finding Logs in Cloud Logging

Use the following base filter in GCP Cloud Logging to find logs for platform services:

resource.type="cloud_run_revision"
resource.labels.service_name="s2r-admin-api"   # or s2r-runtime / s2r-worker / s2r-admin-ui
severity>=WARNING

You can add structured fields to narrow your search:

• Trace a single transaction: jsonPayload.correlationId="<uuid>"
• Filter for runtime errors: jsonPayload.errorCode=~"^S2R-RUN-"
• Find deployment logs: textPayload:"S2R-DEPLOY"

Structured Log Format

All log entries are JSON objects with a consistent structure. The correlationId field is propagated across services (admin-api, runtime, worker), allowing you to trace a single request through the entire system.

{
  "timestamp": "2026-04-29T10:15:32.123Z",
  "severity": "ERROR",
  "service": "admin-api",
  "correlationId": "...",
  "userId": "first.admin@your-org.com",
  "errorCode": "S2R-ADM-0419",
  "message": "...",
  "context": { ... }
}

Verifying Deploy Provenance

Each service logs its version information on startup. This is the most reliable way to determine what code is running. Look for a log line matching this format:

S2R-DEPLOY service=<svc> commit=<short> sha=<full> branch=<branch> timestamp=<iso> image-tag=<tag>

Alternatively, you can inspect the environment variables of a specific Cloud Run revision. This stamp is immutable for the revision.

gcloud run revisions describe <rev> `
    --region=your-region `
    --format='value(spec.containers[0].env)' | Select-String S2R_DEPLOY

See Also

• onboard-first-service.md
• Admin API Documentation and Error Code Catalog: https://s2r.your-domain.com/admin/v3/api-docs

---

All SOAP-to-REST docs