Daily Operations Playbook

Overview

This document outlines the essential daily operational tasks for API publishers managing the platform. It covers routine checks, configuration adjustments, and backup strategies to ensure the smooth and efficient operation of services on day 2 and beyond. (source: docs/customer/daily-ops.md)

1. Operations Dashboard

The Operations Dashboard is the primary interface for daily monitoring. It is designed for rapid loading, even with large data volumes, by reading from pre-computed aggregation tables. (source: docs/customer/daily-ops.md)

Dashboard URL

Access the dashboard at: https://s2r.your-domain.com/dashboard/ (source: docs/customer/daily-ops.md)

Filters

Filters located at the top of the page allow API publishers to refine the data displayed. (source: docs/customer/daily-ops.md)

Name	Type	Default	Description
Source	String	both	Filters traffic by origin: f5 (gateway-edge traffic), runtime (traffic through the platform's REST endpoints), or both.
Direction	String	both	Filters traffic by direction: inbound, outbound, or both.
Time window	String	24h	Sets the aggregation period: 1h, 24h, 7d, or 30d.

Key Performance Indicators (KPIs)

The top row of the dashboard displays key performance indicators for the selected time window: (source: docs/customer/daily-ops.md)

• Requests: Total number of requests.
• Errors: Count of errors and their percentage rate.
• p95 latency: The 95th percentile latency in milliseconds.
• Active services: Number of services that processed at least one request.

Tiles

The dashboard features several tiles for visualizing operational data: (source: docs/customer/daily-ops.md)

• Traffic timeline: Displays requests and errors stacked in 5-minute buckets.
• Top 5 services by traffic: A sortable list of services; clicking a row provides a per-operation breakdown.
• Top 5 services by errors: A sortable list by error rate or absolute count.
• VS-level failure breakdown: Shows virtual-server-scoped failures for gateway-edge traffic, useful for identifying issues with upstream backends.

Drill-down Capabilities

Clicking any service row opens the Service detail view, which provides per-operation traffic, error rate, latency, and a link to recent traffic logs filterable by correlation ID. (source: docs/customer/daily-ops.md)

2. F5 Live Traffic Log

The F5 Live Traffic Log provides near-real-time visibility into individual requests. (source: docs/customer/daily-ops.md)

Live Traffic Log URL

Access the live traffic log at: https://s2r.your-domain.com/observability/f5-live/ (source: docs/customer/daily-ops.md)

Availability

This feature is available when the install profile includes f5_modernization, which is the default setting. (source: docs/customer/daily-ops.md)

Displayed Information

For each request, the log shows: (source: docs/customer/daily-ops.md)

• Correlation ID (clickable for drill-down)
• Service URI
• Method and status code
• Request payload and headers
• Response payload and headers
• Latency

Use Cases

API publishers can use the F5 Live Traffic Log for: (source: docs/customer/daily-ops.md)

• Investigating specific failures reported by an API consumer, using the correlation ID filter.
• Spot-checking newly onboarded services to confirm they are sending data.
• Diagnosing issues related to payload encoding or character sets.

Non-F5 Traffic

For gateway-edge traffic not originating from F5, the Observability → Log Search feature provides similar functionality with a generic source. (source: docs/customer/daily-ops.md)

3. Updating the Service Catalog

Re-running Discovery

The platform automatically aggregates discovered services on a 5-minute cycle. To manually trigger a refresh, which re-aggregates data from the last hour: (source: docs/customer/daily-ops.md)

1. Navigate to Discovery.
2. Click Refresh.

This is useful after a gateway configuration change to confirm that new traffic is being recognized. (source: docs/customer/daily-ops.md)

Importing New WSDL Versions

When a backend team publishes an updated WSDL revision, API publishers can import it as follows: (source: docs/customer/daily-ops.md)

1. Go to Services → Your service → Schema.
2. Click Replace WSDL.
3. Upload or link the new WSDL file.
4. The platform will display a diff showing added, removed, or changed operations and types compared to the existing schema.
5. Review the diff. Existing field mappings are preserved if field names match, new fields are automatically mapped, and removed fields are flagged.
6. Click Publish to create a new immutable revision. Existing API consumers will continue to use the prior revision until the new one is marked as default.

Rolling Back WSDL Versions

To revert to a previous WSDL revision: (source: docs/customer/daily-ops.md)

1. Navigate to Services → Your service → Revisions.
2. Select the desired prior revision.
3. Click Make default.

4. Settings Management

The Settings page provides configuration options for the platform. (source: docs/customer/daily-ops.md)

Settings URL

Access the settings at: https://s2r.your-domain.com/settings/ (source: docs/customer/daily-ops.md)

Authentication Mode

This read-only display shows the current authentication mode: (source: docs/customer/daily-ops.md)

• pristine: Indicates a first-boot state with no administrator configured (transient, flips automatically).
• protected: The normal state where Role-Based Access Control (RBAC) is enforced.
• recovery: An operator-bootstrap path is active. Refer to troubleshooting.md for details on "I'm locked out".

Install Profile

The install profile determines which feature sets are visible. (source: docs/customer/daily-ops.md)

Name	Description
f5_modernization	Enables the F5-only feature set and hides generic-gateway tiles.
generic	Enables the generic-gateway feature set and hides F5-specific tiles.
both	Makes all tiles and features visible.

Saving a change to the install profile reloads the user interface. Changes are rarely needed, and existing services are unaffected. (source: docs/customer/daily-ops.md)

Topology Labels

API publishers can attach free-text labels (e.g., prod, staging, dr) to their installation. These labels are surfaced in audit logs and structured-log fields, allowing for filtering in Cloud Logging by topology. (source: docs/customer/daily-ops.md)

Excluded Consumer IP Ranges

This setting allows API publishers to specify a CIDR list of source IP addresses to exclude from traffic aggregation. This is useful for preventing internal monitoring or synthetic-test sources from skewing dashboard data, ensuring dashboards reflect real API consumer traffic. The input is CIDR-validated and saved atomically. (source: docs/customer/daily-ops.md)

License Information

The License tab displays details about the current platform license: (source: docs/customer/daily-ops.md)

• Current license: Shows the edition (e.g., Community Edition, Trial, Enterprise).
• Status: Indicates if the license is active, expired, or over-cap.
• Expiry: The license expiration date.
• Service count: Displays the current number of services versus the licensed cap.
• Renewal: Allows uploading a renewal token by pasting a JWT or uploading a .token file provided by the account contact. The signature is validated before application.

If a Trial license expires or a Community Edition service cap is reached, a banner will appear on every page until renewed, though existing services will continue to run. (source: docs/customer/daily-ops.md)

5. Update Channel

The platform checks for available updates daily against updates.specaria.io. (source: docs/customer/daily-ops.md)

Updates URL

Access the updates settings at: https://s2r.your-domain.com/settings/updates/ (source: docs/customer/daily-ops.md)

Update Information Displayed

The Updates tab shows: (source: docs/customer/daily-ops.md)

• Current version
• Available version (if newer)
• Release notes
• Update mode

Update Modes

The platform supports two update modes: (source: docs/customer/daily-ops.md)

Name	Description
manual	The default mode where an operator must approve each update.
automatic	The system applies updates automatically during a configured maintenance window.

Operator-Approved Updates (Manual Mode)

In manual mode, API publishers click Apply when an update is available. The platform then performs the following steps: (source: docs/customer/daily-ops.md)

1. Pulls the new container images.
2. Runs Flyway database migrations.
3. Rolls services one at a time.
4. Verifies that each new revision is Ready=True before proceeding to the next service.
5. Reports completion or automatically rolls back on failure.

Automatic Mode

Automatic mode is suitable for non-critical environments. API publishers configure a daily maintenance window (e.g., 02:00–04:00 local time), and the platform applies any pending updates within that window. A summary is emailed the next morning. (source: docs/customer/daily-ops.md)

The trade-off with automatic mode is that while it keeps the platform up-to-date faster, it lacks operator pre-flight checks. An unexpected migration on a heavily customized installation could lead to service interruption. It is recommended to default to manual mode unless running multiple staging environments to validate auto-updates first. (source: docs/customer/daily-ops.md)

6. Backups and Retention

The platform retains different artifact classes for varying default periods. (source: docs/customer/daily-ops.md)

Artefact Retention Policies

Artefact	Default retention	Configurable
Raw archive (full traffic, S3-equivalent bucket)	60 days	Yes — Settings → Retention
Per-service learned examples (curated)	Indefinite (capped per service)	Yes — Services → Your service → Examples
Aggregation tables (Dashboard / KPIs)	13 months	Yes — Settings → Retention
Audit log (operator actions)	13 months	Yes — Settings → Retention
Cloud Logging (structured-log stream)	Per your cloud project's retention policy	Cloud-side

Database Backups

For GCP-native installations, Cloud SQL automated daily backups are enabled by default with a 7-day retention period. For production environments, it is recommended to increase this to 30+ days within your Cloud SQL instance settings. (source: docs/customer/daily-ops.md)

For Helm installations, API publishers must configure backups on their Postgres operator or managed-Postgres instance, as the platform itself does not initiate database backups. (source: docs/customer/daily-ops.md)

Restoring Data

Database restoration is a Postgres-level operation (e.g., point-in-time recovery on Cloud SQL, or operator-managed for Helm). After a fresh database restore, the platform automatically recovers on its next deployment; Flyway detects the schema state and resumes from there. (source: docs/customer/daily-ops.md)

7. Daily Checklist (Suggested)

A suggested daily checklist for API publishers: (source: docs/customer/daily-ops.md)

1. Open the Operations Dashboard and review KPIs and the Top 5 errors.
2. If any item in the Top 5 is highlighted as problematic, drill into the service, check recent failures via the Live Traffic Log, and decide whether to remediate or accept the condition.
3. Glance at Settings → License for any expiring licenses.
4. Glance at Settings → Updates for any pending updates. If in manual update mode, schedule a maintenance window if an update is available.
5. (Weekly) Review the onboarding queue in Discovery for any new services ready for onboarding.
6. (Monthly) Review retention settings, especially for the raw archive, which can grow significantly in high-traffic installations.

Troubleshooting

This section outlines common conditions and their implications: (source: docs/customer/daily-ops.md)

• Trial license expiration or Community Edition service cap reached: The platform will display a banner on every page. Existing services will continue to run, but renewal is required to remove the banner.
• "I'm locked out": This condition relates to the recovery authentication mode. Refer to troubleshooting.md for resolution steps.
• "Update applied but service won't start": This indicates an issue during the update process. Refer to troubleshooting.md for further guidance.

---

All SOAP-to-REST docs