Daily Operations and Platform Health Checks

Overview

This document provides a playbook for day-2 operations of the API platform, available since v1.0. It details the routine checks, monitoring surfaces, and maintenance tasks an operator performs to ensure platform health, manage capacity, and respond to events.

The core operational activities include:

• Monitoring Health: Using the Operations Dashboard to get a high-level view of traffic, errors, and performance.
• Investigating Issues: Drilling down from dashboards into live traffic logs to diagnose specific failures.
• Verifying Data Ingest: Ensuring that traffic data is continuously flowing into the platform.
• Managing the Service Catalog: Refreshing discovered services and importing new schema revisions.
• Platform Configuration: Adjusting system-wide settings like data retention and traffic filtering.
• Managing Updates: Applying platform updates in a controlled manner.
• Capacity Planning: Monitoring usage against baselines to ensure sufficient headroom.

Following these procedures helps maintain a stable and performant platform.

How the Operations Dashboard Works

The Operations Dashboard is designed for high performance, providing sub-second load times even with very large data volumes. This is achieved by separating the data presentation layer from raw log processing.

The platform continuously processes raw traffic logs in the background, calculating and storing key metrics in dedicated aggregation tables. The dashboard exclusively queries these pre-computed tables, not the raw logs. This architecture ensures that the user interface remains fast and responsive, as it does not need to perform expensive computations on demand.

flowchart TD
    subgraph "Data Ingest"
        A[Raw Traffic Logs]
    end
    subgraph "Platform Backend"
        B(Aggregation Process)
        C[Pre-computed Aggregation Tables]
    end
    subgraph "User Interface"
        D[Operations Dashboard]
    end

    A --> B
    B -- Periodically computes KPIs --> C
    C -- Serves fast queries --> D

1. Operations Dashboard

The Operations Dashboard is the primary surface for daily health checks.

URL: https://<your-hostname>/dashboard/

The dashboard provides a consolidated view of platform-wide traffic and health. Key components include:

Filters:

• Source: f5 gateway-edge / runtime / both
• Direction: inbound / outbound / both
• Time window: 1h / 24h / 7d / 30d

Key Performance Indicators (KPIs):

• Requests: Total request count.
• Errors: Total error count and error rate percentage.
• p95 latency: The 95th percentile request latency.
• Active services: The number of services with recent traffic.

Visualization Tiles:

• Traffic timeline: A time-series graph of request volume in 5-minute buckets.
• Top 5 services by traffic: A list of the most active services.
• Top 5 services by errors: A list of the services with the most errors.
• Failure breakdown: A breakdown of failures scoped to the virtual server (for gateway-edge traffic).

From the dashboard, you can click on any service to navigate to a Service detail view, which shows per-operation metrics and provides a link to recent logs filtered by correlation ID.

2. Ingest and Live Traffic Monitoring

Verifying that data is flowing into the platform and inspecting individual transactions are critical operational tasks.

Live Traffic Log

The live traffic log provides a near-real-time stream of individual requests.

URL: https://<your-hostname>/observability/f5-live/ (for f5_modernization install profile)

This view is useful for:

• Investigating a specific failure reported by an API consumer by filtering for a correlation ID.
• Confirming that a newly onboarded service is receiving and processing traffic correctly.
• Diagnosing issues related to payload encoding or character sets by inspecting request and response bodies.

Each log entry includes the correlation ID, service URI, HTTP method, status code, request/response headers and payloads, and latency.

Ingest Health Check

The dashboards and metrics rely on a continuous feed of data from traffic sources. To confirm that data is still being ingested, you can check the freshness of the raw ingest table.

Execute the following SQL query against the platform's database:

SELECT MAX(created_at) FROM s2r_f5_request_log;

If the returned timestamp is recent, data is flowing correctly. If the timestamp is stale, it indicates a problem with the data ingest pipeline.

3. Updating the Service Catalog

As backend services evolve, the platform's service catalog must be updated.

Re-running Discovery

The platform automatically aggregates discovered services from traffic on a 5-minute cycle. To force an immediate refresh, for example after a gateway configuration change, navigate to Discovery → Refresh. This action re-aggregates data from the recent time window to identify new services.

Importing a New WSDL Version

When an API publisher releases a new WSDL for a SOAP-based service, follow these steps to update the service definition:

1. Navigate to Services → [Your Service] → Schema.
2. Click Replace WSDL and either upload the new WSDL file or provide a URL to it.
3. The platform performs a diff between the new and existing schemas. It displays a summary of added, removed, or changed operations and types.
   • Existing field mappings are preserved for fields with matching names.
   • New fields are automatically mapped.
   • Removed fields are flagged for review.
4. After reviewing the changes, click Publish. This creates a new, immutable revision of the service.

To roll back to a previous version, navigate to Services → [Your Service] → Revisions, select the desired prior revision, and click Make default.

4. Platform Settings

Global platform settings are managed from the Settings page.

URL: https://<your-hostname>/settings/

The following settings are available for viewing or configuration:

5. Update Channel

The platform includes a mechanism for managing software updates.

URL: https://<your-hostname>/settings/updates/

The platform checks for available updates daily. The Updates tab displays the current version, any available newer versions, and associated release notes. You can configure the update mode:

• manual (Default): An operator must approve each update before it is applied. This mode is recommended for production environments to ensure updates are applied during planned maintenance windows.
• automatic: Updates are applied automatically within a configured maintenance window.

The update process is designed to be robust, including pulling new images, running database migrations, performing a rolling restart of services, verifying the health of each new revision, and automatically rolling back if a failure is detected.

6. Capacity Planning

The platform is engineered to handle a baseline of approximately 1 million calls per day (~12 requests/second sustained, with bursts up to ~100 rps). To ensure you have sufficient capacity, monitor the following indicators:

• Requests KPI: Check the total request count on the Operations Dashboard over a 24-hour or 30-day window and compare it to the baseline.
• p95 latency Trend: A steady, sustained increase in 95th percentile latency can indicate a saturating backend service or an undersized platform runtime.
• Runtime Instances/Replicas: To avoid cold-start latency, maintain a minimum of one warm instance (min-instances ≥ 1 on Cloud Run, ≥ 1 replica on Helm). For sustained high throughput, scale the runtime horizontally by increasing the number of instances.

Suggested Daily Checklist

1. Open the Operations Dashboard. Scan the main KPIs and the Top 5 errors tile.
2. If any metrics are abnormal, drill down into the affected service, inspect recent failures in the Live traffic log, and determine the appropriate action.
3. Check the License status banner for any upcoming expirations.
4. Review the Updates tab for pending updates. If in manual mode, schedule a maintenance window to apply them.
5. Verify ingest freshness with the SQL query: SELECT MAX(created_at) FROM s2r_f5_request_log;.
6. (Weekly) Review the Discovery queue for any new services that are ready to be onboarded.
7. (Monthly) Review Retention settings. The raw traffic archive can grow large on high-traffic installations.

Troubleshooting

If you encounter issues during operations, refer to the following conditions mentioned in the source:

• Ingest has stopped flowing: If the ingest health check query shows a stale timestamp. See troubleshooting.md#ingest-stopped-flowing.
• Update applied but service won't start: If a service fails to start after a platform update. See troubleshooting.md#update-applied-but-service-wont-start.

See Also

• Data retention
• Backup & restore
• Licensing
• Metrics & SLO reports
• Troubleshooting

All SOAP-to-REST docs