System Health API Endpoint

## Overview

The System Health endpoint provides a consolidated, read-only snapshot of the operational status of the platform's administrative components. It is designed to power user interface elements, such as a "System Health" dashboard, by offering a quick and low-cost way to assess the health of various system parts without performing expensive or side-effecting checks.

The endpoint aggregates status information by querying internal database tables that the administrative services already maintain. It does not poll downstream services directly, which ensures that its response is fast and that its availability is not dependent on the health of the components it is reporting on.

The health report includes the status of key components, a list of recent operational incidents derived from the audit log, and metadata about when the report was generated. To minimize database load from frequent requests, the endpoint employs a 30-second, in-memory cache for its response payload. The data presented is always scoped to the tenant context of the request.

## How It Works

A `GET` request to the `/admin/v1/system/health` endpoint triggers a sequence to gather and return a health snapshot. The process is designed for efficiency and data isolation between tenants.

### Request and Caching Flow

The following diagram illustrates the request handling, caching, and data aggregation flow:

```mermaid
flowchart TD
    A[GET /admin/v1/system/health] --> B{Resolve Tenant Context};
    B -- SaaS: Unbound Request --> K[HTTP 401 Unauthorized];
    B -- Tenant Resolved --> C{Fresh Cache Available?};
    C -- Yes (<30s old) --> D[Return Cached Response];
    C -- No --> E[Build New Health Report];

    subgraph "Data Aggregation"
        direction LR
        E --> F1[Check Cloud SQL];
        E --> F2[Check F5 Relay];
        E --> F3[Check DataPower SSH];
        E --> F4[Check Other Components...];
        E --> G[Load Recent Incidents];
    end

    subgraph "Data Sources"
        DB[(Platform Database Tables)]
    end

    F1 & F2 & F3 & F4 & G --> DB;

    G & F1 & F2 & F3 & F4 --> H[Assemble Payload];
    H --> I[Cache New Response by Tenant];
    I --> J[Return New Response];

1. Tenant Resolution: The system first identifies the tenant associated with the request. In a multi-tenant deployment, if the request cannot be bound to a specific tenant, the endpoint returns an error. In a single-tenant (appliance) deployment, it defaults to the single available tenant.
2. Cache Check: The system checks for a cached health report for the resolved tenant. The cache is stored in-process and keyed by tenant ID.
3. Cache Invalidation: A cached report is considered "fresh" if it was generated less than 30 seconds ago. If a fresh report exists, it is returned immediately.
4. Data Aggregation: If no fresh cache is found, the system builds a new report by executing a series of read-only SQL queries against the platform's database. Each query assesses the status of a specific component.
5. Response Caching: The newly generated report is placed into the in-memory cache, replacing the previous entry for that tenant, and then returned to the caller.

Component Health Assessment

The status of each component is determined by querying specific database tables for freshness or error indicators.

Component	Source Table(s)	Health Logic
Cloud SQL	N/A (direct query)	Executes SELECT 1 and measures latency. Status is warn if latency > 1500ms, err if the query fails.
admin-api	N/A	Always ok. If the endpoint responds, this component is considered healthy.
F5 Relay VM	s2r_f5_sync_state, s2r_traffic_log	Checks the age of the last heartbeat (relay_vm_health_ok_at). It cross-references the last traffic ingest time (observed_at) to avoid false alarms if the heartbeat is stale but traffic is flowing. Status can be ok, warn, err, or idle.
F5 Traffic Ingest	s2r_traffic_log	Checks the timestamp of the most recent log entry (observed_at). Status becomes warn if no traffic has been ingested for over 900 seconds (15 minutes).
DataPower SSH	s2r_datapower_service_map	Checks the timestamp of the last inventory update (updated_at). Status becomes warn if the data is over 48 hours old.
WSDL Probe Queue	s2r_f5_sync_state	Counts instances with a last_wsdl_probe_status of error. Status is warn if any errors are found.
Generated REST API Traffic	s2r_runtime_log	Checks the timestamp of the last API call (occurred_at). Status becomes warn if no traffic is seen for over 24 hours. This may be normal for idle environments.

Incident Reporting

The incidents array is populated by querying the s2r_audit_event table for up to 50 events from the last 24 hours where the result column is err.

API Reference

Get System Health

Returns a snapshot of the system's health.

GET /admin/v1/system/health

Responses

200 OK

A successful response contains a JSON object with the current health status.

Response Body Structure

Field	Type	Description
generatedAt	String	An ISO 8601 timestamp indicating when the report was generated.
cacheSeconds	Integer	The cache lifetime in seconds. The source hardcodes this to 30.
components	Array of Objects	A list of monitored system components and their statuses. See Component Object structure below.
incidents	Array of Objects	A list of recent error events. See Incident Object structure below.

Component Object

Each object in the components array has the following structure:

Field	Type	Description
name	String	The name of the component (e.g., "Cloud SQL", "F5 Relay VM").
status	String	The component's health status. See Troubleshooting for details.
lastCheckAt	String	An ISO 8601 timestamp of the last relevant event or check for this component.
latencyMs	Integer	The latency of the check in milliseconds. Primarily used for the "Cloud SQL" check.
message	String	A human-readable message describing the current status.

Incident Object

Each object in the incidents array is derived from an audit event and has the following structure:

Field	Type	Description
occurredAt	String	ISO 8601 timestamp when the event occurred.
actorEmail	String	Email of the user or principal that performed the action.
actorRole	String	Role of the actor.
actionType	String	The type of action that was attempted.
entityType	String	The type of entity the action was performed on.
entityId	String	The unique identifier of the entity.
correlationId	String	A correlation ID for tracing the request.
severity	String	The result of the event. For incidents, this is always err.
message	String	The error message associated with the event.

401 Unauthorized

This response is returned in a multi-tenant (SaaS) deployment if the request cannot be associated with a specific tenant.

Troubleshooting

Component Status Meanings

The status field for each component indicates its health.

Status	Meaning
ok	The component is operating as expected.
warn	The component is operational but may require attention. For example, a process is running but its data is stale, or a check experienced high latency.
err	The component is in a failure state. For example, a required process is not responsive or a critical check failed.
idle	The component is enabled but has not yet processed any data or performed any actions. This is often a normal state for new or unused features.
unknown	The health check for the component failed to execute, typically due to a database error. The message field will contain details from the underlying exception.

Common Component Issues

• F5 Relay VM:

  • A warn status with a message like "Heartbeat older than 10m" often indicates that the background worker process has scaled to zero due to inactivity. It does not necessarily mean the relay is down, especially if the message also notes that traffic is still flowing.
  • An err status indicates that the heartbeat is stale (e.g., >1 hour old) and no recent F5 traffic has been logged. This suggests a potential outage of the relay VM or a network connectivity issue.

• Cloud SQL:

  • A warn status indicates that a simple SELECT 1 query took longer than 1500ms, suggesting high database load or a network issue.
  • An err status means the application could not connect to the database.

• Generated REST API Traffic:

  • A warn status indicates that no traffic has been recorded for generated REST APIs in over 24 hours. As the status message notes, this is normal for environments where these endpoints are not actively being used and does not indicate a platform failure.

---

All SOAP-to-REST docs