Observability API Reference

API reference for querying traffic metrics, logs, SLOs, and operational dashboards. Provides endpoints for unified observability across all traffic

Overview

The Observability API provides a set of administrative endpoints for monitoring platform health, service performance, and troubleshooting traffic. It serves as the data source for the administrative UI's dashboards, SLO reports, and log viewers.

This API solves the problem of querying vast amounts of traffic data from multiple, disparate sources in a performant and unified manner. It ingests and aggregates data from various points in the request lifecycle, including external gateways (like F5), the platform's own runtime, and other generic connectors.

API consumers, such as the administrative UI or operational scripts, call these endpoints to:

Retrieve high-level Key Performance Indicators (KPIs) for traffic volume, error rates, and latency.
Generate Service Level Objective (SLO) compliance reports.
Search and view detailed logs for individual transactions.
Monitor the health of data ingest and aggregation processes.
Replay or manually manage asynchronous transactions.

How it Works

The Observability API employs several key mechanisms to provide fast and accurate data.

Query Path Selection: Raw vs. Aggregated Data

To ensure responsive dashboards and reports, the API dynamically chooses between querying raw, high-cardinality traffic logs and pre-computed hourly aggregation tables. The decision is based on the requested time window.

flowchart TD
    subgraph Observability API
        A[API Consumer] --> B{GET /admin/v1/metrics};
        B --> C{Time Window > 1 hour?};
        C -- Yes --> D[Query Hourly Aggregation Tables];
        C -- No --> E[Query Raw Traffic Logs];
        D --> F[Return Aggregated Metrics];
        E --> G[Return Precise Metrics];
        F --> H[Response];
        G --> H;
    end

Short-window queries (≤ 1 hour): These queries execute directly against raw log tables (e.g., s2r_traffic_log). This provides high-fidelity data, including precise percentile latencies, with breakdowns by individual service operation. Performance is maintained by indexed lookups on the time column.
Long-window queries (> 1 hour): These queries read from pre-computed hourly aggregation tables (e.g., s2r_f5_agg_service_hourly, s2r_runtime_agg_service_hourly). This approach is significantly faster for large time ranges as it avoids scanning millions of raw log rows.

IMPORTANT

For long-window queries that use aggregated data, latency percentiles are approximated. The p50 latency is derived from the mean (sum/count), while p95 and p99 latencies are represented by the maximum latency recorded in the bucket. This provides a worst-case proxy for performance. True percentile calculations are only available on short-window queries against raw logs.

Unified Traffic Sources

Many endpoints, such as /admin/v1/metrics and /admin/v1/dashboard/summary, consolidate data from all configured traffic sources (F5, Runtime, Generic). This is achieved via a UNION ALL query structure that combines results from each source's aggregation table. A source dimension in the result identifies the origin of each data point. Callers can use the source query parameter to filter results to a specific traffic source.

Error Classification

A transaction is classified as an error if any of the following conditions are met, as defined by the canonical ERROR_PREDICATE_SQL:

The HTTP status code is 4xx or 5xx.
The result_code is one of failed, backend_failure, error, timeout, or response_schema_violation.
The in_body_fault boolean flag is true, indicating a SOAP fault was detected in an HTTP 200 response payload during ingest.

This predicate is applied consistently across metrics, dashboards, and SLO reports.

Onboarded vs. All Traffic

The onboardedOnly=true parameter, available on the /admin/v1/metrics endpoint, changes query behavior to focus only on services formally configured in the platform.

When onboardedOnly=true, queries join with the s2r_service table. This ensures that metrics are grouped by the canonical service_id and reported against the operator-defined service_key and environment, correcting for any historical discrepancies in ingested data.
When onboardedOnly=false (the default), queries group by the service_key and environment values as they appear in the raw log or aggregation tables. This includes traffic for "shadow" services that have been automatically discovered but not formally onboarded.

Data Freshness and Ingest Health

The platform actively monitors its own data pipelines. The /admin/v1/dashboard/summary endpoint exposes health status for:

Aggregation Freshness: It checks if the traffic aggregation tables have been refreshed within a configurable threshold (dashboard_freshness_stale_minutes). A stale table can mean dashboard metrics are not up-to-date.
F5 Ingest Health: It checks the timestamp of the last received F5 log entry. A significant delay indicates a potential stall in the F5 syslog pipeline, which would render F5-sourced metrics untrustworthy.

These checks surface as urgentIssues in the API response if a problem is detected.

API Endpoints

Metrics and Dashboards

GET /admin/v1/metrics

Retrieves aggregated traffic metrics for services and operations over a specified time window. This is a primary data source for observability dashboards.

Request Parameters

Name	Type	Required	Default	Description
`window`	String	No	`15m`	Time window for the query. Valid values: `15m`, `1h`, `24h`, `7d`, `30d`, `60d`. Overridden by `start` and `end`.
`start`	String	No	—	ISO 8601 timestamp for the start of the time range (inclusive).
`end`	String	No	—	ISO 8601 timestamp for the end of the time range (inclusive).
`environments`	List<String>	No	—	A list of environments to filter by.
`serviceKeys`	List<String>	No	—	A list of service keys to filter by.
`source`	String	No	—	Filter by traffic source. Valid values: `f5`, `runtime`, `generic`. If omitted, results from all sources are combined.
`onboardedOnly`	Boolean	No	`false`	If `true`, results are limited to formally onboarded services and grouped by canonical service ID.
`direction`	String	No	—	Filter by service direction. Valid values: `publish`, `consume`. If omitted, services of all directions are included.

Response Body

Returns a JSON array of objects, where each object represents a service or service-operation aggregate.

[
  {
    "environment": "prod",
    "serviceKey": "my-service",
    "operationKey": "my-operation",
    "requestCount": 1000,
    "errorCount": 10,
    "p50LatencyMs": 150.5,
    "p95LatencyMs": 450.0,
    "p99LatencyMs": 800.0,
    "p95ConverterOverheadMs": null
  }
]

For queries with a window > 1 hour, operationKey will be (aggregated).
For queries with a window > 1 hour, latency percentiles are approximations based on sum, count, and max from aggregated data.

GET /admin/v1/dashboard/summary

Provides a comprehensive summary of platform health and traffic KPIs, designed to populate an operations dashboard.

NOTE

The data in this response is synthesized from multiple sources and time windows. For example, request counts are typically from a 24-hour aggregation, while p95 latency is a more recent 1-hour sample from raw logs for responsiveness.

Request Parameters

Name	Type	Required	Default	Description
`trafficThreshold`	Long	No	`50`	Minimum request count for a service to be considered as having traffic.
`p95LatencyTargetMs`	Double	No	`500`	P95 latency target in milliseconds, used for SLO breach calculation.
`availabilityTarget`	Double	No	`99.5`	Availability target percentage, used for SLO breach calculation.
`source`	String	No	—	Filter by traffic source. Valid values: `f5`, `runtime`, `generic`. If omitted, results from all sources are combined.
`direction`	String	No	—	Filter by service direction. Valid values: `publish`, `consume`. If omitted, services of all directions are included.

Response Body

Returns a complex JSON object containing KPIs, timelines, top error services, and operational alerts. Key fields include:

generatedAt: Timestamp of when the summary was generated.
kpis: A block of key performance indicators (e.g., totalServices, requestCount24h, errorPct24h, p95LatencyMs24h, f5IngestStale).
trafficTimeline24h: An array of hourly traffic and error counts for the last 24 hours.
topErrorServices: A list of the top 5 services with the most errors in the last 24 hours.
topVsLevelFailures24h: A list of top virtual-server-level connection failures, which are not attributable to a specific service.
urgentIssues: A list of high-priority operational issues, such as stalled data aggregation, stalled F5 ingest, SLO breaches, or high error rates.
dataFreshness: An object detailing the last refresh time and status (stale or not) for each aggregation data source.

GET /admin/v1/slo-report

Generates a Service Level Objective (SLO) report for services, comparing their actual performance against availability and latency targets.

Request Parameters

Name	Type	Required	Default	Description
`window`	String	No	`1h`	Time window for the report. Valid values: `15m`, `1h`, `24h`, `7d`, `30d`, `60d`.
`start`	String	No	—	ISO 8601 timestamp for the start of the time range.
`end`	String	No	—	ISO 8601 timestamp for the end of the time range.
`environments`	List<String>	No	—	Filter by a list of environments.
`serviceKeys`	List<String>	No	—	Filter by a list of service keys.
`availabilityTarget`	Double	No	`99.5`	The availability target percentage.
`p95LatencyTargetMs`	Double	No	`500`	The p95 latency target in milliseconds.
`source`	String	No	—	Filter by traffic source: `f5`, `runtime`, `generic`.
`direction`	String	No	—	Filter by service direction: `publish`, `consume`.

Response Body

Returns a JSON array where each object represents a service (or operation) and its SLO status.

breach: A boolean indicating if the service breached either the availability or latency target.
availabilityBreach: A boolean indicating if the availability target was breached.
p95LatencyBreach: A boolean indicating if the latency target was breached.
timeline: A detailed array of performance metrics for each time bucket within the window.

GET /admin/v1/service-metric-timeline

Retrieves a time-series dataset for a single service, including request counts, error counts, and average latency.

Request Parameters

Name	Type	Required	Default	Description
`window`	String	No	`24h`	Time window for the query.
`start`	String	No	—	ISO 8601 timestamp for the start of the time range.
`end`	String	No	—	ISO 8601 timestamp for the end of the time range.
`environment`	String	No	—	The environment of the service.
`serviceKey`	String	Yes	—	The key of the service to retrieve metrics for.

Response Body

Returns a JSON object containing summary metrics for the window and an array of points, where each point is a time bucket with its own metrics.

Traffic Logs

GET /admin/v1/traffic-logs

Provides a unified, paginated view of traffic logs from all supported sources (e.g., F5, generic connectors). This is the preferred endpoint for general log browsing.

Request Parameters

Name	Type	Required	Default	Description
`source`	String	No	`all`	Filter by source system. `f5`, `generic`, or `all`.
`vendor`	List<String>	No	—	Filter by a list of specific gateway vendors (e.g., `f5`, `citrix_netscaler`).
`instanceId`	UUID	No	—	Filter by a specific connector instance ID.
`resultCode`	List<String>	No	—	Filter by a list of F5 result codes. Suppresses generic logs.
`serviceUri`	String	No	—	Substring match on the service URI.
`customer`	String	No	—	Substring match on the consumer IP address.
`frontVirtualServer`	String	No	—	Substring match on the frontend virtual server name.
`backVirtualServer`	String	No	—	Substring match on the backend virtual server name.
`window`	String	No	`15m`	Time window for the query.
`start`	String	No	—	ISO 8601 timestamp for the start of the time range.
`end`	String	No	—	ISO 8601 timestamp for the end of the time range.
`page`	Integer	No	`1`	The page number to retrieve.
`pageSize`	Integer	No	`50`	The number of items per page (max 200).

Response Body

Returns a paginated ObservabilityPageRecord object containing a list of log entries in the items field. Each item includes a source and gatewayVendor field to identify its origin.

GET /admin/v1/traffic-logs/{id}

Retrieves the detailed record for a single traffic log entry.

Request Parameters

Name	Type	Required	Default	Description
`id`	String	Yes	—	The ID of the log entry. This is a UUID for F5 logs and a numeric `bigint` for generic logs.
`source`	String	Yes	`f5`	The source of the log entry (`f5` or `generic`). Must match the type of ID provided.
`includeCloud`	Boolean	No	`true`	If `true`, and the payload is archived, it will be fetched from cloud storage.

Response Body

Returns a JSON object with the full details of the log entry, including headers and payloads. For F5 logs, this endpoint merges data from up to four correlated raw events into a single logical transaction view.

GET /admin/v1/logs/search

Performs a free-text and faceted search across F5 and Runtime traffic logs.

Request Parameters

Name	Type	Required	Default	Description
`since`	String	No	1 hour ago	ISO 8601 timestamp for the start of the search window.
`until`	String	No	now	ISO 8601 timestamp for the end of the search window.
`uriContains`	String	No	—	Substring match on URI, service key, or operation key.
`backendContains`	String	No	—	Substring match on backend host or host:port.
`consumerIpContains`	String	No	—	Substring match on consumer IP. Applies to F5 logs only.
`payloadContains`	String	No	—	Substring match on request or response payload. Applies to F5 logs only.
`statusClass`	String	No	`any`	Filter by HTTP status: `success` (2xx), `client_error` (4xx), `server_error` (5xx), or a specific code (e.g., `404`).
`source`	String	No	`both`	Filter by source: `f5`, `runtime`, `both`, or `all`.
`monitorFilter`	String	No	`any`	Filter runtime logs by OpenAPI monitor findings: `has`, `none`, or `any`.
`direction`	String	No	—	Filter runtime logs by direction: `publish` or `consume`.
`limit`	Integer	No	`200`	Maximum number of results to return (max 1000).

Response Body

Returns a JSON object containing the search results in the items field, sorted by time descending.

GET /admin/v1/f5-traffic-logs

(Legacy) Retrieves a paginated list of F5 traffic logs.

NOTE

This endpoint is maintained for backward compatibility. For new integrations, use the unified /admin/v1/traffic-logs endpoint.

Request Parameters

This endpoint accepts numerous F5-specific filters such as instanceId, serviceUri, customer, frontVirtualServer, backVirtualServer, f5Stage, and pagination parameters.

Response Body

Returns a paginated ObservabilityPageRecord where each item is a logical F5 transaction, created by merging up to four raw syslog events that share a global_transaction_id.

GET /admin/v1/f5-traffic-logs/{id}

(Legacy) Retrieves the detailed record for a single F5 traffic log entry, identified by its UUID. It transparently hydrates payload fields from cloud storage if they have been archived.

GET /admin/v1/runtime-logs

Retrieves a paginated list of logs generated by the platform's runtime service.

Request Parameters

Accepts filters for environment, serviceKey, operationKey, resultCode, status, correlationId, time window, and pagination.

Response Body

Returns a paginated ObservabilityPageRecord containing runtime log entries.

Asynchronous Executions

This set of endpoints allows for viewing and managing asynchronous transactions.

GET /admin/v1/async-executions

Retrieves a paginated list of asynchronous executions.

Request Parameters

Accepts filters for environment, serviceKey, operationKey, time window, and pagination.

Response Body

Returns a paginated ObservabilityPageRecord. The status of an execution can be waiting_callback, completed, failed, or expired.

POST /admin/v1/async-executions/{id}/reprocess

Triggers a reprocessing of a stored callback for a given asynchronous execution ID.

POST /admin/v1/async-executions/{id}/callback-reprocess

Triggers a reprocessing of an asynchronous execution using a provided callback payload.

Request Body

The raw SOAP/XML payload of the callback. Content-Type should be text/plain or application/xml.

POST /admin/v1/async-executions/{id}/manual-complete

Manually marks an asynchronous execution as 'completed'.

Request Body

A JSON string representing the final result.

POST /admin/v1/async-executions/{id}/manual-fail

Manually marks an asynchronous execution as 'failed'.

Request Body

A string describing the reason for the failure.

POST /admin/v1/async-executions/{id}/reopen

Re-opens a completed or failed asynchronous execution, setting its status back to waiting_callback.

Other Async Endpoints

GET /admin/v1/async-executions/{id}/replay-preview: Previews the result of replaying an execution.
POST /admin/v1/async-executions/{id}/callback-preview: Previews the result of processing a provided callback payload.

Audit and Debugging

GET /admin/v1/audit-events

Retrieves a paginated list of audit events, showing who changed what configuration and when.

Request Parameters

Accepts filters for actorEmail, actorRole, actionType, entityType, entityId, result, correlationId, messageContains, time window, and pagination.

Response Body

Returns a paginated ObservabilityPageRecord of audit events. The beforeState and afterState fields show the configuration change, with any sensitive values redacted.

GET /admin/v1/debug-traces/{correlationId}

Retrieves detailed debug trace information for a given correlationId, including full request and response payloads.

POST /admin/v1/soap-to-json/preview

Previews the platform's SOAP-to-JSON transformation for a given XML payload.

Request Parameters

Name	Type	Required	Default	Description
`operationId`	UUID	No	—	The ID of the operation context to use for contract-driven mapping.
`serviceId`	UUID	No	—	The service ID, used with `operationKey` as a fallback to find the operation.
`operationKey`	String	No	—	The operation key, used with `serviceId`.
`direction`	String	No	`response`	The direction of the mapping: `request` or `response`.

Request Body

The raw SOAP/XML payload to be transformed. Content-Type can be text/xml, application/xml, or text/plain.

Response Body

Returns a JSON object with the result.

mode: Indicates the transformation mode used: contract (if an operation mapping was found and used) or naive (if a generic fallback walker was used).
json: The resulting JSON payload.
fault: A boolean indicating if a SOAP fault was detected.

GET /admin/v1/observability/bad-url-attempts

Retrieves a summary of requests made to invalid or unmapped URLs, which result in an S2R-RUN-0405 error.

Request Parameters

Name	Type	Required	Default	Description
`windowHours`	Integer	No	`24`	The time window in hours to query. Bounded server-side to `[1, 720]`.

Response Body

Returns a map grouping bad URL attempts by the attempted path.

Configuration

The behavior of the Observability API can be modified with the following system setting.

Key	Type	Default	Description
`dashboard_freshness_stale_minutes`	Long	`60`	The threshold in minutes after which a traffic aggregation table is considered stale. If a source's last refresh is older than this, a warning is raised.

Troubleshooting

Stale Dashboard Data

If the dashboard appears to show frozen or out-of-date metrics, check the dataFreshness and urgentIssues fields in the response from GET /admin/v1/dashboard/summary. A true value for anyStale or an urgent issue related to "Aggregation refresh stalled" indicates a problem with the background aggregation jobs.

Untrustworthy F5 Metrics

If the F5 syslog ingest pipeline from the relay VM is stalled, the dashboard may incorrectly show "0" traffic and "0" errors from that source. The kpis.f5IngestStale flag in the GET /admin/v1/dashboard/summary response will be true in this state. An urgent issue will also be raised.

Asynchronous Operation Errors

When interacting with the /admin/v1/async-executions endpoints, you may encounter the following errors:

HTTP 400 (Bad Request) with code S2R-OBS-0400: The request is invalid. This typically means the provided execution ID does not exist.
HTTP 422 (Unprocessable Entity) with code S2R-OBS-0422: The requested action cannot be performed on the execution in its current state. For example, attempting to reprocess an execution that is not in a replayable state.

Tagsreference

All SOAP-to-REST docs

Loading…