Observability Architecture Overview

Overview

The platform's observability capabilities are a built-in feature, not a separate product. All telemetry data, including conversion calls, configuration changes, and backend tests, is generated automatically and is queryable through the same user interface and Admin API used for managing service mappings.

This system provides a unified view of platform health and traffic, composed of four primary surfaces:

• Dashboards & KPIs: An operational dashboard with key performance indicator (KPI) tiles, timeline charts, and per-service data rollups.
• Metrics & SLO Reports: Aggregated metrics over various time windows and Service Level Objective (SLO) reports for tracking availability and latency targets.
• Logs & Tracing: Structured runtime logs, a live traffic log with full request/response payloads, and distributed tracing capabilities.
• Audit Trail: A chronological record of all configuration changes, detailing who made the change, when it occurred, and the state before and after.

All observability data is accessible through the Observability section of the UI and programmatically via the Admin API under the /admin/v1/* path.

How it Works

The observability architecture is designed around a single, fundamental rule: all time-bucketed metric queries read from pre-computed aggregation tables, never from raw logs at query time. This design ensures consistent, high-performance reporting, even with large volumes of historical data.

This principle applies to all metric surfaces, including dashboards, KPI tiles, timeline charts, and SLO reports, regardless of the traffic source.

flowchart TD
    subgraph "Data Ingest"
        A[Traffic Sources <br/>(Runtime, Connectors)] --> B{Raw Telemetry Logs};
    end

    subgraph "Offline Processing"
        B -- "Processed by" --> C[Background Curator Workers];
        C -- "Generate" --> D[Pre-computed Aggregation Tables];
    end

    subgraph "Data Query"
        D -- "Fast reads for" --> E[Observability UI & API <br/>(Dashboards, Metrics, SLOs)];
        B -- "Slow, targeted reads for" --> F[Log Drill-down <br/>(by Correlation ID)];
    end

This aggregation-first approach provides two main benefits:

1. Performance: Dashboards and reports load in sub-second time because they query small, pre-aggregated tables instead of scanning millions of raw log entries. For example, a reference estate with approximately 175 services can render 24-hour, 7-day, and 1-month metrics almost instantly.
2. Consistency: Because all high-level metric surfaces (dashboards, SLO reports, timelines) query the same aggregated data, the numbers presented are always consistent across the platform.

Role of Raw Logs

Raw logs are reserved for three specific, narrow functions:

1. Ingest Health Monitoring: To provide signals about the health of the data pipeline, such as detecting if data from a specific source has stopped arriving.
2. Specific Event Drill-down: When a user selects a single event in the UI to inspect its details, the platform queries the raw log table for the single row matching that event's correlation ID.
3. Offline Aggregation: Background workers read raw logs to compute and populate the downstream aggregation tables.

When you query a metric and it loads instantly, you are interacting with the pre-computed aggregation read path.

Traffic Sources

All telemetry data includes a source dimension, which identifies the origin of the traffic. This allows you to filter metrics by a specific source or view them in aggregate.

By default, metric views sum data across all sources. You can use the source dimension to narrow the scope of a query.

Correlation IDs

A single, unique correlation ID is generated for each request and travels with it through its entire lifecycle. This ID is present in the initial inbound call, the subsequent backend call, the mapping logic, structured logs, debug traces, and any asynchronous callbacks.

The correlation ID acts as the primary join key across all observability surfaces. If an API consumer reports an error, you can search by the correlation ID to find the exact transaction, view its full request and response payloads, and analyze its distributed trace.

Data Retention

By default, all telemetry and audit data is retained for 60 days. This retention window is configurable to align with your organization's data policies.

For a detailed breakdown of retention periods for specific data tables, see Data Retention. The source does not specify the configuration key for changing the retention period.

Related Pages

• Dashboards & KPIs
• Metrics & SLO reports
• Logs & tracing
• Audit trail
• Admin API — observability endpoints

All SOAP-to-REST docs