Diagnostic Telemetry Collection

Overview

The DiagnosticTelemetryCollector is a system component that builds a block of "Tier-0" diagnostic data. This data is designed to help support personnel understand and triage potential issues within a deployment using only numeric, boolean, or fixed-set enumeration values.

The primary goal is to provide high-level health indicators without requiring access to sensitive information or a full debug bundle. The collected data is embedded within the platform's periodic telemetry heartbeat and can also be included in tenant-specific debug bundles.

The collector operates under several key principles:

• Privacy by Design: All collected values are strictly numeric, boolean, or belong to a closed set of enumerated values (like vendor identifiers). It explicitly excludes any hostnames, IP addresses, user names, free-text, or other potentially sensitive information. The entire output is designed to pass the platform's redaction and validation checks.
• Performance: Data is gathered using efficient queries that target pre-aggregated metrics, state tables, or perform bounded counts. The collector avoids expensive scans of raw, high-volume logs.
• Resilience: The collection process is fail-open for each data group. If a query for a specific group of metrics fails (e.g., due to a timeout or a schema mismatch), that group is omitted from the output, but the overall collection process succeeds. This ensures that telemetry heartbeats are not blocked by failures in diagnostic data collection.

How it Works

The DiagnosticTelemetryCollector provides two main functions: one for collecting deployment-wide data for the platform heartbeat (collect()) and another for collecting tenant-scoped data for debug bundles (collectForBoundTenant()).

flowchart TD
    subgraph "Callers"
        A[Telemetry Heartbeat Emitter]
        B[Debug Bundle Generator]
    end

    subgraph "DiagnosticTelemetryCollector"
        C[collect()]
        D[collectForBoundTenant()]
    end

    subgraph "Data Collection Logic"
        E{Has Deployment-Wide Access?}
        F[Collect Deployment-Wide Data]
        G[Collect Tenant-Scoped Data]
    end

    subgraph "Metric Groups (Collected in Parallel)"
        style G1 fill:#f9f,stroke:#333,stroke-width:2px
        style G2 fill:#f9f,stroke:#333,stroke-width:2px
        style G3 fill:#f9f,stroke:#333,stroke-width:2px
        style G4 fill:#f9f,stroke:#333,stroke-width:2px
        style G5 fill:#f9f,stroke:#333,stroke-width:2px
        style G6 fill:#f9f,stroke:#333,stroke-width:2px
        style G7 fill:#f9f,stroke:#333,stroke-width:2px

        G1[connector_inventory]
        G2[ingest_health]
        G3[discovery_health]
        G4[topology_shape]
        G5[traffic_health]
        G6[async_health]
        G7[relay_health]
    end

    subgraph "Data Sources"
        DB[(Database: Aggregation & State Tables)]
    end

    subgraph "Output"
        H[Diagnostic Data Map]
    end

    A --> C
    B --> D

    C --> F
    D --> E
    E -- Yes --> F
    E -- No --> G

    F --> G1 & G2 & G3 & G4 & G5 & G6 & G7
    G --> G1 & G3 & G5 & G6 & G7

    G1 & G2 & G3 & G4 & G5 & G6 & G7 --> DB
    DB -- Numeric/Enum Data --> G1 & G2 & G3 & G4 & G5 & G6 & G7

    G1 & G2 & G3 & G4 & G5 & G6 & G7 -- Fail-Open --> H

Collection Process

When invoked, the collector executes a series of queries, each corresponding to a specific diagnostic "group". Each group's collection is wrapped in a try/catch block. If a query fails, the error is logged, and that specific group is omitted from the final result, ensuring the collector never throws an exception to its caller.

There are two collection contexts:

1. Deployment-Wide (collect): This is the default mode, used for the platform's telemetry heartbeat. It gathers metrics from across the entire deployment, querying tables like s2r_connector_instance, s2r_generic_ingest_dlq, and s2r_proxy_topology_chain without tenant-specific filters.

2. Tenant-Scoped (collectForBoundTenant): This mode is used when generating a debug bundle for a specific tenant operator.

   • If the operator has platform-level permissions, it behaves identically to the deployment-wide collection.
   • Otherwise, it gathers metrics strictly related to the operator's tenant. The SQL queries are modified to filter by the bound tenant ID.
   • Groups whose underlying data tables do not have a tenant column (ingest_health and topology_shape) are omitted entirely in this mode.

The final output is a map of key-value pairs, where each key is a group name and the value is another map containing the metrics for that group.

Diagnostic Data Structure

The collected data is organized into several groups, each representing a different aspect of the system's health. The entire block is typically embedded under a diagnostics key in the final telemetry payload.

connector_inventory

Provides an inventory of configured connectors, grouped by vendor type.

This group is sourced from the s2r_connector_instance table. It will be omitted if no connectors are configured. The number of vendors returned is limited to 32.

ingest_health

Reports on the health of the generic data ingestion pipeline, focusing on the dead-letter queue (DLQ).

This group is always present in deployment-wide collection, returning 0 for all values if the DLQ is empty.

discovery_health

Contains counts related to service discovery and traffic example learning.

topology_shape

Describes the configured proxy topology of the system.

traffic_health

Summarizes recent traffic health based on hourly aggregation tables. The metrics are calculated over the last full hour bucket.

This group is sourced from s2r_agg_service_hourly and s2r_runtime_agg_service_hourly. If there was no traffic in the last hour, this group is omitted.

async_health

Reports on the backlog depth for asynchronous and replay-related queues.

This group is always present, returning 0 if the queues are empty.

relay_health

Provides health and freshness indicators for the F5 relay, data ingestion, and WSDL probing functions.

This group is sourced from s2r_f5_sync_state and s2r_traffic_log. If no F5 relay is enabled on the deployment, this entire group is omitted.

Example Payload

The following shows an example of the diagnostic block structure. The values are for illustration only.

{
  "telemetry": {
    "schema_version": 1,
    "...": "...",
    "diagnostics": {
      "connector_inventory": {
        "f5_bigip": {
          "configured": 5,
          "enabled": 4
        },
        "generic_rest": {
          "configured": 10,
          "enabled": 10
        }
      },
      "ingest_health": {
        "dlq_depth": 0,
        "dlq_24h": 0,
        "ingest_lag_seconds": 0
      },
      "discovery_health": {
        "discovered_services": 1500,
        "learned_examples": 897,
        "services_zero_examples": 23
      },
      "topology_shape": {
        "proxy_chain_count": 4,
        "max_chain_depth": 3,
        "distinct_gateway_types": [
          "api_gateway_a",
          "api_gateway_b"
        ]
      },
      "traffic_health": {
        "mean_latency_ms": 120,
        "max_latency_ms": 850,
        "error_rate_5m_pct": 1
      },
      "async_health": {
        "async_queue_depth": 5,
        "replay_pending": 0
      },
      "relay_health": {
        "relay_enabled": true,
        "relay_heartbeat_age_seconds": 45,
        "ingest_age_seconds": 15,
        "wsdl_probe_errors": 1
      }
    }
  }
}

API Reference

The component is exposed as a Spring @Component and can be injected where needed.

collect()

Collects a deployment-wide set of diagnostic metrics. This method is intended for use by the platform's telemetry heartbeat emitter.

• Returns: Map<String, Object> containing the diagnostic data. Groups with collection errors are omitted.
• Throws: Never throws an exception.

collectForBoundTenant()

Collects a set of diagnostic metrics scoped to the currently authenticated tenant. This is used for generating tenant-specific debug bundles.

• Behavior:
  • If the caller has deployment-wide access, this method returns the same full, deployment-wide data as collect().
  • For a standard tenant, it returns data filtered to that tenant. The ingest_health and topology_shape groups are omitted, as their data sources are not tenant-scoped.
• Returns: Map<String, Object> containing the tenant-scoped diagnostic data.
• Throws: Never throws an exception.

Troubleshooting

Omitted Data Groups

The most common operational behavior to note is that a diagnostic group may be missing from the output. This is intentional fail-open behavior.

• Cause: A group is omitted if the underlying database query fails for any reason, such as a database timeout, a missing table after a partial upgrade, or an unexpected schema change.
• Diagnosis: When a group's collection fails, a WARN level log entry is written. Administrators can search for logs matching the pattern:

  S2R-CDD telemetry-v2 group '<group_name>' skipped (fail-open): <error_message>
  

  For example: S2R-CDD telemetry-v2 group 'traffic_health' skipped (fail-open): query timed out

Additionally, some groups are omitted by design under certain conditions:

• connector_inventory: Omitted if no connectors are configured.
• traffic_health: Omitted if there was no traffic in the last hour.
• relay_health: Omitted if the F5 relay feature is not enabled.
• ingest_health and topology_shape: Omitted from tenant-scoped collection (collectForBoundTenant).

All SOAP-to-REST docs