License Heartbeat Telemetry

Overview

The Specaria Heartbeat Client is an internal component responsible for sending periodic liveness and telemetry data to the Specaria platform. This process, known as a "heartbeat," serves to validate the license status and provide anonymized, high-level usage metrics and health information.

The client runs as part of the admin-api service. It is triggered automatically on a daily schedule and once upon application startup. On each trigger, it gathers data from the platform's database and runtime environment, constructs a JSON payload, and sends it via an HTTP POST request to the Specaria API endpoint associated with the active license.

The system is designed to be fail-open, meaning that any failure during the heartbeat process (e.g., network errors, database issues) will not crash the product or interrupt its operation.

How It Works

The heartbeat process is initiated by one of two triggers: a daily scheduled task or a one-time event on application startup. The client then follows a sequence of steps to build and send the payload, with several gates and decision points.

sequenceDiagram
    participant Scheduler as Scheduler/Startup
    participant HeartbeatClient as SpecariaHeartbeatClient
    participant DB as Database (s2r_system_setting)
    participant LicenseValidator as License Validator
    participant SpecariaAPI as Specaria API

    Scheduler->>HeartbeatClient: Trigger (daily cron / on boot)
    HeartbeatClient->>LicenseValidator: Get current license state
    LicenseValidator-->>HeartbeatClient: LicenseState (or null)
    alt License absent/disabled
        HeartbeatClient-->>Scheduler: Stop
    end
    HeartbeatClient->>DB: Read 'specaria.heartbeat_sender'
    DB-->>HeartbeatClient: Sender owner
    alt Another component is sender
         HeartbeatClient->>DB: Write component versions
         DB-->>HeartbeatClient: OK
         HeartbeatClient-->>Scheduler: Stop (stand down)
    end
    HeartbeatClient->>DB: Write component versions
    DB-->>HeartbeatClient: OK
    HeartbeatClient->>HeartbeatClient: buildPayload()
    Note right of HeartbeatClient: Gathers metrics from DB, <br/>health probes, env vars.
    HeartbeatClient->>HeartbeatClient: redact(payload)
    alt Redaction fails
        HeartbeatClient->>DB: Record status 'redactor-rejected'
        HeartbeatClient-->>Scheduler: Stop
    end
    HeartbeatClient->>LicenseValidator: Get raw JWT
    LicenseValidator-->>HeartbeatClient: JWT
    HeartbeatClient->>SpecariaAPI: POST /v1/licenses/{jti}/heartbeat
    SpecariaAPI-->>HeartbeatClient: HTTP Response (e.g., 2xx, 4xx, 5xx)
    alt Transient Failure (e.g., 5xx)
        HeartbeatClient->>HeartbeatClient: Wait 60s
        HeartbeatClient->>SpecariaAPI: POST /v1/licenses/{jti}/heartbeat (Retry)
        SpecariaAPI-->>HeartbeatClient: HTTP Response
    end
    HeartbeatClient->>DB: Record final status and timestamp
    DB-->>HeartbeatClient: OK

Triggering and Jitter

• Scheduled: A daily heartbeat is triggered by a cron job, configured by s2r.specaria.heartbeat-cron (default 0 5 0 * * * UTC). To prevent multiple installations from contacting the API simultaneously, a random jitter of up to ±2 hours is applied before the job executes.
• Cold Start: A heartbeat is also sent once when the application starts. This is throttled to prevent spamming the API during frequent restarts (e.g., autoscaling events); it will not fire if a successful heartbeat was sent within the last 20 hours.

Sender Election

To ensure an installation sends exactly one heartbeat, a leader election mechanism is used.

1. The client checks for a setting with the key specaria.heartbeat_sender in the s2r_system_setting table.
2. If the value is set to another component's name (e.g., s2r-release-controller), this client will stand down and not send a heartbeat. It will still publish its known component versions to the database for the active sender to use.
3. If the key is not present, or if its value is admin-api, this client proceeds with sending the heartbeat.
4. The client will also stand down if the active license is the embedded bootstrap license, as its identifier is not known to the Specaria platform and any heartbeats would be discarded.

Payload Generation and Redaction

1. Build: The client constructs a payload containing liveness status, usage metrics, and environment information. Data is sourced from the database, environment variables, and the active license state.
2. Redact: Before serialization, the entire payload is processed by the SpecariaRedactor. This is a privacy backstop that ensures no disallowed data patterns (e.g., hostnames, URLs) are ever sent. If the redactor rejects the payload, the heartbeat is dropped and an error is logged.
3. Serialize: The redacted payload is serialized into a JSON string.

HTTP POST and Retry Logic

1. Request: The client sends the JSON payload in an HTTP POST request to the URL specified by s2r.specaria.heartbeat-url-template. The request is authenticated using the raw license JWT as a Bearer token in the Authorization header.
2. Retry: If the request fails with a transient error (e.g., a 5xx server error or a network timeout), the client waits for 60 seconds and attempts the request one more time.
3. Permanent Failures: Certain 4xx status codes (e.g., 410 Gone, 429 Too Many Requests) are treated as permanent failures for the current cycle, and no retry is attempted.
4. Status Logging: The final outcome of the cycle (e.g., accepted-202, dropped-404, redactor-rejected) is recorded in the s2r_system_setting table with the key specaria.last_heartbeat_status. If successful, a timestamp is also recorded under the key specaria.last_heartbeat_at.

Telemetry Tiers and Consent

The heartbeat payload is structured into tiers, allowing operators to control the level of detail shared.

• Liveness Heartbeat: The core heartbeat is always sent as long as a valid, non-disabled license is present. It provides a basic liveness signal to the platform.
• Basic Telemetry: Every heartbeat includes a block of basic telemetry, such as product version, uptime, and high-level health flags. This tier is not optional.
• Optional Telemetry: Richer data blocks, inventory and diagnostics, are considered optional. They are included only if the installation-wide setting specaria.telemetry_optional_enabled is true (the default). Operators can disable this tier via an administrative API endpoint.

Heartbeat Payload Structure

The client sends an HTTP POST request with a JSON body.

• Method: POST
• Endpoint: The URL is derived from the s2r.specaria.heartbeat-url-template configuration property, with {jti} replaced by the license's JWT ID.
  • Default: https://api.specaria.io/v1/licenses/{jti}/heartbeat
• Headers:
  • Authorization: Bearer <raw-license-jwt>
  • Content-Type: application/json
  • User-Agent: s2r-licensing/1
  • Accept: application/json

Payload Body

The JSON payload has the following structure. All fields are subject to redaction to remove sensitive data patterns before being sent.

{
  "product_version": "<string>",
  "agent_version": "<string>",
  "success": "<boolean>",
  "metrics": {
    "max_discovered_services": "<integer>",
    "max_converted_services": "<integer>"
  },
  "environment": "production",
  "host_fingerprint": "sha256-<string>",
  "telemetry": {
    "schema_version": 1,
    "product_id": "<string>",
    "release_version": "<string>",
    "service_versions": {
      "admin-api": "<string>",
      "admin-ui": "<string>",
      "worker": "<string>",
      "f5-ingest": "<string>",
      "runtime": "<string>",
      "s2r-release-controller": "<string>"
    },
    "uptime_seconds": "<long>",
    "deployment_type": "<string>",
    "cloud_provider": "<string>",
    "orchestrator": "<string>",
    "instance_id": "<uuid>",
    "error_count_5m": "<long>",
    "request_count_5m": "<long>",
    "controller_status": "<string>",
    "last_update_outcome": {
      "release_version": "<string>",
      "status": "<string>",
      "at": "<iso-8601-timestamp>"
    },
    "inventory": {
      "service_inventory": [
        {
          "name": "<string>",
          "image_tag": "<string>",
          "replicas": "<integer>"
        }
      ],
      "capabilities": ["<string>"],
      "config_keys_set": ["<string>"],
      "integrations": ["<string>"],
      "schema_versions": {
        "admin-api": "<string>"
      },
      "resource_profile": {
        "cpu_class": "<string>",
        "mem_class": "<string>",
        "storage_class": "<string>"
      }
    },
    "diagnostics": {
      "...": "..."
    }
  }
}

Field Descriptions

Top-Level Fields

telemetry Object

inventory Object (Optional)

The inventory block is only included if optional telemetry is enabled. Each sub-field is additionally gated by a specific consent key in the license. If consent is not granted for a category, its corresponding field is omitted.

Configuration

The heartbeat client's behavior can be configured via application properties and environment variables.

Administrative Operations

An operator can control the inclusion of optional telemetry blocks (inventory and diagnostics).

The setOptionalTelemetryEnabled(boolean enabled) method updates the specaria.telemetry_optional_enabled setting in the s2r_system_setting table. This method is exposed via an administrative API endpoint (POST /admin/v1/specaria/telemetry/optional).

• Setting this to true enables the optional telemetry tier.
• Setting this to false disables it, and only the liveness signal and basic telemetry will be sent.

Troubleshooting

The outcome of each heartbeat cycle is recorded in the s2r_system_setting table under the key specaria.last_heartbeat_status. This provides visibility into the client's operation.

All SOAP-to-REST docs