Heartbeat Client

Overview

The SpecariaHeartbeatClient is a system component responsible for sending periodic liveness and telemetry data to the Specaria platform API. This process, known as a "heartbeat," serves as a proof-of-liveness for an active installation and provides usage metrics that are compared against license entitlements.

The client is triggered on a daily schedule and also once upon application startup. It gathers a variety of metrics, including product version, component health, service counts, and optional diagnostic information. This data is assembled into a JSON payload, redacted for privacy, and sent via HTTP POST to the central Specaria licensing endpoint.

The client operates with fail-open semantics, meaning that any failure during the heartbeat process (e.g., network errors, database issues) will not crash the application. The heartbeat is simply dropped for that cycle and retried on the next scheduled run.

The client's primary inputs are the active license state, configuration properties, and data queried from the local database. Its main output is an HTTP request to the Specaria platform. It also records the status of each heartbeat attempt in the s2r_system_setting table for administrative visibility.

How it Works

The heartbeat process follows a defined sequence to ensure data is collected, redacted, and transmitted reliably and securely.

flowchart TD
    subgraph "Heartbeat Client Logic"
        A[Daily Schedule or<br>Application Startup] --> B{Validate License};
        B -- Invalid/Disabled --> Z1[End Cycle];
        B -- Valid --> C[Publish Component Versions to DB];
        C --> D{Check Sender Ownership};
        D -- Not Owner --> Z2[End Cycle];
        D -- Is Owner --> E[Assemble Heartbeat Payload];

        subgraph "Payload Assembly"
            direction LR
            E1[Basic Telemetry<br>(Version, Uptime, Health)]
            E2["Optional Telemetry<br>(Inventory, Diagnostics)"]
            E --> E1 & E2;
        end

        E1 & E2 --> F[Redact Payload];
        F --> G[POST to Specaria Platform];
        G --> H{Handle Response};
        H -- Success (2xx) --> I[Record Success in DB];
        H -- Permanent Error (4xx) --> J[Record Failure in DB];
        H -- Transient Error (5xx) --> K[Wait 60s & Retry POST];
        K -- Success --> I;
        K -- Failure --> J;
    end

1. Trigger: The process is initiated either by a daily cron schedule (configurable via s2r.specaria.heartbeat-cron) or once when the application starts (ApplicationStartedEvent).

   • The scheduled trigger includes a random jitter of up to ±2 hours to distribute load from multiple installations.
   • The startup trigger is throttled and will be skipped if a successful heartbeat was sent within the last 20 hours.

2. License Validation: The client retrieves the current license state. If the license is absent or disabled, the cycle is aborted as there is no valid identity (jti) to authenticate the request.

3. Component Version Publishing: The client unconditionally gathers version information for all product components and writes it to the s2r_system_setting table. This ensures that whichever component is designated to send the heartbeat has access to accurate, up-to-date version data for the entire installation.

4. Sender Ownership: The client checks the specaria.heartbeat_sender setting in the database. To prevent duplicate reporting, only one component in an installation is designated as the heartbeat sender. If another component holds this role, the client stands down for the current cycle. It also stands down if the installation is running on the embedded "bootstrap" license, as its temporary identifier is not recognized by the platform.

5. Payload Assembly: If designated to send, the client constructs the JSON payload. The payload is separated into two tiers:

   • Basic Telemetry: Always included. Contains essential data like product version, uptime, overall health status, and high-level counts.
   • Optional Telemetry: Included only if the specaria.telemetry_optional_enabled setting is true. This tier contains richer, consent-gated data blocks, including inventory (enabled features, configuration keys) and diagnostics.

6. Redaction: The assembled payload undergoes two redaction steps:

   • First, specific metric keys listed in the license's meta.heartbeat_redacted_fields are removed.
   • Second, the entire payload is processed by the SpecariaRedactor to remove any values that match disallowed patterns (e.g., hostnames, URLs). If the redactor rejects the payload, the heartbeat is dropped and an error is logged.

7. Transmission: The final, redacted JSON payload is sent via HTTP POST to the endpoint specified in s2r.specaria.heartbeat-url-template. The request is authenticated using the raw license JWT as a Bearer token. The HTTP client automatically honors HTTPS_PROXY, NO_PROXY, and SSL_CERT_FILE environment variables.

8. Response Handling and Retry:

   • A 2xx status code indicates success.
   • A transient failure (e.g., 5xx status, timeout) triggers a single retry after a 60-second delay.
   • A permanent failure (e.g., 401, 404, 410, 429) is not retried within the same cycle.

9. Status Recording: The final outcome of the cycle (e.g., accepted-202, dropped-503, redactor-rejected) is written to the specaria.last_heartbeat_status key in the database. On success, specaria.last_heartbeat_at is updated with the current timestamp.

Heartbeat Payload

The client sends a JSON object with a structure compliant with the Specaria platform integration requirements. The payload includes top-level metadata and a nested telemetry object.

Below is an example of a heartbeat payload. Values are for illustration only.

{
  "product_version": "1.7.22",
  "agent_version": "1.7.22",
  "success": true,
  "metrics": {
    "max_discovered_services": 150,
    "max_converted_services": 75
  },
  "environment": "production",
  "host_fingerprint": "sha256-a1b2c3d4...",
  "telemetry": {
    "schema_version": 1,
    "product_id": "s2r",
    "release_version": "1.7.21",
    "service_versions": {
      "admin-api": "1.7.22",
      "discovery-ingest": "1.5.1",
      "s2r-release-controller": "1.6.21"
    },
    "uptime_seconds": 3600,
    "deployment_type": "CUSTOMER_VPC",
    "cloud_provider": "gcp",
    "orchestrator": "cloud-run",
    "instance_id": "f81d4fae-7dec-11d0-a765-00a0c91e6bf6",
    "error_count_5m": 5,
    "request_count_5m": 1000,
    "controller_status": "ok",
    "last_update_outcome": {
      "release_version": "1.7.21",
      "status": "success",
      "at": "2023-10-27T10:00:00Z"
    },
    "inventory": {
      "service_inventory": [
        {
          "name": "admin-api",
          "image_tag": "1.7.22",
          "replicas": 0
        }
      ],
      "capabilities": [
        "feature-a",
        "feature-b"
      ],
      "config_keys_set": [
        "CLOUD_PROVIDER",
        "S2R_INSTANCE_ID"
      ],
      "integrations": [
        "f5"
      ],
      "schema_versions": {
        "admin-api": "V175"
      },
      "resource_profile": {
        "cpu_class": "medium",
        "mem_class": "medium",
        "storage_class": "unknown"
      }
    },
    "diagnostics": {
      // Structure and content defined by the DiagnosticTelemetryCollector component.
    }
  }
}

Configuration

The SpecariaHeartbeatClient is configured through Spring properties and environment variables.

Database Settings

The client reads from and writes to the s2r_system_setting table to coordinate state and store results. All keys are stored with a platform-wide scope.

API Reference

Outbound Request

The client makes an outbound call to the Specaria platform.

• Endpoint: POST /v1/licenses/{jti}/heartbeat
• Host: Configured by s2r.specaria.heartbeat-url-template.
• Headers:
  • Authorization: Bearer <license-jwt>
  • Content-Type: application/json
  • Accept: application/json
  • User-Agent: s2r-licensing/1
• Body: The JSON payload described in Heartbeat Payload.

Administrative Endpoint

The client's behavior can be controlled via an administrative API endpoint.

• Endpoint: POST /admin/v1/specaria/telemetry/optional
• Description: Enables or disables the sending of optional telemetry blocks (inventory and diagnostics). This action sets the specaria.telemetry_optional_enabled system setting.
• Request Body: The source does not specify the request body format.

Troubleshooting

The client is designed to be resilient and provide clear logging for its state.

All SOAP-to-REST docs