Discovery Ingest Relay Health Reconciliation

Overview

The Discovery Ingest Health Service is an automated reconciliation component that manages the lifecycle of discovery ingest relay Virtual Machines (VMs). These relay VMs are deployed into an API publisher's environment to receive and forward discovery data, such as syslog traffic, to the platform's ingest endpoint.

This service solves the operational problem of ensuring these critical, customer-hosted components remain available, healthy, and correctly configured without manual intervention. It runs periodically within the platform's worker service to perform the following key functions:

• Health Monitoring: Probes each relay VM to ensure it is running and its internal services are responsive.
• Auto-Recreation: Automatically tears down and recreates unhealthy or unresponsive VMs.
• Configuration Management: Publishes runtime configuration to a central object store, which relays fetch dynamically.
• Image Version Control: Compares the relay's running container image digest against a desired version and flags divergences.
• Liveness Tracking: Monitors a database-backed heartbeat signal from each relay to detect silent failures.
• Automated Deprovisioning: Decommissions relay VMs associated with discovery sources that have been disabled.
• Disk Pressure Management: Monitors disk usage on each relay and can automatically extend the boot disk to prevent ingest failures due to a full disk.

The service is cloud-agnostic by design, interacting with different cloud providers through a RelayVmGateway interface. It reads its target list of relays from the platform database and updates their status in the s2r_f5_sync_state table.

How It Works

The core logic resides in the reconcileActiveRelays method, which executes a reconciliation loop over all active and enabled relay VMs defined in the database. For each relay, the service performs a sequence of checks and actions to converge its state to the desired configuration.

flowchart TD
    subgraph "For Each Active Relay"
        A(Start Reconciliation) --> B{Discovery Source Enabled?};
        B -- No --> C{Auto-Deprovision Enabled?};
        C -- Yes --> D[Deprovision Relay VM];
        C -- No --> E[Log Orphan and Skip];
        B -- Yes / Indeterminate --> F[Check/Create Relay VM];
        F --> G{Relay VM Healthy?};
        G -- No --> H[Dispatch Asynchronous VM Recreation];
        G -- Yes --> I[Probe Relay Health Endpoint];
        I --> J[Check Image Version];
        J --> K[Check Liveness Heartbeat];
        K --> L[Check Disk Usage];
        L --> M{Disk Auto-Extend Triggered?};
        M -- Yes --> N[Resize VM Disk];
        M -- No --> O[Persist Final Status to DB];
        N --> O;
        H --> O;
        D --> End(Done);
        E --> End;
        O --> End;
    end

Reconciliation Steps

1. Load Targets: The service queries the s2r_f5_instance and s2r_f5_sync_state tables to get a list of all relays that are marked as enabled.

2. Discovery Gate: For each relay, it checks the s2r_discovery_vendor_enabled table to see if the corresponding discovery source is enabled.

   • If disabled and the S2R_INGEST_DEPROVISION_ON_DISCOVERY_DISABLED flag is true, the service deprovisions the relay VM and updates its status to deleted.
   • If disabled and the flag is false (the default), it logs a warning (S2R-INGEST-ORPHAN) and takes no further action for that relay.
   • If enabled or the status is indeterminate (e.g., due to a database error), it proceeds with health reconciliation to ensure service continuity.

3. Health Check and Recreation: The service ensures the relay VM is healthy.

   • It verifies the underlying VM exists and is in a RUNNING state via the RelayVmGateway. If not, it creates it.
   • It probes the relay's /healthz endpoint. A healthy relay must return an ok status, be in vm-local-ingest mode, and have its local ingest container sub-process running.
   • If the VM is unhealthy but was created recently (within the relayStartupGrace window), it is marked as starting and left to bootstrap.
   • If the VM is unhealthy and outside the grace period, an asynchronous recreation task is dispatched. This task, which deletes and then creates the VM, runs on a separate thread with a 15-minute timeout to avoid blocking the main reconciliation loop. The relay is marked as recreating.

4. Status Evaluation (for Healthy Relays): If the VM is healthy, the service performs deeper analysis based on the data returned from the /healthz probe.

   • Image Divergence: It compares the running container image digest reported by the relay against the desired digest stored in the s2r_system_setting table. If the digests differ for longer than the relayDivergenceGrace period, the relay's status is updated to stale-image.
   • Liveness Heartbeat: It checks the timestamp of the relay's last database self-report. If this is older than the relayHeartbeatStale threshold, a S2R-RELAY-NO-HEARTBEAT warning is logged.
   • Disk Pressure: It checks the boot disk usage. If usage exceeds relayDiskWarnPercent, a S2R-RELAY-DISK-PRESSURE warning is logged. If it exceeds relayDiskGrowPercent and auto-extend is enabled, a disk resize operation is triggered.

5. State Persistence: The service writes the final determined status (ok, starting, recreating, stale-image), health message, IP address, and self-reported image details back to the s2r_f5_sync_state table.

Relay VM Provisioning

When a relay VM is created or recreated, the service generates and executes a startup script on the instance. This script is responsible for bootstrapping the entire relay environment.

The script performs these critical tasks:

• Installs dependencies like docker.io and python3.
• Installs a self-healing mechanism (s2r-update-relay-installer.service) to ensure key operational scripts are always present.
• Deploys two primary systemd services:
  • s2r-discovery-ingest-local.service: Runs a Docker container of the platform's worker image. This container exposes an HTTP endpoint for the syslog relay script to forward data to.
  • s2r-discovery-ingest-relay.service: Runs a Python-based syslog server that listens for traffic, queues it, and forwards it to the local ingest container.
• Deploys a self-heal reconciliation timer (s2r-discovery-ingest-reconcile.timer). This periodically runs a script that fetches the desired container image from the control plane and restarts the s2r-discovery-ingest-local service if needed, ensuring the relay automatically converges to the correct version.
• Writes necessary configuration files to the VM, including environment files and a bootstrap JSON file for the relay script containing its initial configuration URL.
• Installs and configures a logging agent to forward container logs.

The relay's runtime configuration is published to the platform's central ObjectStore (GCS or S3). The VM is given a signed URL to download this configuration upon startup and periodically refresh it.

Service Method Reference

The service exposes one primary method for initiating the reconciliation process.

reconcileActiveRelays()

Triggers a full reconciliation of all active discovery ingest relay VMs. It iterates through each target, assesses its health and configuration, and takes corrective action as needed.

• Parameters: None.
• Returns: A Map<String, Object> containing a summary of the reconciliation run.

Example Response

The returned map provides a high-level overview of the state of the relay fleet after the run completes.

{
  "status": "DONE",
  "targetCount": 10,
  "healthy": 7,
  "starting": 1,
  "recreated": 0,
  "staleImage": 1,
  "noHeartbeat": 0,
  "orphanLogged": 1,
  "deprovisioned": 0,
  "diskGrown": 0,
  "failed": 1,
  "time": "2023-10-27T10:00:00Z"
}

Response Fields:

Configuration

The service is configured via environment variables.

Troubleshooting

The service emits structured log messages to indicate specific conditions or failures. These can be used for alerting and diagnostics.

| Log Message / Status | Condition | Meaning and Action | --- | --- | | recreating status | A relay VM was found to be unhealthy and an asynchronous recreation task has been dispatched. | The system is attempting to self-heal. The process involves deleting the old VM and creating a new one, which can take several minutes. The recreation has a 15-minute timeout. | stale-image status | A relay is running a container image that does not match the desired version, and the divergence has persisted beyond the grace period. | The relay's self-healing mechanism may have failed. The system will continue to flag this until the relay's container is updated. The relay is otherwise healthy and forwarding traffic. | S2R-INGEST-ORPHAN | A relay VM exists for a discovery source that is currently disabled, and auto-deprovisioning is turned off. | This is an informational warning. The relay VM is left running but is considered "orphaned." To remove it, either re-enable the discovery source or enable auto-deprovisioning via S2R_INGEST_DEPROVISION_ON_DISCOVERY_DISABLED. | S2R-RELAY-DIVERGENCE | A relay is running a container image that does not match the desired version. | This warning indicates a version mismatch that has persisted beyond the configured grace period. The relay's self-heal mechanism should eventually correct this. | S2R-RELAY-NO-HEARTBEAT | A relay has not sent a liveness heartbeat to the database within the configured staleness threshold. | The relay VM may be offline, or its internal container may be stuck or unable to reach the database. This is a high-priority signal that ingest may have stopped. The system will eventually mark the relay as unhealthy and recreate it if it cannot recover. | S2R-RELAY-DISK-PRESSURE | A relay's boot disk usage has exceeded the warning threshold (relayDiskWarnPercent). | This is an early warning that the disk is filling up. If auto-extend is enabled, the system will resize the disk if usage reaches the relayDiskGrowPercent threshold. If not, manual intervention may be required to free up space or resize the disk. | S2R-RELAY-DISK-GROW | The service has initiated an automatic resize of a relay's boot disk. | This is an informational message. The system is proactively managing disk space. The relay's internal scripts will handle extending the filesystem onto the newly allocated space. | S2R-RELAY-DISK-GROW-BLOCKED | An attempt to resize a disk was blocked. | The log message will contain a reason, such as at-max-size, unsupported-on-platform, resize-failed, or filesystem-grow-failed. This indicates that automatic disk management has hit a limit and manual intervention may be required.

All SOAP-to-REST docs