Discovery Ingest Relay Health Service

Overview

The Discovery Ingest Health Service is a worker component responsible for the complete lifecycle management of Discovery Ingest Relay Virtual Machines (VMs). These relay VMs are dedicated instances that receive high-volume syslog data from API gateways and forward it to the platform's central ingest service.

This service automates the provisioning, configuration, health monitoring, and maintenance of these relays. It operates in a cloud-agnostic manner by interacting with the underlying cloud infrastructure through a generic RelayVmGateway interface. Its primary function is to periodically reconcile the state of all active relays against their desired configuration, ensuring they are healthy and operational.

Key responsibilities include:

• Provisioning and Deprovisioning: Creating new relay VMs when required and tearing them down when their associated discovery source is disabled.
• Health Reconciliation: Periodically checking the health of each relay VM and its internal services. Unhealthy VMs are automatically recreated.
• Configuration Management: Publishing runtime configuration to a central object store and ensuring VMs are bootstrapped with the correct settings.
• Automated Maintenance: Performing tasks such as automatic disk resizing to prevent outages due to disk pressure.
• Status Monitoring: Detecting and reporting on operational issues like container image divergence, missed heartbeats, and message shedding.

The service runs as a background process within the worker, iterating through all configured relays to enforce their state without manual intervention.

How it Works

The DiscoveryIngestHealthService executes its main logic within the reconcileActiveRelays method. This process runs periodically, performing a reconciliation loop for every active and enabled relay instance defined in the database.

sequenceDiagram
    participant Worker as Health Service
    participant DB as Database
    participant Cloud as Cloud Provider Gateway
    participant Blob as Object Store
    participant VM as Relay VM

    Worker->>DB: Load active relay targets
    Worker->>DB: Read desired container image digest

    loop For each relay target
        Worker->>DB: Check if discovery is enabled for gateway type
        alt Discovery is disabled
            opt deprovisionOnDiscoveryDisabled is true
                Worker->>Cloud: Delete VM instance
                Worker->>DB: Mark relay as deprovisioned
            else
                Worker->>Worker: Log S2R-INGEST-ORPHAN (once)
            end
            continue
        end

        Worker->>Blob: Publish runtime config, get signed URL
        Worker->>Cloud: Ensure firewall rule and internal IP exist
        Worker->>Cloud: Get VM instance details

        alt VM does not exist
            Worker->>Cloud: Create new VM instance
        end

        Worker->>VM: Probe /healthz endpoint
        alt VM is unhealthy
            opt Outside startup grace period
                Worker->>Worker: Dispatch async VM recreate task
                Worker->>DB: Update status to 'recreating'
            else
                Worker->>DB: Update status to 'starting'
            end
        else VM is healthy
            Worker->>Worker: Classify image, heartbeat, disk, & throughput
            opt Image is stale
                Worker->>DB: Update status to 'stale-image'
                Worker->>Worker: Log S2R-RELAY-DIVERGENCE
            end
            opt Heartbeat is stale
                Worker->>Worker: Log S2R-RELAY-NO-HEARTBEAT
            end
            opt Disk pressure > grow threshold
                Worker->>Cloud: Resize VM boot disk
                Worker->>Worker: Log S2R-RELAY-DISK-GROW
            end
            opt Throughput shedding > warn threshold
                Worker->>Worker: Log S2R-RELAY-SHEDDING
            end
            Worker->>DB: Update status to 'ok' with latest metrics
        end
    end

The reconciliation process for a single relay follows these steps:

1. Load Targets: The service queries the s2r_f5_instance and s2r_f5_sync_state tables for all relays that are marked as enabled.
2. Discovery Gate: For each relay, it checks the s2r_discovery_vendor_enabled table. If discovery for the corresponding gateway type is explicitly disabled, the service takes action based on the S2R_INGEST_DEPROVISION_ON_DISCOVERY_DISABLED setting:
   • If false (default), it logs an S2R-INGEST-ORPHAN warning once and takes no further action.
   • If true, it deprovisions the relay VM via the RelayVmGateway and updates its status in the database.
3. Ensure Healthy State: If discovery is enabled, the service proceeds to ensureRelayHealthy.
   • It generates a runtime configuration for the relay and publishes it as a JSON object to the platform's ObjectStore. It obtains a 24-hour signed URL for the VM to download this configuration.
   • It ensures the necessary cloud firewall rules and a static internal IP address exist via the RelayVmGateway.
   • It checks if the VM instance exists. If not, it is created with a detailed startup script.
   • If the instance exists, its health is checked by probing its /healthz endpoint.
4. Auto-Healing: If a VM is found to be unhealthy and is past its initial relayStartupGrace window, the service dispatches an asynchronous task to recreate it. This task, managed by RelayRecreateDispatcher, deletes the old VM and creates a new one without blocking the main reconciliation loop. The recreation attempt is subject to a 15-minute timeout.
5. Status Classification: For healthy relays, the service analyzes the data returned from the /healthz endpoint:
   • Image Divergence: It compares the relay's self-reported running container digest against a desired digest read from the s2r_system_setting table. If the digests differ for longer than the relayDivergenceGrace period, the relay is marked as stale-image.
   • Heartbeat: It checks the timestamp of the relay's last self-report. If it's older than the relayHeartbeatStale duration, an S2R-RELAY-NO-HEARTBEAT warning is logged.
   • Disk Pressure: It evaluates disk usage against relayDiskWarnPercent and relayDiskGrowPercent. If usage is high, it logs S2R-RELAY-DISK-PRESSURE. If it exceeds the growth threshold and relayDiskAutoExtendEnabled is true, it triggers a boot disk resize, up to relayDiskMaxGb.
   • Message Shedding: It checks the rate of dropped syslog messages. If the drop percentage exceeds relayDropWarnPercent, an S2R-RELAY-SHEDDING warning is logged.
6. Persist State: The final determined status, IP address, health metrics, and self-reported image information are written back to the s2r_f5_sync_state table.
7. Summary Report: After processing all targets, the method returns a summary map detailing the outcomes of the run.

Relay VM Startup Script

When creating or recreating a relay VM, the service generates a comprehensive startup script. This script is responsible for bootstrapping the VM's environment, installing necessary software, and configuring all required services. This ensures that each VM is provisioned identically and includes self-healing mechanisms.

Below is an example of the structure and key components of the generated script.

#!/bin/bash
set -euo pipefail
export DEBIAN_FRONTEND=noninteractive

# 1. Install self-healing and reconciliation scripts
mkdir -p /opt /usr/local/share/s2r
install -m 0755 ... /opt/update_relay.sh
install -m 0755 ... /opt/relay_reconcile.sh

# 2. Install systemd service to ensure scripts exist on boot
cat >/etc/systemd/system/s2r-update-relay-installer.service <<'...'
# ... service definition ...
systemctl enable s2r-update-relay-installer.service

# 3. Install dependencies
apt-get update
apt-get install -y python3 ca-certificates curl docker.io cloud-guest-utils e2fsprogs xfsprogs

# 4. Configure log rotation to cap disk usage
# Docker container logs
mkdir -p /etc/docker
cat >/etc/docker/daemon.json <<'DOCKERLOGEOF'
{
  "log-driver": "json-file",
  "log-opts": { "max-size": "100m", "max-file": "3" }
}
DOCKERLOGEOF
# Systemd journal
# ... journald.conf.d cap ...
# /var/log/syslog via hourly cron
# ... logrotate config and cron job ...

# 5. Configure and start Docker
systemctl enable docker
systemctl restart docker

# 6. Authenticate with artifact registry
# ... obtains token and runs docker login ...

# 7. Write environment and config files for services
cat >/etc/s2r-discovery-ingest.env <<'ENVEOF'
# ... env vars for local ingest container ...
ENVEOF
cat >/etc/s2r-discovery-ingest-reconcile.env <<'RECENVEOF'
# ... env vars for self-heal script ...
RECENVEOF
cat >/etc/s2r-discovery-ingest-relay-bootstrap.json <<'CFGEOF'
# ... bootstrap config for python relay script ...
CFGEOF

# 8. Install systemd units for services
cat >/etc/systemd/system/s2r-discovery-ingest-local.service <<'...'
# ... unit for local ingest container ...
cat >/etc/systemd/system/s2r-discovery-ingest-relay.service <<'...'
# ... unit for python syslog relay ...
cat >/etc/systemd/system/s2r-discovery-ingest-reconcile.timer <<'...'
# ... timer for jittered self-heal ...

# 9. Retire legacy pre-rename services to prevent conflicts
for legacy_unit in s2r-relay-reconcile.timer ...; do
    systemctl stop "$legacy_unit" >/dev/null 2>&1 || true
    # ... disable, rm, docker rm ...
done
systemctl daemon-reload

# 10. Pull container image and start all services
docker pull <local_ingest_image>
systemctl enable s2r-discovery-ingest-local.service
systemctl restart s2r-discovery-ingest-local.service
# ... enable and restart other services and timers ...

This script configures the VM with:

• A Docker container (s2r-discovery-ingest-local) running the worker image to act as a local ingest endpoint.
• A Python-based syslog relay (s2r-discovery-ingest-relay.service) that listens for syslog and forwards to the local ingest container.
• A self-healing mechanism (s2r-discovery-ingest-reconcile.timer) that periodically runs a script to ensure the ingest container is running the desired image version.
• Strict log rotation policies for Docker, journald, and /var/log/syslog to prevent disk exhaustion.

API Reference

The service exposes one primary public method for initiating the reconciliation process.

reconcileActiveRelays

Triggers a full reconciliation of all active discovery ingest relays.

• Inputs: None. The service reads its required state from the database and environment configuration.
• Output: A Map<String, Object> containing a summary of the actions taken during the run.

Example Success Response:

{
  "status": "DONE",
  "targetCount": 5,
  "healthy": 4,
  "starting": 0,
  "recreated": 1,
  "staleImage": 0,
  "noHeartbeat": 0,
  "orphanLogged": 0,
  "deprovisioned": 0,
  "diskGrown": 1,
  "shedding": 0,
  "failed": 0,
  "time": "2023-10-27T10:00:00.123456Z"
}

Response Fields:

Configuration

The behavior of the DiscoveryIngestHealthService and the relay VMs it manages is controlled by environment variables. The service supports new variable names with a one-release fallback to legacy names for backward compatibility.

| Name (Environment Variable) | Type | Default | Description | --- | --- | --- | | S2R_RELAY_VM_PROVIDER | String | gcp | Selects the cloud provider implementation for the RelayVmGateway. | S2R_PROJECT_ID | String | "" | The default cloud project ID for relay VM resources. | S2R_DISCOVERY_INGEST_RELAY_REGION | String | me-west1 | Default cloud region for relay VMs. Falls back to S2R_F5_RELAY_REGION, then S2R_REGION.

| S2R_DISCOVERY_INGEST_RELAY_ZONE | String | [region]-b | Default cloud zone for relay VMs. Falls back to S2R_F5_RELAY_ZONE.

| S2R_DISCOVERY_INGEST_RELAY_MACHINE_TYPE | String | e2-standard-4 | Machine type for relay VMs. Falls back to S2R_F5_RELAY_MACHINE_TYPE.

| S2R_DISCOVERY_INGEST_RELAY_IMAGE_FAMILY | String | debian-12 | Base OS image family for relay VMs. Falls back to S2R_F5_RELAY_IMAGE_FAMILY.

| S2R_DISCOVERY_INGEST_RELAY_IMAGE_PROJECT | String | debian-cloud | Cloud project containing the base OS image. Falls back to S2R_F5_RELAY_IMAGE_PROJECT.

| S2R_DISCOVERY_INGEST_VM_INGEST_IMAGE | String | [region]-docker.pkg.dev/[proj]/[repo]/s2r-worker:dev | The full reference to the Docker image for the local ingest container on the relay VM. Falls back to S2R_F5_VM_INGEST_IMAGE.

| S2R_DISCOVERY_INGEST_RELAY_VM_SERVICE_ACCOUNT | String | s2r-worker-sa@[proj].iam.gserviceaccount.com | The email of the service account assigned to the relay VM. Falls back to S2R_F5_RELAY_VM_SERVICE_ACCOUNT.

| S2R_INGEST_DEPROVISION_ON_DISCOVERY_DISABLED | Boolean | false | If true, the service will deprovision relays whose discovery source is disabled. Otherwise, it only logs a warning.

| Relay VM Behavior | | |

| S2R_DISCOVERY_INGEST_RELAY_SYSLOG_PORT | Integer | 514 | TCP/UDP port on the relay VM for receiving syslog data. Falls back to S2R_F5_RELAY_SYSLOG_PORT.

| S2R_DISCOVERY_INGEST_RELAY_HEALTH_PORT | Integer | 18080 | Port on the relay VM for the health check endpoint (/healthz). Falls back to S2R_F5_RELAY_HEALTH_PORT.

| S2R_DISCOVERY_INGEST_RELAY_LOCAL_INGEST_PORT | Integer | 18081 | Internal port used by the syslog forwarder to talk to the local ingest container. Falls back to S2R_F5_RELAY_LOCAL_INGEST_PORT.

All SOAP-to-REST docs