Discovery Ingest Relay VM Orchestration

Overview

The Discovery Ingest VM Service is an orchestration component responsible for the complete lifecycle management of dedicated relay virtual machines (VMs). These relay VMs are designed to receive syslog data from external sources and forward it to the platform's central ingest endpoints. This service automates the provisioning, configuration, health monitoring, and decommissioning of these VMs, providing a managed bridge for discovery data.

This service resides within the platform's administration API. It is invoked to ensure that a relay VM is available and healthy for a given data source. It abstracts the underlying cloud provider details by delegating all direct cloud infrastructure operations (e.g., creating VMs, managing IP addresses) to a RelayVmGateway component. The specific implementation of this gateway is determined at startup by the S2R_RELAY_VM_PROVIDER environment variable.

For configuration, the service generates runtime parameters for the relay VM, publishes them to the platform's central ObjectStore (such as GCS or S3), and provides the VM with a secure, short-lived URL to download this configuration upon startup.

How it Works

The core function of the service is to ensure a relay VM is running and healthy. This process, encapsulated in the ensureRelay method, involves coordination between the service, the cloud provider gateway, and the object store.

sequenceDiagram
    participant Admin Service
    participant DiscoveryIngestVmService
    participant Object Store
    participant Cloud Provider Gateway
    participant Relay VM

    Admin Service->>+DiscoveryIngestVmService: ensureRelay(instance)
    DiscoveryIngestVmService->>+Object Store: put(config.json)
    Object Store-->>-DiscoveryIngestVmService: Signed URL for config
    DiscoveryIngestVmService->>+Cloud Provider Gateway: ensureFirewallRule()
    Cloud Provider Gateway-->>-DiscoveryIngestVmService: OK
    DiscoveryIngestVmService->>+Cloud Provider Gateway: ensureInternalAddress()
    Cloud Provider Gateway-->>-DiscoveryIngestVmService: Static IP Address
    DiscoveryIngestVmService->>+Cloud Provider Gateway: getInstanceJson()
    alt VM does not exist
        DiscoveryIngestVmService->>Cloud Provider Gateway: createRelayInstance(startupScript)
        Note over Cloud Provider Gateway, Relay VM: VM boots, runs startup script
        Relay VM-->>Object Store: GET /config.json (via signed URL)
        Relay VM-->>Relay VM: Starts relay & ingest services
        DiscoveryIngestVmService->>+Relay VM: Poll /health endpoint
        Relay VM-->>-DiscoveryIngestVmService: 200 OK
    end
    alt VM is unhealthy
        DiscoveryIngestVmService->>+Cloud Provider Gateway: deleteInstance()
        Cloud Provider Gateway-->>-DiscoveryIngestVmService: OK
        DiscoveryIngestVmService->>DiscoveryIngestVmService: Re-run creation steps
    end
    DiscoveryIngestVmService-->>-Admin Service: RelayVmInfo (IP, status)

The orchestration flow is as follows:

1. Initiation: An internal service calls ensureRelay with details about the data source, including an F5InstanceRecord.
2. Configuration Publication: The service generates a runtime configuration payload for the relay components. This configuration is uploaded as a JSON object to a predefined path in the platform's ObjectStore. The service then obtains a 24-hour signed URL for this object.
3. Network Provisioning: The service instructs the RelayVmGateway to ensure the necessary firewall rules and a static internal IP address for the VM are in place.
4. Instance Check: It queries the RelayVmGateway to determine if the VM instance already exists. Instance and address names are derived deterministically from the data source's unique ID.
5. VM Creation: If the VM does not exist, the service: a. Constructs a detailed startup-script. This script bootstraps the entire VM environment. b. Builds the full VM instance definition as a JSON object, including machine type, disk, network interface (with the static IP), and the startup script. c. Calls gateway.createRelayInstance to provision the VM. d. Enters a waiting loop (waitForRelayHealth), polling the VM's health endpoint until it becomes healthy or a timeout (relayBootstrapTimeout) is reached.
6. Health Verification: If the VM already exists (or after it is newly created), its health is checked via the relayHealthy method. A healthy state requires:
   • The VM instance status is RUNNING.
   • The relay's health endpoint (:18080/) is reachable and reports status: "ok".
   • The relay mode is vm-local-ingest.
   • The local ingest container's status is UP or ok.
7. Recreation on Failure: If the health check fails and the recreateOnUnhealthy flag is set, the service deletes the existing VM and recreates it from scratch. If the flag is not set, it throws an IllegalStateException.
8. Completion: Once a healthy VM is confirmed, the service returns a RelayVmInfo object containing the instance name, IP address, and status.

VM Startup and Self-Healing

The startup-script provisioned with each VM is critical for its operation and resilience. It performs several key tasks:

• Prerequisites: Installs necessary packages like docker.io, python3, and curl.
• Log Capping: Configures docker, journald, and logrotate to enforce size and file limits on logs. This is a critical fix to prevent disk exhaustion from high-volume syslog traffic.
• Self-Healing Scripts: Installs two key helper scripts, /opt/update_relay.sh and /opt/relay_reconcile.sh. A systemd service (s2r-update-relay-installer.service) runs on every boot to ensure these scripts are present and executable, defending against their accidental removal.
• Container Configuration: Creates environment files on disk (/etc/s2r-discovery-ingest.env, /etc/s2r-discovery-ingest-reconcile.env) that provide configuration to the containers.
• Systemd Services: Sets up and enables several systemd services:
  • s2r-discovery-ingest-local.service: Manages the local ingest Docker container (s2r-worker), which processes the data.
  • s2r-discovery-ingest-relay.service: Runs the Python-based syslog relay script that listens for traffic and forwards it to the local ingest container.
  • s2r-discovery-ingest-reconcile.timer: A timer that periodically (every 5 minutes, with jitter) triggers s2r-discovery-ingest-reconcile.service. This service executes /opt/relay_reconcile.sh, which contacts the control plane (via the local ingest container) to fetch the desired container image tag and updates the container if necessary.
• Log Forwarding: Installs and configures the Google Cloud Ops Agent to collect logs from the Docker container and forward them to a central logging service.

Service API

The service exposes the following public Java methods for internal use.

| Method | Description | ensureRelay(F5InstanceRecord instance, String ingestUrl, boolean recreateOnUnhealthy) | The primary orchestration method. Ensures a relay VM is provisioned and healthy. If the VM is found to be unhealthy, it is recreated only if recreateOnUnhealthy is true. | ensureRelay(F5InstanceRecord instance, String ingestUrl) | A convenience wrapper for ensureRelay that defaults recreateOnUnhealthy to false. | ensureRelayHealthy(F5InstanceRecord instance, String ingestUrl) | A convenience wrapper for ensureRelay that defaults recreateOnUnhealthy to true, ensuring a healthy VM is always present by recreating it if necessary. | deleteRelay(F5InstanceRecord instance) | Deletes the relay VM instance and its associated static internal IP address.

| relayHealth(F5InstanceRecord instance, F5SyncStateRecord syncState) | Fetches health information from the relay VM's health endpoint. It enriches this data with VM instance status and CPU/memory metrics from the cloud provider's monitoring service.

Configuration

The service is configured via environment variables. Many variables support a fallback to a legacy name (e.g., S2R_F5_*) to ensure backward compatibility during a migration period. The service will always prefer the new S2R_DISCOVERY_INGEST_* name if it is set.

| Variable | Type | Default Value | Description | --- | --- | --- | | S2R_RELAY_VM_PROVIDER | String | gcp | Selects the cloud provider implementation for the RelayVmGateway. | S2R_PROJECT_ID | String | "" (empty string) | The default cloud project ID for relay VM operations. | S2R_REGION | String | me-west1 | The default cloud region for the platform. Used as a fallback for the relay VM region. | S2R_DISCOVERY_INGEST_RELAY_REGION | String | Value of S2R_REGION | The default cloud region for provisioning relay VMs. Falls back to S2R_F5_RELAY_REGION, then S2R_REGION. | S2R_DISCOVERY_INGEST_RELAY_ZONE | String | [defaultRegion]-b | The default cloud zone for provisioning relay VMs. Falls back to S2R_F5_RELAY_ZONE. | S2R_DISCOVERY_INGEST_RELAY_MACHINE_TYPE | String | e2-standard-4 | The machine type for the relay VM. Falls back to S2R_F5_RELAY_MACHINE_TYPE.

All SOAP-to-REST docs