F5 Catalog and Relay Warmup on Startup

Overview

The F5CatalogWarmupRunner is a system component that automatically runs when the Admin API service starts up. Its purpose is to proactively populate, or "warm up," the system's cache and state for all configured F5 BIG-IP instances. This ensures that data about F5 configurations and managed relay Virtual Machines (VMs) is fresh and available immediately after a service deployment or restart, preventing initial API calls from experiencing delays.

The warmup process is triggered by the ApplicationReadyEvent, which signals that the application is fully initialized. The process operates on a per-tenant basis, ensuring data isolation. For each tenant, it identifies all enabled F5 instances and performs three key actions for each one:

1. Fetches a catalog snapshot of the F5 instance's configuration.
2. Performs a WAF policy scan if policies are configured.
3. Ensures any associated managed relay VM is healthy and provisioned.

This startup routine is critical for system readiness, especially the asynchronous dispatch of relay VM provisioning, which prevents long-running cloud operations from blocking the service's startup and delaying its "ready" status.

How it Works

The warmup process follows a specific sequence of operations upon application startup. The entire flow is designed to be robust, with error handling at each step and mechanisms to avoid blocking the main application thread.

flowchart TD
    subgraph Admin API Service Startup
        A[Application Ready] --> B{For each Tenant};
        B --> C{Find enabled F5 instances};
        C --> D{Any instances?};
        D -- No --> E[Log and skip];
        D -- Yes --> F{For each F5 instance};
        F --> G[1. Warm Catalog Snapshot];
        G --> H[Persist Catalog State];
        H --> I[2. Warm Policy Scan];
        I --> J[Persist Scan State];
        J --> K[3. Warm Managed Relay];
    end

    subgraph "Async Relay VM Warmup"
        K -- dispatches --> L{Is Relay Enabled?};
        L -- No --> M[Stop];
        L -- Yes --> N[Dispatch to background executor];
        N --> O[Ensure Relay VM is Healthy];
        O --> P[Persist Relay State];
    end

    subgraph Error Handling
        G -- on exception --> G_Fail[Record Catalog Failure];
        I -- on exception --> I_Fail[Record Scan Failure];
        O -- on exception --> O_Fail[Record Relay Unhealthy];
    end

Detailed Steps

1. Trigger: The process begins when the application context emits an ApplicationReadyEvent.
2. Tenant Iteration: The F5CatalogWarmupRunner iterates through all tenants configured in the system, executing the warmup process independently for each one.
3. Instance Discovery: For the active tenant, the system queries the database for all F5InstanceRecord entries. It filters this list to include only instances that are enabled and have a managementUrl defined. If no such instances are found, it logs a message and the process for that tenant concludes.
4. Catalog Snapshot: For each enabled instance, the system first attempts to fetch a catalog snapshot by calling the F5DiscoveryService.
   • On Success: The retrieved snapshot data, along with a status (e.g., ok), is persisted to the database via f5SyncStateRepository.markCatalogLoaded.
   • On Failure: If an exception occurs, a failure record is created. This record includes a status of failed, the exception message, and a placeholder data structure. This failure state is then persisted to the database.
5. Policy Scan: After the catalog snapshot attempt, the system proceeds to warm up the WAF policy scan data if the instance has selectedPolicyNames configured.
   • On Success: The results of the policy scan are persisted to the database via f5SyncStateRepository.markManualScan.
   • On Failure: Similar to the catalog snapshot, any exception during the scan results in a failed status and error message being recorded in the database.
6. Managed Relay Warmup: The final step for an instance is to check and warm up its managed relay VM.
   • The system checks if the instance's sync state has relayEnabled set to true. If not, this step is skipped.
   • If enabled, the task to ensure the relay is healthy is dispatched asynchronously to a dedicated, bounded thread pool (RelayWarmupDispatcher). This prevents the main startup thread from blocking on potentially slow cloud infrastructure operations (e.g., creating a VM), which could take several minutes.
   • Each asynchronous task has a 15-minute timeout (WARMUP_TIMEOUT).

Asynchronous Relay Provisioning

The runManagedRelayWarmup task executes in a background thread and performs the following:

1. Calls DiscoveryIngestVmService.ensureRelayHealthy for the instance. This service handles the logic of checking, creating, or recreating the relay VM to ensure it is operational.
2. On Success: It updates the database with the relay VM's details (name, IP address, status, etc.) by calling f5SyncStateRepository.markRelayProvisioned.
3. On Failure: If an exception occurs (e.g., the VM creation fails or times out), it updates the relay's health status to unhealthy in the database via f5SyncStateRepository.markRelayHealth.

Configuration

The relay warmup process depends on a public URL for the discovery ingest service. This URL is resolved using a specific order of precedence.

Troubleshooting

• Meaning: The system logs a warn message (e.g., Startup F5 relay warmup failed for instance=...). The relay's health status is marked as unhealthy in the database, with the error message from the exception.
• Action: Check application logs for the specific error message. The issue will be automatically retried by the periodic relay health reconciliation task. | Missing Ingest URL Configuration | The S2R_DISCOVERY_INGEST_PUBLIC_URL (or S2R_F5_INGEST_PUBLIC_URL) environment variable is not set, AND the ingest_public_url field is not set on the F5 instance record in the database. |
• Meaning: The application throws an IllegalStateException with message S2R-ADM-0427 ingest public URL is not configured.... The relay warmup for that instance is aborted.
• Action: Configure the S2R_DISCOVERY_INGEST_PUBLIC_URL environment variable for the Admin API service, or ensure the ingest_public_url is correctly set on the F5 instance configuration via the API.

See Also

• Ongoing Relay VM Health Reconciliation: A separate scheduled service (DiscoveryIngestHealthService) periodically checks the health of all active relay VMs and attempts to reconcile any that are unhealthy, independently of this startup process.

All SOAP-to-REST docs