Master Discovery Table Refresh

Overview

The Master Discovery Refresh Service is a background process responsible for populating and maintaining the s2r_master_discovery materialized table. This table serves as a pre-computed cache of all discovered API services, which allows the user-facing API to respond quickly without performing expensive data computations in the request path.

The service operates on a configurable schedule. In each run, it queries live traffic data from API gateways, processes this data through a series of enrichment and aggregation steps, and then "upserts" the results into the master discovery table.

It supports two main sources of traffic data:

1. F5 Gateways: It processes traffic on a per-instance basis for all enabled F5 instances.
2. Generic Gateways: It processes traffic for non-F5 gateways, grouped by vendor (gateway_type) and specific gateway instance (connector_instance_id).

The service can also be triggered manually for a specific F5 instance or generic gateway scope, which is useful for bootstrapping a new gateway or forcing an immediate refresh.

How it Works

The service's primary function is a scheduled job that orchestrates the refresh process for all configured gateways. To prevent data corruption from concurrent writes, the service uses a locking mechanism at multiple levels: a global lock for the entire scheduled cycle, and per-instance or per-scope locks for individual refresh operations.

Scheduled Refresh Cycle

The scheduledRefresh method is the main entry point, triggered by an internal scheduler.

flowchart TD
    A[Scheduler triggers refresh] --> B{Service enabled?};
    B -- No --> Z[Stop];
    B -- Yes --> C{Acquire global cycle lock};
    C -- Failure (already running) --> Z;
    C -- Success --> D[Find all enabled F5 instances];
    D --> E[For each F5 instance...];
    E --> F["Run F5 Refresh Logic (see below)"];
    F --> E;
    E -- All F5 instances processed --> G[Find generic gateway scopes with traffic];
    G --> H[For each generic scope...];
    H --> I["Run Generic Refresh Logic (see below)"];
    I --> H;
    H -- All generic scopes processed --> J[Release global cycle lock];

The cycle performs the following high-level steps:

1. Checks if the service is enabled (s2r.master-discovery.refresh.enabled) and if a refresh cycle is already running. If not, it proceeds.
2. Iterates through each enabled F5 instance and calls the refreshInstance method. Errors during the refresh of one instance are logged but do not stop the cycle from processing other instances.
3. After processing all F5 instances, it calls refreshGenericVendors to handle all non-F5 traffic. This method identifies all unique combinations of generic gateway vendors and instances (gateway_type, connector_instance_id) that have observed traffic.
4. For each generic scope, it calls the refreshGenericVendor method. Errors are isolated per scope.
5. Finally, it releases the global lock, allowing the next scheduled cycle to run.

F5 Instance Refresh Flow

The refreshInstance(UUID instanceId) method processes data for a single F5 gateway instance.

The refresh follows these steps:

1. Fetch Raw Data: Computes raw discovery rows from live traffic data for the instance by calling f5LearningRepository.computeMasterDiscoveryLive. The computation is capped at a limit of 5,000 rows.
2. Filter Invalid URIs: Removes any rows where the serviceUri is empty or simply /.
3. Aggregate Rows: Collapses the raw data, which may have multiple rows per service, into a single, canonical row for each unique combination of serviceUriCanonical and environment. This step uses environment definitions and a vocabulary of known operation names to correctly group the data.
4. Apply Manual Overrides: Fetches any manual overrides defined for the instance and applies them to the corresponding aggregated rows. Overrides are matched using a composite key of serviceUriCanonical and environment.
5. Link to Service Inventory: Enriches rows by matching their serviceBackendUrl against the service inventory. If a match is found, the serviceId and serviceStatus fields are populated.
6. Set Discovery Status: Each row's discoveryStatus is set to complete or incomplete based on the logic in the isComplete method (see below).
7. Upsert Data: The final, processed list of rows is written to the s2r_master_discovery table. This operation updates existing rows or inserts new ones.
8. Clean Up Stale Overrides: After the upsert, the service removes any manual overrides that no longer correspond to any discovered service in the current batch, preventing the accumulation of obsolete overrides.

Generic Gateway Refresh Flow

The refreshGenericVendor(String gatewayType, UUID connectorInstanceId) method processes data for a single non-F5 gateway scope. This flow is a streamlined version of the F5 path.

The refresh follows these steps:

1. Fetch Raw Data: Computes raw discovery rows from live traffic for the specified gatewayType and connectorInstanceId by calling f5LearningRepository.computeMasterDiscoveryLiveGeneric. The computation is capped at a limit of 5,000 rows.
2. Filter Invalid URIs: Same as the F5 flow.
3. Aggregate Rows: Same as the F5 flow.
4. Link to Service Inventory: Same as the F5 flow.
5. Set Discovery Status: Same as the F5 flow.
6. Upsert Data: The final rows are written to the s2r_master_discovery table, scoped to the gatewayType and connectorInstanceId.

Discovery Status Logic

A discovered service row is marked with a discoveryStatus of complete if it meets a specific set of criteria, indicating that enough information has been gathered to consider it fully identified. A row is considered "complete" if it satisfies all of the following conditions:

• Has a non-blank serviceUri.
• Has a non-blank environment.
• AND has at least one of the following:
  • A non-blank serviceBackendUrl or a non-empty list of serviceBackendUrls.
  • A complete topology path, defined as having both a non-empty frontVirtualServers list AND either a non-empty backVirtualServers list or a non-blank dataPowerDomain.

If these conditions are not met, the discoveryStatus is set to incomplete.

Example Outcome

A successful refresh for an F5 instance will produce a log message similar to the following, indicating the number of master service rows and associated consumer rows that were updated or inserted.

INFO Master discovery refresh for instance {instanceId}: 42 master rows + 120 consumer rows in 1532ms

Similarly, a successful refresh for a generic gateway scope will produce a log message like this:

INFO Master discovery refresh for generic scope my-vendor/a1b2c3d4-e5f6-7890-1234-567890abcdef: 15 master rows + 35 consumer rows in 890ms (source=generic)

Configuration

The service's schedule is configured via application properties.

Service API Reference

While not exposed via HTTP, the service has public methods that can be invoked by other components within the platform for synchronous refreshes.

Troubleshooting

The service logs detailed messages for both successful and failed operations. The following are common warnings and errors.

All SOAP-to-REST docs