F5, Runtime, and Generic Traffic Aggregation Refresh Service

Overview

The Aggregation Refresh Service is a background process responsible for transforming high-volume, raw traffic logs into pre-aggregated, hourly summary tables. This process is critical for the performance of observability dashboards, which query these summary tables instead of the raw logs directly.

The service operates on three primary data sources:

1. F5 Traffic Logs: Data from F5 BIG-IP instances, captured in s2r_traffic_log.
2. Runtime Traffic Logs: Data from the platform's own runtime, captured in s2r_runtime_log.
3. Generic Traffic Logs: Data from other supported gateways (e.g., Citrix, HAProxy, NGINX), captured in s2r_generic_request_log.

It runs as a scheduled, multi-tenant job that processes data for all enabled sources. It can also be triggered on-demand to refresh data for a specific F5 instance if it is detected to be stale, ensuring that dashboards remain reasonably up-to-date without incurring the performance penalty of synchronous, on-demand aggregation.

How it Works

The service employs two primary mechanisms for refreshing data: a periodic scheduled refresh for all sources and an on-demand asynchronous refresh for stale F5 instance data.

flowchart TD
    subgraph "Scheduled Refresh (Periodic)"
        direction TD
        Scheduler["@Scheduled Timer\n(e.g., every 30 min)"] --> ScheduledRun[scheduledRefresh]
        ScheduledRun --> TenantLoop{Per-Tenant Loop}
        TenantLoop --> F5Refresh["F5 Incremental Refresh\n(for each enabled instance)"]
        TenantLoop --> RuntimeRefresh[Runtime Aggregation Refresh]
        TenantLoop --> GenericRefresh[Generic Aggregation Refresh]
        F5Refresh --> AggTables[(Aggregation Tables)]
        RuntimeRefresh --> AggTables
        GenericRefresh --> AggTables
    end

    subgraph "On-Demand Stale Refresh (F5 Only)"
        direction TD
        APIConsumer["API Consumer\n(e.g., Dashboard UI)"] --> StaleCheck{refreshIfStale}
        StaleCheck -- "Data is fresh" --> ReturnStale["Returns immediately\n(with existing data)"]
        StaleCheck -- "Data is stale" --> SubmitAsync["Submits background job"]
        SubmitAsync --> ReturnStale
        SubmitAsync --> AsyncExecutor["Single-thread executor"]
        AsyncExecutor --> BackgroundRefresh["runStaleCheckAndRefresh"]
        BackgroundRefresh --> F5RefreshAsync["F5 Incremental or Full Refresh"]
        F5RefreshAsync --> AggTables
    end

Scheduled Refresh

The primary refresh mechanism is a scheduled job that runs at a configurable interval (defaulting to every 30 minutes).

1. The scheduledRefresh method is triggered by the platform's scheduler. It uses a lock (AtomicBoolean running) to prevent multiple scheduled jobs from running concurrently.
2. It iterates through each tenant in the system using a PerTenantJobRunner. All subsequent database queries for that iteration are scoped to the current tenant.
3. Within each tenant's context, the refreshAllSourcesForCurrentTenant method executes the aggregation for each data source independently:
   • F5 Traffic: It fetches all enabled F5InstanceRecords and calls incrementalRefresh for each.
   • Runtime Traffic: It calls refreshRuntimeServiceAggregation.
   • Generic Traffic: It calls refreshGenericServiceAggregation.
4. A failure in refreshing one source (e.g., a single F5 instance) is logged but does not interrupt the refresh process for other sources.
5. Upon completion, it logs a structured message S2R-AGG-REFRESH with success and failure counts, which can be used for monitoring the health of the aggregation process.

On-Demand Stale Refresh (F5 Only)

To provide fresher data for interactive dashboards without blocking API requests, the service implements an asynchronous, on-demand refresh mechanism for F5 instances.

1. An external caller, such as an observability API endpoint, can invoke refreshIfStale(instanceId).
2. The method first checks an in-memory timestamp. If a refresh for the given instance has occurred within the staleness threshold (STALE_THRESHOLD_MS, 30 minutes), it does nothing.
3. If the data is considered stale, it checks if an asynchronous refresh for that same instance is already in progress. If so, it does nothing to avoid redundant work.
4. If a refresh is needed, it submits a task to a dedicated single-thread background executor (asyncRefreshExecutor) and returns immediately. The calling API proceeds with the currently available (stale) data.
5. The background task (runStaleCheckAndRefresh) then performs a more rigorous check against the s2r_f5_agg_refresh_state table in the database. Based on this state, it triggers either an incrementalRefresh or a fullRefresh for the specified F5 instance.

This design ensures that API response times are not affected by long-running aggregation queries, while still providing a mechanism to update data that has become significantly out-of-date between scheduled runs.

Aggregation Stages

The refresh process consists of several distinct aggregation stages, each populating a specific summary table. For F5 traffic, an incremental refresh processes data from the last known point (or the last 8 days on a cold start), while a full refresh always processes the last 8 days.

F5 Service Aggregation

The refreshServiceAggregation method aggregates data from s2r_observed_traffic_hourly into s2r_agg_service_hourly. It also performs a second pass to join against the raw s2r_traffic_log to calculate latency sketches (sum, count, max), as latency data is not present in the intermediate hourly table.

F5 Consumer Aggregation

The refreshConsumerAggregation method populates the s2r_agg_consumer_hourly table, summarizing traffic by the consumer's IP address (client_ip and gateway_client_ip). It aggregates hit counts and compiles a list of service URIs accessed by each consumer within the hour.

F5 VS Failure Aggregation

The refreshVsFailureAggregation method is specifically designed to power dashboards monitoring VS-level connection failures. It reads the raw s2r_traffic_log and counts events that indicate a failure at the virtual server level (e.g., service_id is null and frontend_uri is empty or /). It stores these hourly counts in s2r_f5_agg_vs_failure_hourly.

F5 URI Failure Attribution

The refreshUriFailureAttribution method solves a specific problem where F5 logs for backend errors often lack the original request URI. To correctly attribute these failures, it performs a multi-step process:

1. Sibling URI Resolution (SQL): It queries the raw s2r_traffic_log, joining failure events to other log events that share the same global_transaction_id. It finds the earliest "sibling" event that has a non-empty frontend_uri and uses that as the "real" URI for the failure.
2. Canonicalization (Java): The raw URIs resolved in the previous step are processed in Java. Each URI is passed to MasterDiscoveryAggregator.canonicalUri to produce a standardized, canonical form. This ensures that different raw URIs that represent the same logical endpoint (e.g., due to case differences) are grouped together.
3. Upsert (SQL): The final aggregated counts per canonical URI are written to the s2r_f5_agg_uri_failure_hourly table.

F5 Observed Topology

The refreshObservedTopology method rebuilds the s2r_observed_topology table, which maps the observed relationships between front-end virtual servers, back-end virtual servers, service URIs, and backend hosts. It aggregates hit counts over 24-hour and 7-day windows from s2r_observed_traffic_hourly.

Runtime and Generic Traffic Aggregation

The refreshRuntimeServiceAggregation and refreshGenericServiceAggregation methods perform aggregations for runtime and generic gateway traffic, respectively. They read from their corresponding raw log tables (s2r_runtime_log, s2r_generic_request_log) and write to dedicated hourly summary tables (s2r_runtime_agg_service_hourly, s2r_generic_agg_service_hourly). This allows observability dashboards to present a unified view of traffic across all sources by taking the UNION of the F5, runtime, and generic aggregation tables.

Data Retention

After each refresh cycle for an F5 instance, a cleanup process runs. It deletes records older than a defined retention period from the aggregation tables to manage storage growth.

• 8 days: s2r_agg_service_hourly, s2r_agg_consumer_hourly, s2r_f5_agg_vs_failure_hourly, s2r_f5_agg_uri_failure_hourly. The same retention is applied to s2r_runtime_agg_service_hourly and s2r_generic_agg_service_hourly.
• 13 months: s2r_observed_traffic_hourly.

Configuration

The service's behavior is controlled by the following application properties and internal constants.

API Reference

The service exposes public methods that can be invoked by other components within the platform.

refreshIfStale

Triggers a non-blocking, asynchronous refresh for a specific F5 instance if its aggregated data is older than 30 minutes.

• Method: public void refreshIfStale(UUID instanceId)
• Parameters:
  • instanceId (UUID): The ID of the F5 instance to check and potentially refresh.
• Behavior:
  • The method returns immediately, preventing the caller from blocking.
  • If a refresh is initiated, it runs on a background thread. The caller will see existing data, which will be updated once the background job completes.
  • If a refresh for the specified instance is already in progress, no new refresh is scheduled.

fullRefresh

Triggers a blocking, full refresh for a specific F5 instance, reprocessing data from the last 8 days.

• Method: public void fullRefresh(UUID instanceId)
• Parameters:
  • instanceId (UUID): The ID of the F5 instance to refresh.
• Behavior:
  • This method is typically called internally by the stale-check logic for a cold start (when no previous refresh state exists).
  • It synchronously executes all aggregation and cleanup stages for the specified instance.

Troubleshooting

The service provides structured logs to aid in monitoring and troubleshooting.

Common Issues

• Stale Data on Dashboards: Dashboards may show data that is up to 30 minutes old by design. This is the default refresh interval. The refreshIfStale mechanism helps close this gap for frequently viewed F5 instances but is not instantaneous.
• Database Deadlocks: In older versions, concurrent refreshes could cause deadlocks on the s2r_f5_agg_service_hourly table. This is now mitigated by a pg_advisory_xact_lock. If deadlock errors reappear, it could indicate a problem with this locking mechanism or contention on other tables.
• Zero Failures Reported: If dashboards show zero failures for services known to be experiencing errors, investigate the refreshUriFailureAttribution process. Failures in this stage can prevent errors from being correctly counted. Check for uri_failure attribution refresh failed warnings in the logs.

All SOAP-to-REST docs