Generic Traffic Learning Service

Overview

The Generic Traffic Learning Service is a scheduled background process that curates API traffic from generic, non-F5 sources. It functions as a multi-source learned-example curator, mirroring the capabilities of the F5-specific traffic learning service but adapted for different log formats.

The primary problem this service solves is the ingestion and processing of API traffic logs that are bulk-inserted into a database, rather than being streamed via individual HTTP requests. The s2r-generic-ingest service populates the s2r_generic_request_log table, and this service polls that table for new records to process.

The service's main responsibilities are:

• Curating "Good Examples": It identifies successful API transactions and persists them as distinct, learnable examples in the s2r_generic_learned_example table.
• Traffic Observation: It aggregates all processed traffic—both successful and failed—into hourly rollups in the s2r_observed_traffic_hourly table for discovery and analytics.
• Topology Discovery: It records the relationships between frontend and backend components in the s2r_observed_topology table.
• Field-Level Learning: For SOAP payloads, it extracts individual leaf fields and their values into s2r_generic_learned_field_value to aid in API onboarding and data encoding analysis.

This service operates independently of F5-related traffic processing to ensure that a failure in one does not impact the other.

How it Works

The service runs periodically, processing new traffic logs in batches. Its operation is designed to be robust, idempotent, and safe for concurrent execution in a multi-instance environment.

Scheduling and Concurrency Control

• Scheduled Execution: The service is invoked by a scheduler at a configurable interval (s2r.generic-learning.scheduler.delay-ms), with an initial startup delay (s2r.generic-learning.scheduler.initial-delay-ms).
• Re-entrancy Guard: A single instance of the service will not execute a new processing tick if the previous one is still running. This prevents overlapping runs and resource contention on a single node.
• Cross-Instance Locking: To prevent multiple service instances in a cluster from processing the same logs simultaneously, the service acquires a cluster-wide advisory lock (pg_try_advisory_xact_lock) at the start of each run. The lock is transaction-scoped, ensuring it is automatically released upon commit or rollback, which prevents orphaned locks. If the lock cannot be acquired, the tick is skipped.

Watermark and State Management

The service uses an in-memory watermark (lastProcessedId) to track its progress through the s2r_generic_request_log table.

• Initialization: On its first run after a process start, the service initializes the watermark by querying the s2r_generic_learned_example table for the highest generic_request_log_id it has already processed. This makes the watermark durable across restarts, preventing the service from re-processing the entire log history from the beginning.
• Advancement: The watermark is only advanced after a batch of rows has been successfully processed and the corresponding database transaction has been committed. If a transaction fails, the watermark remains unchanged, and the batch will be retried on the next tick.
• Idempotency: The system is designed to be idempotent. If a worker restarts and re-processes logs (e.g., if the watermark is reset to 0), the INSERT ... ON CONFLICT logic in the database prevents duplicate examples and ensures that counters like seen_count are not incorrectly inflated. A special WHERE clause in the UPDATE statement checks the source log ID to make re-processing a no-op.

Processing Flow

Each processing tick follows a series of steps to fetch, process, and persist traffic data.

flowchart TD
    subgraph Service Tick
        A[Scheduler Triggers] --> B{Acquire Advisory Lock?}
        B -- Yes --> C[Fetch Batch from s2r_generic_request_log]
        B -- No --> D[Skip Tick]
        C --> E{For each row in batch}
        E -- row --> F[Parse & Identify]
        F --> G[Update Discovery Tables<br>(s2r_observed_traffic_hourly, s2r_observed_topology)]
        G --> H{Is Traffic Successful?}
        H -- Yes --> I[Upsert to s2r_generic_learned_example]
        I --> J[Enforce Example Cap]
        J --> K[Learn Leaf Fields from SOAP]
        K --> L[End Row Processing]
        H -- No --> L
        L -- next row --> E
        E -- end of batch --> M{Commit Transaction}
        M -- Success --> N[Advance Watermark]
        M -- Failure --> O[Rollback & Log Error<br>(Watermark Unchanged)]
    end

1. Fetch Batch: The service queries the s2r_generic_request_log table for rows with an id greater than the current watermark, up to a limit of BATCH_SIZE. It includes a COMMIT_LAG_SECONDS delay to avoid processing rows from transactions that may not have committed yet.
2. Parse and Identify: For each row, the service:
   • Classifies the payload as SOAP, XML, JSON, or NONE.
   • Derives a stable service_backend_key from the backend_uri. If the URI only contains a path, it creates a synthetic key based on front_virtual_server or frontend_uri.
   • Masks sensitive header values (e.g., Authorization, Cookie) with ***.
   • Determines an operationName from the payload (e.g., SOAP operation name, XML root element, or request path).
   • Calculates a SHA-256 hash of the request payload to uniquely identify the example.
   • Attempts to resolve a canonical service_id by matching the backend URI against onboarded service profiles.
3. Update Discovery Data: For all traffic (both success and failure), it updates two tables:
   • s2r_observed_traffic_hourly: Increments hit counters for the corresponding hour, service, and infrastructure components.
   • s2r_observed_topology: Records the observed link between frontend and backend virtual servers.
4. Curate Good Example: This step only runs for successful traffic. A transaction is considered a failure if the HTTP status is 4xx/5xx or if the response body contains a SOAP Fault.
   • An INSERT ... ON CONFLICT statement is executed to save the transaction to s2r_generic_learned_example. If an example with the same signature already exists, its seen_count is incremented.
   • The service then enforces a cap (LEARNED_EXAMPLE_CAP) on the number of examples stored per service and operation, deleting the oldest ones to stay within the limit.
5. Learn Leaf Fields: If the payload is SOAP, the service extracts individual leaf fields and their values from the request and response bodies. It upserts this data into s2r_generic_learned_field_value, which collects a sample set of observed values for each field.

Data Persistence Example

The service fetches new log entries using a query that respects the watermark and a commit safety lag.

select id, event_id, gateway_type, connector_instance_id, event_timestamp,
       frontend_uri, backend_uri,
       front_virtual_server, back_virtual_server,
       client_ip::text as client_ip_text,
       http_method, http_status,
       request_payload, response_payload,
       request_headers, response_headers
  from s2r_generic_request_log
 where id > ?
   and received_at <= now() - make_interval(secs => ?)
 order by id asc
 limit ?

The parameters correspond to the current lastProcessedId watermark, COMMIT_LAG_SECONDS, and BATCH_SIZE.

Configuration

The service's scheduler can be configured via the following application properties.

Internal Limits and Behavior

The service operates with several hardcoded limits that define its behavior. These are not externally configurable.

API Reference

The service exposes public methods primarily for scheduling and testability.

runScheduled()

This is the main entrypoint invoked by the Spring scheduler. It wraps the core logic in a re-entrancy guard and top-level exception handling. It takes no parameters and has no return value.

runOnce()

This method executes a single pass of the curator logic: it attempts to acquire the lock, fetches and processes one batch of rows, and advances the watermark. It is public for testability and manual invocation.

• Returns: int – The number of log rows scanned in the batch. This is not the same as the number of examples inserted, as many rows may be skipped.

Troubleshooting

The service logs its state and any issues encountered. The following log messages can help diagnose operational problems.

See Also

• Generic Ingest Service: The upstream component (s2r-generic-ingest) responsible for receiving logs from various sources and populating the s2r_generic_request_log table that this service reads from.
• F5 Traffic Learning: A parallel service that performs similar traffic curation functions specifically for logs originating from F5 BIG-IP systems.

All SOAP-to-REST docs