Worker Service API Reference

Overview

The Worker Service provides a set of internal HTTP endpoints for orchestrating background tasks and data ingestion. These endpoints are not intended for general API consumer use but are called by schedulers or automation to perform periodic system maintenance and processing.

The service exposes functionality for:

• Health Checks: Verifying liveness, readiness, and database connectivity for the worker and discovery-ingest services.
• Data Maintenance: Triggering cleanup of historical runtime and audit logs, purging expired debug captures, and managing raw log archival.
• Processing Tasks: Initiating alert evaluations, synthetic monitor runs, and F5 configuration synchronization.
• Data Ingestion: Receiving F5 traffic logs for analysis and learning.

The endpoints are implemented in the WorkerController class and interact with various backend services to execute their tasks.

Health Check Endpoints

These endpoints provide status information about the service and its dependencies, such as the database. They are typically used by container orchestration platforms for liveness and readiness probes.

Liveness Probes

Liveness probes indicate that the service process is running.

GET /health/live : Returns the operational status of the main s2r-worker service.

Example Success Response:

{
  "status": "UP",
  "service": "s2r-worker",
  "time": "2023-10-27T10:00:00.123456Z"
}

GET /ingest/health/live : Returns the operational status of the s2r-discovery-ingest service.

Example Success Response:

{
  "status": "UP",
  "service": "s2r-discovery-ingest",
  "time": "2023-10-27T10:00:01.456789Z"
}

Readiness Probes

Readiness probes indicate that the service is ready to accept traffic. These endpoints perform a basic database query to confirm connectivity.

GET /health/ready : Returns the readiness status of the main s2r-worker service, including database connectivity and schema validation status. The overall status will be DOWN if the database query fails.

Example Success Response:

{
  "schema_migrations": "UP_TO_DATE",
  "status": "UP",
  "db": "reachable"
}

GET /ingest/health/ready : Returns the readiness status of the s2r-discovery-ingest service. It is functionally identical to /health/ready but identifies the service differently in its response.

Example Success Response:

{
  "schema_migrations": "UP_TO_DATE",
  "status": "UP",
  "db": "reachable",
  "service": "s2r-discovery-ingest"
}

Task Execution Endpoints

These endpoints trigger specific, long-running background jobs. They are all invoked via POST requests and do not accept a request body.

Data Cleanup and Archival

POST /worker/v1/run-cleanup

Triggers the cleanup of historical structured log data. This endpoint is responsible for deleting records from the s2r_runtime_log and s2r_audit_event tables based on a configured retention period.

How it works:

1. The service determines the data retention period in days by querying the s2r_system_setting table for the default_retention_days key. It uses the setting for the default tenant. If no value is configured or the value is less than 1, it defaults to 60 days.
2. It deletes records from the s2r_runtime_log table that are older than the determined retention period.
3. It calls the s2r_purge_audit_events database function to purge records from the s2r_audit_event table. This function operates with elevated permissions (SECURITY DEFINER) to purge records across all tenants.

   IMPORTANT

   The retention period used for audit events is the greater of the configured retention period and a mandatory minimum of 30 days. This floor (AUDIT_MIN_RETENTION_DAYS) prevents excessive deletion of audit data.

4. The endpoint returns a summary of the deleted records.

Example Success Response:

{
  "status": "DONE",
  "retentionDays": 90,
  "deletedRuntimeLogs": 10520,
  "deletedAuditEvents": 5410,
  "deletedTotal": 15930,
  "note": "raw F5 + debug-trace retention is handled by RawLogArchiveService",
  "time": "2023-10-27T11:00:00.987654Z"
}

POST /worker/v1/run-archive

Triggers the raw log archive service to move data from the primary database to long-term storage.

POST /worker/v1/run-archive-expiry

Triggers the raw log archive service to apply expiry policies to data in long-term storage.

POST /worker/v1/run-aggregation-prune

Triggers the raw log archive service to prune old aggregated data.

POST /worker/v1/run-debug-capture-purge

Triggers the debug capture write service to purge expired debug captures based on their retention policies.

Data Processing and Synchronization

POST /worker/v1/run-alerts

Triggers a full evaluation cycle for both standard alerts and synthetic monitors. The response combines the results from both services.

Example Success Response:

{
  "alertsEvaluated": 15,
  "alertsTriggered": 2,
  "syntheticMonitoring": {
    "monitorsEvaluated": 5,
    "monitorsFailed": 1
  },
  "time": "2023-10-27T12:00:00.112233Z"
}

POST /worker/v1/run-synthetic-monitors

Triggers an evaluation cycle for synthetic monitors only.

POST /worker/v1/run-f5-sync

Triggers the F5 synchronization service to scan for and apply configuration changes from associated F5 devices.

Data Ingestion Endpoints

These endpoints are used to ingest F5 traffic data for processing by the traffic learning service.

F5 Traffic Ingestion

There are two endpoints available for ingesting F5 traffic data. Both are functionally identical and delegate to the same internal processing logic.

• POST /worker/v1/f5/traffic-ingest
• POST /ingest/v1/f5/traffic

These endpoints accept a request body containing the traffic data and optional authentication headers.

Request Parameters

How it works

The following diagram illustrates the request flow for F5 traffic ingestion.

flowchart TD
    subgraph "API Consumer"
        A[F5 Device / Log Shipper]
    end

    subgraph "Worker Service"
        B1["POST /worker/v1/f5/traffic-ingest"]
        B2["POST /ingest/v1/f5/traffic"]
        C{Ingest Logic}
        D[Traffic Learning Service]
    end

    subgraph "Response"
        R_OK[200 OK]
        R_400[400 Bad Request]
        R_403[403 Forbidden]
        R_409[409 Conflict]
    end

    A -- "Sends traffic data" --> B1
    A -- "Sends traffic data" --> B2
    B1 --> C
    B2 --> C

    C -- "Validates and authenticates" --> D
    D -- "Processes data" --> R_OK

    C -- "Invalid arguments" --> R_400
    C -- "Invalid multi-tenant credential" --> R_403
    C -- "Illegal state (e.g., config conflict)" --> R_409

The service passes the request body and authentication details to the F5TrafficLearningService. It includes specific error handling for common failure scenarios.

Responses

• 200 OK: The request was successfully processed. The response body contains the result from the traffic learning service.
• 400 Bad Request: The request was malformed (e.g., invalid data format). This corresponds to an IllegalArgumentException during processing.
• 403 Forbidden: The provided credential is not valid for the target tenant. This occurs when a deployment-wide token is used in a multi-tenant environment, which is not permitted. This corresponds to an IngestTenantBinding.UnboundIngestCredentialException.
• 409 Conflict: The service is in a state that prevents processing the request. This corresponds to an IllegalStateException.

Troubleshooting

F5 Traffic Ingestion Failures

When calling the F5 traffic ingestion endpoints, you may encounter the following errors:

Readiness Probe Failures

If the /health/ready or /ingest/health/ready endpoints return a status of DOWN, it indicates a problem with a critical dependency, most commonly the database. The db field in the response will typically indicate if the database was reachable. Check service logs and database connectivity.

All SOAP-to-REST docs