Raw Log Archiving and Retention Service

Overview

The Raw Log Archive Service is an automated background process responsible for managing the lifecycle of raw request logs and aggregated data. Its primary function is to prevent unbounded database growth by tiering older log data to a configured object store and eventually purging it, according to configurable retention policies.

This service operates through three distinct, scheduled jobs:

1. Archiving: Moves raw log payloads (from s2r_f5_request_log and s2r_debug_trace) from the primary database to an object store.
2. Expiry: Deletes archived logs from the object store and their corresponding pointer rows from the database after a long-term retention period.
3. Pruning: Deletes old, expired data from various aggregation and sample tables.

The service is driven by a combination of application configuration properties and dynamic system settings stored in the database. It can be configured to operate in two modes: archiving to an object store before cleaning the database, or simply deleting logs from the database directly.

How It Works

The service's functions are executed by three independent, scheduled tasks. Each task has its own schedule and can also be triggered manually via an API. Concurrency is managed internally; if a scheduled or manual run is triggered while a previous run of the same task is still in progress, the new run is skipped.

1. Log Archiving

This task runs on a schedule (default: every 15 minutes) to manage raw logs that have exceeded their database retention period. Its behavior depends on the raw_log_storage_mode system setting and whether an object store is configured.

flowchart TD
    subgraph Archiving Process
        direction LR
        A[Start Archiving] --> B{Object Store Mode?};
        B -- "db_and_gcs" --> C[Find logs older than<br/>`raw_log_db_retention_days`];
        C --> D{Batch of 500 rows};
        D -- For each row --> E[Upload Gzipped JSON<br/>to Object Store];
        E -- Success --> F[Null payload columns in DB,<br/>set object key];
        E -- Failure --> G[Increment retry counter<br/>(max 5 attempts)];
        B -- "db_only" or disabled --> H[Delete logs from DB older than<br/>`raw_log_db_retention_days`];
    end

    subgraph Parallel Sweeps
        direction TB
        S1[Start Sweeps] --> S2[Sweep `s2r_generic_request_log`];
        S1 --> S3[Sweep `s2r_generic_ingest_dlq`];
    end

    A --> S1;

Archive Mode (db_and_gcs)

When raw_log_storage_mode is set to db_and_gcs and an object store is enabled, the service performs the following steps:

1. It queries for rows in s2r_traffic_log (an alias for s2r_f5_request_log) and s2r_debug_trace that are older than the raw_log_db_retention_days setting and have not yet been archived.
2. It processes these logs in batches of up to 500 (ARCHIVE_BATCH_SIZE).
3. For each log, it constructs a JSON object containing the log data, compresses it using Gzip, and uploads it to the configured object store. The object key is structured as follows:
   • F5 Request Logs: f5-request-log/<instance_id>/<yyyy>/<MM>/<dd>/<log_id>.json.gz
   • Debug Traces: debug-trace/<yyyy>/<MM>/<dd>/<correlation_id>/<trace_id>.json.gz
4. Upon successful upload, the service updates the database row: it sets the gcs_uri column to the object key, records the current time in archived_at, and nullifies the large payload and header columns (request_payload, response_payload, etc.) to reclaim database space.
5. If an upload fails, the service increments the archive_attempts counter for that row. A row will be retried on subsequent runs until it succeeds or the attempt count reaches 5 (MAX_ARCHIVE_ATTEMPTS).

Delete-Only Mode (db_only)

If raw_log_storage_mode is db_only, or if the object store adapter is not enabled, the service simply deletes rows from s2r_traffic_log and s2r_debug_trace that are older than raw_log_db_retention_days. No data is sent to the object store.

Other Log Sweeps

In parallel with the main archiving flow, this task also cleans two other tables on every run:

• s2r_generic_request_log: Deletes rows older than generic_raw_log_retention_days (default: 60).
• s2r_generic_ingest_dlq: Deletes rows older than a fixed 7 days.

These sweeps are always deletions and are performed in chunks of 500 rows (GENERIC_DELETE_CHUNK_SIZE).

2. Archive Expiry

This task runs on a longer schedule (default: every 6 hours) to permanently delete old archives from the object store.

1. It queries for rows in s2r_traffic_log and s2r_debug_trace whose archived_at timestamp is older than the raw_log_gcs_retention_days setting (default: 365).
2. It processes these rows in pages of 100 (EXPIRY_PAGE_SIZE), with a cap of 5000 (EXPIRY_MAX_PER_RUN) expirations per run to limit impact.
3. For each row, it first sends a delete command to the object store for the object key stored in the gcs_uri column. The service can correctly parse both modern bare object keys and legacy `<REDACTED-BUCKET-URI> URI formats.
4. After successfully deleting the object (or if the object store is disabled), it deletes the corresponding pointer row from the database.

3. Aggregation Pruning

This daily task (default: every 24 hours) cleans up old data from various aggregated data tables to keep them from growing indefinitely.

• It deletes rows from s2r_observed_traffic_hourly, s2r_observed_topology, and s2r_f5_traffic_hourly that are older than the aggregation_retention_days setting (default: 365).
• It deletes rows from s2r_spec_refinement_sample that are older than a fixed 30 days.

Manual Triggers and API

The service's three main tasks can be triggered manually via API endpoints. This is useful for forcing an immediate cleanup or for operational diagnostics. When a task is triggered while another instance of the same task is already running, the new request is skipped.

Run Archive Task

Triggers a single run of the log archiving and sweeping process.

• Service Method: runArchive()
• Success Response: A JSON object detailing the work performed.
• Skipped Response: {"status": "SKIPPED", "reason": "already_running"}

Example Success Response Payload

{
  "status": "DONE",
  "storageMode": "db_and_gcs",
  "usedGcs": true,
  "dbRetentionDays": 14,
  "f5LogsArchived": 500,
  "f5LogsDeleted": 0,
  "debugTracesArchived": 27,
  "debugTracesDeleted": 0,
  "genericRawLogRetentionDays": 60,
  "genericRawLogsDeleted": 1250,
  "genericIngestDlqRetentionDays": 7,
  "genericIngestDlqDeleted": 88,
  "time": "2023-10-27T10:00:00.123456Z"
}

Response Fields

Run Archive Expiry Task

Triggers a single run of the object store expiry process.

• Service Method: runGcsExpiry()
• Success Response: A JSON object detailing the number of expired items.
• Skipped Response: {"status": "SKIPPED", "reason": "already_running"}

Response Fields

Run Aggregation Pruning Task

Triggers a single run of the aggregation table pruning process.

• Service Method: runAggregationPrune()
• Success Response: A JSON object detailing the number of deleted rows from each table.
• Skipped Response: {"status": "SKIPPED", "reason": "already_running"}

Response Fields

Configuration

The service is configured through a mix of application properties (for scheduling) and system settings stored in the s2r_system_setting table (for retention policies).

Application Properties

These properties configure the scheduler's timing and can be set via environment variables or other standard Spring Boot configuration methods.

System Settings

These settings are read from the s2r_system_setting database table and control the retention logic. They can be adjusted at runtime without restarting the application.

Troubleshooting

All SOAP-to-REST docs