Raw Log Archiving and Retention Service

Overview

The Raw Log Archive Service is an automated, background pipeline responsible for managing the lifecycle of raw API transaction logs and related data. It implements a tiered retention strategy to balance data availability for recent events with the cost-effectiveness of long-term storage.

The service performs three primary functions on independent schedules:

1. Archiving: Moves raw log payloads from the primary database to a configured object store for long-term retention. This applies to s2r_f5_request_log and s2r_debug_trace tables. Once archived, the large payload columns in the database are nulled to reclaim space.
2. Expiry: Permanently deletes log archives from the object store and their corresponding pointer rows from the database after a long-term retention period has passed.
3. Pruning: Deletes aged data from various aggregation and reporting tables to keep them at a manageable size.

The service is designed to run automatically based on a configured schedule. It can also be triggered manually via an API for operational purposes. Its behavior is primarily controlled by system settings that define retention periods and storage modes.

How it Works

The service is composed of three distinct, scheduled jobs that operate independently. Concurrency locks (AtomicBoolean) prevent multiple instances of the same job from running simultaneously. If a scheduled job is triggered while a previous run is still active, the new run is skipped.

1. Log Archiving

The archiving process runs every 15 minutes by default. Its goal is to move log payloads older than a configured database retention period (raw_log_db_retention_days) out of the primary database. The behavior depends on the raw_log_storage_mode system setting.

flowchart TD
    subgraph Archiving Process
        direction LR
        Start(Start Archive Run) --> CheckMode{Storage Mode?}

        CheckMode -->|'db_and_gcs'| CheckObjectStore{Object Store Enabled?}
        CheckMode -->|'db_only'| DeleteOnlyPath(Delete from DB)

        CheckObjectStore -->|Yes| ArchivePath(Archive to Object Store)
        CheckObjectStore -->|No| DeleteOnlyPath

        subgraph ArchivePath
            direction TB
            A1[Select due rows from s2r_traffic_log] --> A2[For each row] --> A3{Archive Succeeded?}
            A3 -->|Yes| A4[Null DB payloads, set gcs_uri]
            A3 -->|No| A5[Increment archive_attempts]
            A5 --> A6[End Row]
            A4 --> A6
        end

        subgraph DeleteOnlyPath
            direction TB
            D1[DELETE from s2r_traffic_log] --> D2[DELETE from s2r_debug_trace]
        end

        DeleteOnlyPath --> SweepLogs(Sweep Generic & DLQ Logs)
        ArchivePath --> SweepLogs
        SweepLogs --> End(End Run)
    end

Storage Modes:

• db_and_gcs (Default): This is the tiered storage mode.

  1. The service identifies rows in s2r_traffic_log and s2r_debug_trace older than raw_log_db_retention_days.
  2. For each row, it constructs a JSON object containing the log data.
  3. The JSON payload is compressed with GZIP.
  4. The compressed data is uploaded to the configured object store. The object key is structured based on the log type, date, and a unique identifier.
     • f5-request-log/{instance}/{yyyy/MM/dd}/{id}.json.gz
     • debug-trace/{yyyy/MM/dd}/{correlation_id}/{id}.json.gz
  5. Upon successful upload, the database row is updated: the gcs_uri column is set to the object key, archived_at is set to the current timestamp, and the large payload columns (request_payload, response_payload, etc.) are set to NULL.
  6. If an upload fails, the archive_attempts counter on the row is incremented. A row will no longer be considered for archiving after 5 failed attempts (MAX_ARCHIVE_ATTEMPTS).

• db_only: In this mode, logs are not sent to object storage. Rows in s2r_traffic_log and s2r_debug_trace older than raw_log_db_retention_days are deleted directly.

s2r_traffic_log Archiving Strategy

To avoid performance issues on large tables, the service uses an intelligent "walking window" strategy to find archivable rows in s2r_traffic_log.

• Cursor: The process maintains a time-based cursor (f5ArchiveCursor) marking its position in the table. On startup, the cursor is initialized to 1 day before the most recently archived log, preventing a full table scan of already-processed history.
• Windowed Scans: The service scans for logs in small, time-bounded windows (f5ArchiveWindow) starting from the cursor.
• Dynamic Window Sizing: If a query within a window times out, the window size is halved to reduce query complexity. If a window contains no archivable rows, the cursor advances past it, and the window size may be doubled for the next attempt, up to a maximum of one day.

Generic Log and DLQ Sweeps

As part of the archive run, the service also prunes two other tables:

• s2r_generic_request_log: Deletes rows older than generic_raw_log_retention_days (default 60). Deletions are performed in small chunks (500 rows) to avoid long-running transactions.
• s2r_generic_ingest_dlq: Deletes rows older than a fixed 7 days. This also uses a chunked delete approach.

2. Archive Expiry

The expiry process runs every 6 hours by default. It is responsible for enforcing the long-term retention policy (raw_log_gcs_retention_days) on the data in object storage.

1. The service queries for rows in s2r_traffic_log and s2r_debug_trace that were archived longer ago than the configured retention period.
2. It processes these rows in pages (100 rows per page, up to 5000 per run).
3. For each identified row: a. It deletes the corresponding object from the object store using the key stored in the gcs_uri column. b. After the object is successfully deleted, it deletes the pointer row from the database.
4. The process continues page by page until no more expired rows are found or the per-run limit is reached.

3. Aggregation Pruning

The aggregation pruning process runs daily by default. It cleans up historical data from summary tables to keep them performant.

1. It reads the aggregation_retention_days setting (default 365 days).
2. It issues DELETE statements against the following tables for rows older than the retention period:
   • s2r_observed_traffic_hourly
   • s2r_observed_topology
   • s2r_f5_traffic_hourly
3. It also prunes s2r_spec_refinement_sample using a fixed 30-day retention period.

Manual Triggers

The service exposes API endpoints that allow for manual triggering of the three main jobs. These are useful for operational tasks or immediate cleanup. A request to trigger a job that is already running will be skipped.

runArchive()

Manually triggers a single run of the log archiving and generic log sweeping process.

• Response Body: A JSON object detailing the outcome of the run.

Example Response:

{
    "status": "DONE",
    "storageMode": "db_and_gcs",
    "usedGcs": true,
    "dbRetentionDays": 14,
    "f5LogsArchived": 487,
    "f5LogsDeleted": 0,
    "debugTracesArchived": 12,
    "debugTracesDeleted": 0,
    "genericRawLogRetentionDays": 60,
    "genericRawLogsDeleted": 1530,
    "genericIngestDlqRetentionDays": 7,
    "genericIngestDlqDeleted": 55,
    "time": "2023-10-27T10:00:00.123456Z"
}

runGcsExpiry()

Manually triggers a single run of the object store archive expiry process.

• Response Body: A JSON object summarizing the number of expired items.

Example Response:

{
    "status": "DONE",
    "gcsRetentionDays": 365,
    "f5LogsExpired": 500,
    "debugTracesExpired": 23,
    "time": "2023-10-27T12:00:00.987654Z"
}

runAggregationPrune()

Manually triggers a single run of the aggregation table pruning process.

• Response Body: A JSON object summarizing the number of deleted rows from each table.

Example Response:

{
    "status": "DONE",
    "aggregationRetentionDays": 365,
    "deletedObservedTrafficHourly": 1205,
    "deletedObservedTopology": 88,
    "deletedTrafficHourly": 2400,
    "specRefinementSampleRetentionDays": 30,
    "deletedSpecRefinementSamples": 500,
    "time": "2023-10-28T01:00:00.555555Z"
}

Configuration

The service's behavior is controlled by a combination of application properties (for scheduling) and dynamic system settings (for retention policies).

System Settings

These settings are stored in the s2r_system_setting table and can be modified at runtime.

Application Properties

These properties configure the scheduler for the service's tasks.

Environment Variables

The object store bucket name is configured via the ObjectStore adapter, which reads one of the following environment variables.

• S2R_RAW_ARCHIVE_BUCKET
• S2R_OBJECT_STORE_BUCKET

Troubleshooting

All SOAP-to-REST docs