Data Retention and Sweeps
Details the platform's data retention policies, default periods, and configuration for runtime logs, audit trails, metrics, and raw traffic.
Overview
The platform retains several classes of operational data for a configurable, finite period. This includes runtime logs, ingested traffic, metrics aggregations, and the audit trail. After the retention period expires, a background process sweeps and deletes the old data.
This mechanism ensures that data storage requirements remain bounded while keeping operational data accessible for a useful window. All retention processes run within the API publisher's own database infrastructure; no data is exported to an external service for retention.
The primary retention period is controlled by a system-wide setting, default_retention_days, but several specific data classes have their own configurable retention periods.
How it Works
The Sweep Mechanism
Data retention is managed by a background process running in the worker service. This process periodically executes sweeps to delete data that has aged past its configured retention window.
To prevent long-running database locks that could impact system performance, sweeps delete data in bounded chunks. This ensures that even large cleanup operations do not cause significant contention on database tables. The exact queries and schedules for each data type are defined internally.
Drill-Down vs. Aggregated Data
The platform distinguishes between two fundamental types of data, each with a different retention strategy:
- Raw Logs: These are high-volume, per-event records used for detailed analysis and debugging (e.g., "show me exactly what happened for a specific correlation ID"). This includes raw ingested traffic and debug traces. Raw logs are typically kept in the primary database for a short period (e.g., 14 days) for immediate drill-down access. For longer-term analysis, they can be archived to object storage.
- Aggregation Tables: These are compact, pre-computed rollups (e.g., hourly metrics) that power dashboards, key performance indicator (KPI) reports, and service-level objective (SLO) calculations. Because they are much smaller than raw logs, they are retained for a much longer period (e.g., 365 days) to enable long-range trend analysis. Dashboards query these aggregation tables directly for fast performance.
This two-tiered approach allows for both fast, long-term trend analysis via dashboards and detailed, short-term debugging via raw logs.
Data Residency
All data subject to retention policies resides within the API publisher's own infrastructure. This includes the primary database where operational data is stored and, if configured, an object storage bucket for cold archives. The retention process is a sweep over data you already hold; no data is exported from your network as part of this process.
Data Retention Periods
The following table summarizes the default retention periods for different classes of data and whether they are configurable.
| Data | Default Retention | Configurable | Notes |
|---|---|---|---|
| Runtime logs (per-call records) | default_retention_days (60 days) | Yes | Cleaned up by the configured retention-days sweep. |
| Audit trail (every config change) | default_retention_days (60 days) | Yes | Records who/what/before-after for each change, with a correlation ID. |
| Raw ingested traffic (gateway/connector feed) | 14 days in the database; up to 365 days when archived to object storage | Yes | A short database window is maintained for drill-down; a longer cold archive can be kept in your own storage bucket. |
| Debug traces | 14 days in the database; up to 365 days in object storage | Yes | Follows the same archive model as raw traffic. |
| Generic-HTTP receiver raw log | 60 days | Yes | Deleted by a bounded-delete sweep. |
| Ingest parse-failure queue (DLQ) | 7 days | No | This is a product invariant. Stale parse failures are considered to have low diagnostic value after one week. |
| Metric aggregation tables (hourly rollups, topology) | 365 days | Yes | These are the long-lived, pre-computed time-series that power dashboards. |
Configuration
Retention periods for most data types are configurable. The primary setting is default_retention_days, which governs runtime logs and the audit trail.
Configuration Reference
| Name | Type | Default | Required | Description |
|---|---|---|---|---|
default_retention_days | integer | 60 | No | The default retention period in days for runtime logs and the audit trail. |
How to Change Retention Settings
Configurable retention windows can be modified through the administration UI or via the API.
- Navigate to Settings → Retention in the UI.
- Modify the desired retention values.
Alternatively, you can use the /admin/v1/settings API endpoint. Changes made via either method are recorded in the audit trail and take effect on the next scheduled sweep.
Note: Lengthening a retention window will cause data to be kept for a longer period, increasing storage usage. Shortening a window will cause the next sweep to delete any data that is now older than the new, shorter window.
API Reference
The system settings, including default_retention_days, can be managed via the following administrative API endpoints.
| Method & Path | Description |
|---|---|
GET /admin/v1/settings | Reads the current system settings. |
PATCH /admin/v1/settings | Modifies one or more system settings, including retention periods. The request body format is not specified in the source. |
The source does not specify the exact request or response payloads for these endpoints.
Troubleshooting and Considerations
- Shortening Retention Periods: When you shorten a retention period, the next background sweep will permanently delete all data that is now older than the new setting. This action is irreversible.
- Fixed Retention for DLQ: The 7-day retention period for the Ingest parse-failure queue (DLQ) is a fixed product invariant and cannot be changed. This is because the diagnostic value of stale parsing failures diminishes rapidly.
- Internal Tables: A small number of internal sampling tables also have fixed retention periods that are not operator-tunable.