System Setting Secret Encryption Migration

Describes the automatic, one-time process that encrypts plaintext system setting secrets at rest. Explains the trigger, scope, and fail-soft behavior.

Overview

The SettingSecretEncryptionMigrator is an internal system component that performs a one-time, idempotent migration to encrypt sensitive system settings that were previously stored as plaintext in the database. This process runs automatically on application startup to enhance security by ensuring secrets are encrypted at rest.

This migration targets the following specific system setting keys in the s2r_system_setting table:

f5_password
alert_smtp_password
f5_ingest_token

In addition to encrypting the current values, the migrator also scrubs these same secrets from historical audit logs (system_setting audit rows) where they may have been recorded in plaintext.

The component is designed to be robust and fail-soft. It runs for each tenant in a multi-tenant environment. If a migration fails for any reason, it does not interrupt system operation, and the process is automatically retried on the next application startup.

How It Works

The migration logic is encapsulated in the SettingSecretEncryptionMigrator component, which is triggered once the application is fully initialized.

flowchart TD
    A[Application Startup] --> B(ApplicationReadyEvent Fired);
    B --> C{SettingSecretEncryptionMigrator};
    C --> D{Iterate Through All Tenants};
    subgraph For Each Tenant
        direction LR
        D --> E[Encrypt Plaintext Secrets];
        D --> F[Scrub Secrets from Audit Logs];
    end
    E --> G((Log Encryption Results));
    F --> H((Log Scrubbing Results));

Trigger: The process begins when the application fires an ApplicationReadyEvent, signaling that it is ready to service requests. The SettingSecretEncryptionMigrator listens for this event and calls its migrate() method.
Tenant Iteration: The migration operates on a per-tenant basis. It uses the PerTenantJobRunner to execute the migration logic once for every tenant configured in the system. In a single-tenant installation, it runs exactly once for the default tenant.
Secret Encryption: For each tenant, the migrator calls systemSettingRepository.encryptLegacyPlaintextSecrets().
- This method identifies the target secret settings (f5_password, alert_smtp_password, f5_ingest_token) that are stored in plaintext.
- It uses the key from the S2R_CREDENTIAL_KEY environment variable to encrypt the values.
- The process is idempotent; it checks if a value is already encrypted and skips it if so. This ensures that running the migration multiple times, or concurrently across multiple application replicas, has no adverse effects.
Audit Log Scrubbing: Concurrently within the same tenant pass, the migrator calls auditRepository.scrubLegacySecretSettingValues().
- This method finds historical audit records containing the full system settings map and removes the plaintext secrets.
- This scrubbing action is also idempotent and does not require the encryption key. It can succeed even if the encryption step fails.

IMPORTANT

The encryption key is provided exclusively through the S2R_CREDENTIAL_KEY environment variable. This key must be available to the application environment. It is intentionally not used in database-level migration runners (like Flyway) and is never written to SQL.

Fail-Soft and Retry Behavior

The migration is designed to never cause a service outage.

Read Compatibility: System components that read these settings are capable of handling both the legacy plaintext format and the new encrypted format.
Failure Handling: If the encryption or scrubbing process fails for a tenant (e.g., due to a missing encryption key), the migrator logs a warning and proceeds. The failure does not break functionality like F5 integration or SMTP alerts.
Automatic Retry: Any failed migration attempts are automatically retried the next time the application starts.

Configuration

The migration process depends on a single environment variable for its encryption key.

Name	Type	Default	Required	Description
`S2R_CREDENTIAL_KEY`	String	—	Yes	The key used for `pgcrypto` functions to encrypt and decrypt secrets. This variable must be present in the application's environment for the secret encryption to succeed.

Manual Invocation

The SettingSecretEncryptionMigrator is designed to run automatically on application startup. The source does not provide a mechanism for an administrator to trigger this migration manually via an API endpoint or CLI command. The primary public method is migrate(), which is invoked internally by the onReady event listener.

Upon successful execution for a tenant, the application log will contain messages similar to the following:

INFO S2R secret settings: re-encrypted 2 legacy plain-text value(s) tenant=<tenant_id>
INFO S2R secret settings: scrubbed secrets from 5 legacy audit row(s) tenant=<tenant_id>

Troubleshooting

If the migration process encounters an error, it logs a warning and continues, retrying the operation on the next application startup.

Log Message	Condition	Impact
`S2R secret settings: legacy re-encrypt skipped tenant={} ({}); readers still accept legacy values, retried next start`	A `RuntimeException` occurred during the encryption of secrets for a specific tenant. This is often caused by a missing or invalid `S2R_CREDENTIAL_KEY`.	The secrets for that tenant remain in plaintext. System functionality is unaffected because readers support both formats. The migration for this tenant will be retried on the next startup.
`S2R secret settings: legacy audit scrub skipped tenant={} ({}); audit reads still redact, retried next start`	A `RuntimeException` occurred while scrubbing secrets from audit logs for a specific tenant.	Plaintext secrets may remain in historical audit logs for that tenant. The impact is low, as audit log readers are designed to redact these secrets dynamically. The scrubbing operation will be retried on the next startup.
`S2R secret settings: legacy re-encrypt skipped ({}); readers still accept legacy values, retried next start`	A `RuntimeException` occurred at a higher level, preventing the migrator from iterating through tenants.	The entire migration process is aborted for all tenants. The operation will be retried on the next application startup.