Alert Evaluation Service

Overview

The Alert Evaluation Service is a background worker component responsible for monitoring the health of production services. It detects service degradation by evaluating key performance metrics against configured thresholds for error rates, latency, and consecutive failures.

This service operates by periodically querying service metric data from platform logs. When a metric crosses a predefined threshold, the service can trigger an alert, persist the new alert state, and send a notification email to designated recipients. When the metric returns to normal, a corresponding "clear" notification is sent.

The service is designed for a multi-tenant environment. It processes each tenant's services independently, using the tenant's specific configuration for alert thresholds and SMTP notification settings. This ensures that alert evaluation and notification are properly scoped and isolated between tenants.

How it Works

The alert evaluation process is initiated by a call to the runEvaluation method. The service performs a series of steps to identify services to monitor, gather metrics, evaluate alert conditions, and send notifications.

flowchart TD
    subgraph Alert Evaluation Run
        A[Start Evaluation] --> B{Query all active 'prod' services};
        B --> C{Group services by Tenant};
        C --> D{For each Tenant...};
    end

    subgraph "Per-Tenant Processing"
        D --> E[1. Bind Tenant Context];
        E --> F{2. Load Tenant's SMTP & Alert Settings};
        F --> G{3. For each Service in Tenant...};
    end

    subgraph "Per-Service Evaluation"
        G --> H{Is alerting enabled?};
        H -- No --> G;
        H -- Yes --> I[4. Load metrics for 6 alert types];
        I --> J[5. Evaluate metrics vs. thresholds];
        J --> K{State changed? (e.g., OK to Firing)};
        K -- No --> L;
        K -- Yes --> M[6. Send Email Notification];
        M --> L[7. Update Alert State in DB];
        L --> G;
    end

    G --> Z[Collate Results];
    D --> Z;
    Z --> End[Return Summary Report];

Multi-Tenant Processing

The service is built to support a multi-tenant architecture. The evaluation process respects tenant boundaries to ensure data and configuration isolation.

1. Fetch Targets: The service queries the s2r_service table for all active services in the prod environment.
2. Group by Tenant: These services are grouped by their owning tenant_id.
3. Tenant-Scoped Execution: The service iterates through each group. For each tenant, it establishes a TenantContext before processing that tenant's services. This context ensures that all subsequent database lookups for configuration (like SMTP settings and default alert thresholds) are scoped to the current tenant. Platform-wide defaults are used as a fallback.

Per-Service Evaluation

Within a tenant's context, each service is evaluated individually.

1. Load Settings: The service first loads the tenant's effective SMTP settings and default alert thresholds from the s2r_system_setting table. A critical feature is the selectSmtpBlock logic: the system uses either all of a tenant's custom SMTP settings or all of the platform defaults. It never mixes them, preventing a tenant's SMTP host from being used with the platform's credentials, for example.
2. Check if Enabled: The evaluation is skipped for any service where alerting is explicitly disabled.
3. Gather Metrics: For each service, the service queries two main data sources, s2r_runtime_log (runtime metrics) and s2r_traffic_log (F5 network metrics), to gather data for six distinct alert types.
4. Evaluate Conditions: The gathered metrics are compared against thresholds. Thresholds can be configured per-service in s2r_service_alert_config. If not set there, they fall back to the tenant-wide defaults loaded in step 1.
5. Persist State: The evaluateAndPersist method is called for each alert type. It compares the current evaluation result with the last known state stored in the s2r_alert_state table.
6. Send Notifications: If the alert state has changed (e.g., from inactive to active, or vice-versa), the sendMail method is invoked.
   • It determines the recipient list by checking for a service-specific override (smtp_recipients) before falling back to the tenant's default list.
   • If SMTP is enabled for the tenant and recipients are defined, it constructs and sends an email.
   • The mail sender is dynamically configured using the tenant's specific SMTP settings (host, port, credentials).
   • A record of the email attempt (both "sent" and "failed") is written to the s2r_alert_mail_history table for auditing.
7. Update State: The s2r_alert_state table is updated to reflect the new state (is_active), evaluation time, and a JSON blob containing the detailed metrics that led to the result.

Alert Types

The service evaluates six types of alerts for each service, drawing from two different data sources.

Configuration

Alerting behavior is controlled by settings at the tenant and individual service levels. Service-level settings always override tenant-level settings.

Tenant-Level System Settings

These settings are stored in the s2r_system_setting table and are scoped per tenant. They provide the default behavior for all services within that tenant.

Service-Level Alert Configuration

These settings are stored in the s2r_service_alert_config table and apply to a single service, overriding the tenant-level defaults.

API Reference

The service is an internal component and exposes its functionality through a Java method, not a web API endpoint.

runEvaluation() Method

This is the main entry point for the evaluation process. It is called by the platform's scheduling mechanism.

• Signature: public Map<String, Object> runEvaluation()
• Returns: A Map<String, Object> containing a summary of the evaluation run.

Example Return Payload

The returned map provides a high-level summary of the actions taken during the run.

{
  "status": "DONE",
  "smtpEnabled": true,
  "evaluatedServices": 5,
  "triggeredAlerts": 1,
  "clearedAlerts": 0,
  "results": [
    {
      "serviceId": "a1b2c3d4-e5f6-7890-1234-567890abcdef",
      "serviceKey": "my-service-1",
      "serviceName": "My Service 1",
      "consecutiveFailure": {
        "alertType": "prod-consecutive-failures",
        "state": "ok",
        "notified": false,
        "detail": {
          "threshold": 5,
          "observed": 0,
          "lastOccurredAt": null,
          "locations": [],
          "window": "latest_calls"
        }
      },
      "errorRate": {
        "alertType": "prod-error-rate",
        "state": "triggered",
        "notified": true,
        "detail": {
          "thresholdPct": 10.0,
          "observedPct": 15.5,
          "totalCalls": 200,
          "failedCalls": 31,
          "windowMinutes": 5,
          "lastOccurredAt": "2023-10-27T10:00:00Z",
          "locations": []
        }
      },
      "latencyIncrease": { "...": "..." },
      "f5ConsecutiveFailure": { "...": "..." },
      "f5ErrorRate": { "...": "..." },
      "f5LatencyIncrease": { "...": "..." }
    }
  ],
  "time": "2023-10-27T10:00:05.123Z"
}

Troubleshooting

SMTP Password Decryption Failure

The alert_smtp_password setting is stored in an encrypted format.

When decryption fails, the service logs a warning with the code S2R-WRK-0419, and the evaluation proceeds using an empty password. This will likely cause SMTP authentication to fail, which will be recorded in the mail history.

Investigating Notification Failures

If alerts are triggering but notifications are not being received, consult the s2r_alert_mail_history table. This table records every mail delivery attempt, its status (sent or failed), and an error message if the delivery failed. Common causes include:

• alert_smtp_enabled is false for the tenant.
• No recipients are configured at either the service or tenant level.
• Incorrect SMTP server settings (host, port, credentials).
• Network connectivity issues between the worker and the SMTP host.

Database Schema

The Alert Evaluation Service interacts with several key tables in the platform database.

All SOAP-to-REST docs