Synthetic Monitor Service for SOAP Backends

Overview

The Synthetic Monitor Service is an automated background process that executes synthetic monitoring checks against configured SOAP backend services. It periodically sends pre-defined SOAP requests to backend endpoints to verify their availability, correctness, and performance.

This service operates in a multi-tenant environment, ensuring that monitoring jobs for one tenant are isolated and do not interfere with others. The main entry point, runEvaluation, identifies all tenants with enabled monitors and executes the checks for each one independently.

The service reads monitor configurations from the s2r_service_synthetic_monitor table, fetches credentials and mTLS settings from the associated s2r_backend_profile, executes the HTTP request, and records detailed results in the s2r_backend_test_preview table. It also updates the status of the monitor configuration with the outcome of the run.

How it Works

The synthetic monitoring process runs in a series of steps, starting with a top-level evaluation run that iterates through all applicable tenants and their configured monitors.

flowchart TD
    subgraph Synthetic Monitor Job
        A[Start: runEvaluation] --> B{Get tenants with<br>enabled monitors};
        B --> C{For each tenant...};
        C --> D[Run as tenant];
    end

    subgraph Tenant-Specific Evaluation
        direction TD
        D --> E{Find enabled monitors<br>for the tenant};
        E --> F{For each monitor...};
        F --> G{Is service 'active'?};
        G -- No --> H[Skip];
        G -- Yes --> I{Is monitor due to run?};
        I -- No --> H;
        I -- Yes --> J[Execute Check];
    end

    subgraph Check Execution
        direction TD
        J --> K[Perform credential safety check];
        K -- Mismatch --> L[Fail: Withhold credentials];
        K -- Match --> M{Resolve auth & mTLS};
        M --> N[Send HTTP POST to endpoint];
        N --> O{Retry on failure?};
        O -- Yes --> N;
        O -- No --> P{Process result};
    end

    subgraph Result Recording
        direction TD
        P --> Q[Record execution in<br>preview log];
        Q --> R[Update monitor's<br>last-run status];
        R --> S[Add to job summary];
    end

    S --> T{...next monitor};
    F -.-> T;
    T --> F;
    
    H --> T;
    L --> P;

    subgraph Final Summary
        direction TD
        U[Aggregate results<br>from all tenants] --> V[Return job summary];
    end
    
    C -- All tenants processed --> U;

1. Tenant Discovery

The runEvaluation method begins by querying the s2r_service_synthetic_monitor table for all distinct tenant IDs that have at least one monitor with enabled = true. This ensures the job only considers tenants with active monitoring configurations.

2. Tenant-Bound Evaluation

For each tenant identified, the service executes the evaluateBoundTenant method within that tenant's security context. This guarantees that all database queries for monitor configurations, credentials, and result persistence are correctly scoped to the specific tenant. A failure during one tenant's evaluation is logged (S2R-TENANT-JOB-FAIL) and does not stop the process for other tenants.

3. Target Selection and Scheduling

Within a tenant's context, the service fetches all enabled synthetic monitor configurations. For each monitor, it performs two checks before execution:

1. Service Status: The associated service must have a status of active. Monitors for services in other states are skipped.
2. Scheduling (dueState): The monitor must be due to run. A monitor is considered due if either of the following conditions is met:
   • Scheduled: The time since last_run_at exceeds the schedule_hours interval.
   • Idle: The time since the service's last_api_usage_at and the monitor's last_run_at both exceed the idle_after_hours interval.

If a monitor is not due, it is skipped with the reason not_due.

4. Check Execution

If a monitor is due, the execute method is called:

1. Credential Safety Check: Before resolving credentials, the service verifies that the endpoint_url stored on the monitor configuration matches the profile_endpoint_url from the associated backend profile (scheme, host, and port). If they do not match and the monitor requires credentials (via auth_type or mTLS), the execution is aborted to prevent sending secrets to an unintended destination.
2. Authentication and mTLS: The service resolves the required authentication headers and mTLS context based on the backend profile's auth_type, credentials envelope, and mtls_enabled flag.
3. HTTP Request: An HTTP POST request is sent to the monitor's endpoint_url. The request includes standard SOAP headers, any resolved Authorization or custom API key headers, and the configured SOAP payload.
4. Retries: If the request fails, it is retried up to retry_count times, with a delay of retry_backoff_ms between each attempt.
5. Result: A successful execution is one that receives an HTTP 2xx status code. Any other status code or a network exception is considered a failure.

5. Result Recording

After execution, the results are persisted:

• Preview Record: A detailed record of the check is inserted into the s2r_backend_test_preview table. This includes the request and response payloads, HTTP status, latency, and any error messages.
• State Update: The s2r_service_synthetic_monitor table is updated with the last_run_at timestamp, last_status, and last_error_message. If the run was successful, last_success_at is also updated.

Finally, the runEvaluation method returns a summary map containing aggregate counts of all checks performed across all tenants.

Authentication

The service resolves authentication credentials from the credentials envelope stored in the backend profile associated with the monitor. The auth_type field on the monitor configuration dictates which authentication mode to use.

The following auth_type values are supported:

If an auth_type is specified but the corresponding fields are missing from the decrypted credentials envelope, the check will fail with an "auth failed" error.

Credential and mTLS Security

To prevent accidental exposure of secrets, the service enforces a strict endpoint matching rule before attaching any credentials or mTLS certificates to a request.

This check is performed if mtls_enabled is true or if auth_type is anything other than none. The service compares the endpoint_url from the synthetic monitor configuration against the profile_endpoint_url from the associated backend profile.

If the scheme, host, or port of these two URLs do not match, the check is immediately failed, and no request is sent. The result will contain the following error message: credentials withheld: the synthetic monitor endpoint differs from the backend profile endpoint (scheme, host or port). Save the synthetic monitor again so it uses the backend's current endpoint.

A S2R-WRK-SYNTH warning is also logged with the reason endpoint-mismatch.

Configuration Reference

The behavior of each synthetic monitor is defined by fields in the s2r_service_synthetic_monitor table. The service uses the following fields during an evaluation run.

The following fields from the associated s2r_backend_profile are also used:

Execution Results

Job Summary

The runEvaluation method returns a JSON object that summarizes the results of the entire job across all tenants.

{
  "status": "DONE",
  "evaluatedServices": 15,
  "executedChecks": 10,
  "successfulChecks": 8,
  "failedChecks": 2,
  "results": [
    {
      "serviceId": "a1b2c3d4-e5f6-7890-1234-567890abcdef",
      "serviceKey": "my-service",
      "environment": "production",
      "operationId": null,
      "status": "ok",
      "reason": "scheduled",
      "httpStatus": 200,
      "errorMessage": null
    },
    {
      "serviceId": "b2c3d4e5-f6a7-8901-2345-67890abcdef0",
      "serviceKey": "another-service",
      "environment": "production",
      "operationId": null,
      "status": "failed",
      "reason": "idle",
      "httpStatus": 503,
      "errorMessage": "backend returned HTTP 503"
    },
    {
      "serviceId": "c3d4e5f6-a7b8-9012-3456-7890abcdef01",
      "serviceKey": "inactive-service",
      "environment": "staging",
      "operationId": null,
      "status": "skipped",
      "reason": "not_due",
      "httpStatus": null,
      "errorMessage": null
    }
  ],
  "time": "2023-10-27T10:00:00.123456789Z"
}

Preview Records

For each executed check, a detailed record is inserted into the s2r_backend_test_preview table. These records contain the full request and response payloads (trimmed to 250,000 characters), latency, HTTP status, and other diagnostic information. These records are created with a test_type of synthetic-monitor.

Troubleshooting

All SOAP-to-REST docs