Generic HTTP Receiver for Traffic Ingestion

Overview

The generic HTTP receiver provides a vendor-neutral ingest path for traffic-driven discovery. It allows any API gateway, load balancer, or edge appliance capable of emitting HTTP traffic logs to send that data to the platform. This "bring-your-own-gateway" approach complements the dedicated, vendor-specific connectors.

The receiver exposes a single HTTP endpoint that accepts transaction data in a standardized JSON format. Once ingested, this traffic data flows into the same downstream pipeline as data from dedicated connectors. It is processed by the same aggregation, discovery, and learned example generation systems.

This component is ideal for integrating traffic sources for which no dedicated connector exists. It can be deployed as a public-facing service or as a local container within a private network, similar to the deployment pattern for dedicated connector relays.

How It Works

The data flow begins when an API publisher's edge device sends an HTTP POST request containing a JSON envelope for each transaction to the /ingest/v1/generic/traffic endpoint.

The receiver is designed for high-volume, reliable ingestion and follows these steps for each received event:

1. Authentication: The receiver first verifies the sender's credentials using HTTP Basic authentication.
2. Parsing: It attempts to parse the request. If a vendor-specific parser is specified (via header or query parameter), that parser is used to transform the native log format into the canonical JSON envelope. Otherwise, the body is parsed directly as a canonical envelope.
3. Validation: The receiver validates all fields in the JSON envelope against the required schema. If any validation errors occur, the original request body is written to a dead-letter queue (DLQ) for later inspection, and a 200 OK is returned to the client to prevent retries of a malformed event.
4. Deduplication: It checks if the event_id has been seen before. Duplicates are acknowledged and discarded to prevent double-counting.
5. Stale Event Check: It checks the timestamp_iso. Events older than 24 hours are considered stale, acknowledged, and silently dropped.
6. Persistence: If the event is valid, new, and not stale, it is written to the unified traffic-log table.
7. Downstream Processing: From the unified log table, the data is picked up by the aggregation and discovery pipeline to identify services and generate learned examples.

This process ensures that misbehaving senders do not retry indefinitely and that no valid record is silently lost.

The following diagram illustrates the ingestion flow and decision logic:

flowchart TD
    A[API Publisher Edge] -- "POST JSON Envelope" --> B(Ingest Endpoint);

    subgraph "Receiver Processing"
        B --> C{Validate & Parse};
        C -- "Valid" --> D{Is event_id a duplicate?};
        C -- "Invalid" --> G[Write to Dead-Letter Queue];

        D -- "No" --> E{Is timestamp > 24h old?};
        D -- "Yes" --> H[Acknowledge Duplicate];

        E -- "No" --> F[Write to Unified Log Table];
        E -- "Yes" --> H;
    end

    subgraph "Downstream"
        F --> I[Aggregation & Discovery];
    end

    subgraph "Response to Caller"
        G -- "Prevents retry" --> R1[Return 200 OK];
        H -- "Prevents retry" --> R1;
        F -- "Success" --> R1;
        B -- "Oversized Payload" --> R2[Return S2R-INGEST-0413];
        B -- "Internal Server Error" --> R3[Return S2R-INGEST-0500];
    end

Ingest Endpoint

The receiver exposes a single endpoint for ingesting all traffic data.

POST /ingest/v1/generic/traffic

This endpoint accepts a single traffic event formatted as a JSON envelope.

Authentication

Requests must be authenticated using HTTP Basic authentication. The credential is set at deploy time.

Vendor-Specific Parsers

For edge devices that emit logs in a native format rather than the canonical JSON envelope, you can specify a per-vendor parser to normalize the data. The receiver will route the raw request body through the specified parser.

Select a parser using one of the following methods:

• HTTP Header (recommended): X-S2R-Vendor: <gateway_type>
• Query Parameter: ?parser=<gateway_type>

The X-S2R-Vendor header takes precedence if both are provided. If neither is present, the receiver assumes the request body is already in the canonical JSON envelope format.

The recognized gateway_type values are: f5, datapower, citrix_netscaler, haproxy, nginx, kemp, a10, radware, layer7, oracle-oag, webmethods, akana, tibco_mashery, and generic. Use generic when sending the canonical envelope directly or when no other type fits.

Responses

• 200 OK: The event was successfully accepted, or it was a malformed/duplicate/stale event that was handled and should not be retried.
• S2R-INGEST-0413: The request body is larger than the 30 MB limit.
• S2R-INGEST-0500: A genuine server-side error occurred (e.g., database is unreachable). The client should retry the request.

JSON Envelope Schema

All requests to the ingest endpoint must have a Content-Type of application/json and a body conforming to the following schema. The total POST body size is capped at 30 MB.

Example Payload

The following is an example of a minimal valid JSON envelope.

{
  "envelope_version": "1",
  "gateway_type": "generic",
  "event_id": "tx-2026-05-25-0001",
  "timestamp_iso": "2026-05-25T08:00:00Z",
  "frontend_uri": "/SoapService/services/Charge",
  "backend_uri": "http://<BACKEND_IP>:9080/SoapService/services/Charge",
  "client_ip": "<CLIENT_IP>",
  "gateway_client_ip": "<GATEWAY_HOP_IP>",
  "front_virtual_server": "soap-vip-payments",
  "http_method": "POST",
  "http_status": 200,
  "latency_ms": 42
}

Troubleshooting

The receiver is designed to be resilient and provide clear feedback to both clients and operators.

See Also

• Connectors Overview
• Traffic-Driven Discovery
• Learned Examples

All SOAP-to-REST docs