Generic HTTP Receiver for Traffic Ingestion

Overview

The Generic HTTP Receiver provides a vendor-neutral ingest path for traffic logs. It allows any edge device capable of sending HTTP requests—such as a gateway, load balancer, application server, or appliance—to feed traffic data into the platform for discovery. This "bring-your-own-gateway" approach complements the dedicated, vendor-specific connectors.

The receiver exposes a single REST endpoint that accepts a standardized JSON payload. Once ingested, this traffic data flows through the same downstream pipeline as data from dedicated connectors. It is processed, aggregated into discovery tables, and used to generate learned examples of API behavior.

This component is ideal for API publishers whose edge infrastructure is not covered by a dedicated connector but can export traffic logs over HTTP.

How It Works

The receiver operates by accepting HTTP POST requests containing traffic data at a single endpoint. The data can be sent in a canonical JSON format or a supported vendor-native format, which the receiver then normalizes.

The processing flow is as follows:

1. An API publisher's edge device sends a single transaction log as an HTTP POST request to the receiver's ingest endpoint.
2. The receiver authenticates the request using HTTP Basic authentication.
3. It inspects the request for a vendor hint (an X-S2R-Vendor header or a parser query parameter) to determine if a specific parser is needed. If no hint is present, it expects the request body to be in the canonical JSON envelope format.
4. The request body is parsed into the canonical envelope structure.
5. The receiver validates the envelope's fields, checks the event_id for duplicates, and verifies the timestamp_iso is not older than 24 hours.
6. Valid, unique, and recent events are written to a unified traffic-log table.
7. From the traffic-log table, the data is aggregated and used by downstream discovery and learning processes.

Invalid, duplicate, or stale events are handled gracefully to prevent senders from retrying indefinitely. See the Error Handling and Validation section for details.

flowchart TD
    A[API Publisher's Gateway] -- HTTP POST --> B{Generic HTTP Receiver};
    subgraph "Platform"
        B -- "1. Authenticate & Parse" --> C{Validate & Deduplicate};
        C -- "Valid Event" --> D[Traffic Log Storage];
        C -- "Invalid/Stale/Duplicate Event" --> E[Acknowledge with 200 OK];
        D -- "Flows to" --> F[Downstream Discovery Pipeline];
    end

API Endpoint Reference

The receiver exposes one endpoint for ingesting traffic data.

Ingest Traffic

POST /ingest/v1/generic/traffic

This endpoint accepts a single traffic event per request. The event can be sent directly as the canonical JSON envelope or as a vendor-native log format that the receiver can parse.

Authentication

Requests must be authenticated using HTTP Basic authentication. The credential is set at deploy time. The receiver performs a constant-time comparison of the provided credential.

Request Parameters

Responses

JSON Envelope Format

When not using a vendor-specific parser, the body of the POST request must be a JSON object conforming to the canonical envelope format.

Envelope Fields

Recognized gateway_type Values

The gateway_type field should be one of the following recognized strings: f5, datapower, citrix_netscaler, haproxy, nginx, kemp, a10, radware, layer7, oracle-oag, webmethods, akana, tibco_mashery, or generic.

Example Envelope

{
  "envelope_version": "1",
  "gateway_type": "generic",
  "event_id": "tx-2026-05-25-0001",
  "timestamp_iso": "2026-05-25T08:00:00Z",
  "frontend_uri": "/SoapService/services/Charge",
  "backend_uri": "http://<BACKEND_HOST_AND_PORT>/SoapService/services/Charge",
  "client_ip": "<CLIENT_IP>",
  "gateway_client_ip": "<GATEWAY_IP>",
  "front_virtual_server": "soap-vip-payments",
  "http_method": "POST",
  "http_status": 200,
  "latency_ms": 42
}

Vendor-Specific Parsers

For edge devices that emit logs in a native format (e.g., syslog or a vendor-specific JSON structure), the receiver can automatically parse the raw body into the canonical envelope. This avoids the need for the sending system to perform the transformation itself.

To select a parser, provide one of the following hints in the request:

• X-S2R-Vendor: <gateway_type> (HTTP Header, takes priority)
• ?parser=<gateway_type> (URL Query Parameter)

The <gateway_type> value must be one of the supported parser types. If no hint is provided, the receiver assumes the request body is already in the canonical JSON envelope format.

The platform includes built-in parsers for: DataPower, Citrix NetScaler, HAProxy, nginx, Kemp, A10, Radware, Layer7, webMethods, Akana, TIBCO Mashery, and Oracle OAG. An identity parser is also available for generic.

Error Handling and Validation

The receiver is designed to be resilient and provide clear feedback without causing cascading failures.

• Validation Failures: If an envelope fails validation, the receiver collects all violations at once, not just the first. The raw request body and the list of errors are written to a dead-letter queue (DLQ) for later inspection. The receiver then returns a 200 OK to the sender to prevent infinite retries of a malformed event.
• Duplicate Events: If an event with a previously seen event_id is received, it is acknowledged with a 200 OK but is not processed again.
• Stale Events: Events with a timestamp_iso older than 24 hours are considered stale. They are silently dropped and the receiver returns 200 OK to signal completion to the sender.
• Oversized Payloads: If the total POST body size exceeds 30 MB, the request is rejected with an S2R-INGEST-0413 error.
• Server-Side Errors: For genuine internal failures, such as a database connection error, the receiver returns an S2R-INGEST-0500 error. This signals to the sender that the error is transient and the request should be retried.

The dead-letter queue preserves the raw body, detected format, parse error, and source IP for each failed event, and its volume is monitored to alert operators of potential sender issues.

See Also

• Connectors Overview
• Traffic-Driven Discovery
• Learned Examples

All SOAP-to-REST docs