Learned Examples for Traffic-Driven Discovery

Learned examples are real request/response pairs captured from production traffic, used to provide ground-truth samples for service onboarding and mapping

Overview

Learned examples are real request/response pairs captured from production traffic at the gateway edge. They are a curated dataset derived from the raw traffic logs that the platform observes for service discovery.

This component solves the problem of creating realistic sample data for API onboarding and testing. Instead of API publishers inventing synthetic payloads, they can use the actual messages that their services exchange in a production environment.

Learned examples serve two primary purposes:

Onboarding: They provide concrete, per-operation sample payloads (the actual XML consumers send and backends return) to work from during the service onboarding process.
Mapping Validation: Because each example pairs a real request with its corresponding real response, it serves as ground truth for validating conversion mappings. This allows API publishers to confirm that generated mappings correctly handle data shapes that actually occur in production, including edge cases that synthetic samples might miss.

Learned examples are available starting from version 1.0 of the platform.

How it Works

Learned examples are an offline, curated dataset derived from raw traffic logs. They are not read directly from the live traffic stream.

The data flow is as follows:

Ingest: Connectors at the gateway edge (such as the F5 connector) capture full transaction artifacts from live traffic. These artifacts are written to a raw traffic log.
Curation: A background curator worker process reads from the raw traffic log. This worker derives the learned-example dataset from the raw log rows.
De-duplication: The curator does not store every transaction. It maintains a representative, de-duplicated set of examples for each service and SOAP operation. When a transaction with a previously seen request hash occurs, the curator increments a seen count for the existing example rather than storing a new, near-identical copy. This ensures the example set remains small, relevant, and efficient to browse.
Storage: The curated learned examples are stored. They are keyed by service and SOAP operation.
Surfacing: The platform's discovery and observability UIs read from this curated dataset to display "good examples" of representative live traffic for a service.

This process runs alongside the creation of aggregation tables that power the main discovery view. Both datasets use the raw log as their source of truth.

Data Capture

For each transaction, the raw traffic log captures the following, provided the connector and edge device support it:

Request and response bodies
Request and response headers
Transaction metadata (e.g., front-end/back-end endpoints, virtual servers, client IP, method, SOAP action, HTTP status, result code, latency)

The learned example's fields mirror what the raw ingest row contained. If a connector's edge does not capture bodies (e.g., some load balancers are configured for URI-level visibility by default), the metadata is still learned, but the body fields in the example will be empty.

The system preserves UTF-8, Hebrew, right-to-left, and gendered-text content end-to-end, ensuring payloads are captured faithfully.

Learned Example Data Structure

A learned example is a data record keyed by service and SOAP operation. It contains the following fields derived from a captured transaction.

Field	Description
Request Payload	The captured request body.
Request Hash	A hash of the request payload, used for de-duplication and grouping.
Response Payload	The captured response body corresponding to the request.
Request Headers	The captured request headers.
Response Headers	The captured response headers.
Request Method	The HTTP method of the request.
SOAP Action	The SOAP action associated with the request.
Front-end and Back-end Identity	Identifiers for the front-end and back-end services involved in the transaction.
`seen count`	An integer counter that is incremented each time a transaction with the same request hash is observed. This allows the most representative messages to be identified.

Troubleshooting

Empty Response Bodies in Examples

If a learned example shows an empty response body for a service that is known to return response data, this is typically a detail of the curation and derivation process, not an indication that the edge is failing to log responses.

To diagnose this, follow these steps:

Find the correlation ID for the transaction in question.
Check the raw traffic log for the row with the same correlation ID.
If the raw log row contains the response body, the data was captured successfully. The gap is downstream of ingest, likely in the curation logic.
If the raw log row is also missing the response body, the issue is with the edge logging configuration.

Note that some operations, such as fire-and-forget or acknowledgement-only patterns, legitimately have no response body. For these, an empty response example is expected and correct.