Traffic Derivation Core for SOAP and REST Analysis

Overview

The TrafficDerivationCore is an internal library of stateless, vendor-agnostic helper methods for analyzing observed API traffic. It provides a canonical implementation for common tasks required by traffic-learning services, such as classifying payload types, generating stable identifiers for services and requests, and extracting specific details from SOAP and XML payloads.

This component solves the problem of code duplication between different traffic-learning curators (e.g., F5TrafficLearningService and GenericTrafficLearningService) by centralizing pure, side-effect-free logic. Its functions are called by these services during the processing of ingested traffic data.

Inputs to these methods are typically raw data from traffic logs, such as request payloads, URIs, and backend connection details. Outputs are standardized classifications, canonical strings, or extracted data used for bucketing, deduplication, and generating learned API examples.

How it works

The TrafficDerivationCore provides utilities that operate in several key areas: payload classification, canonical signature generation, and payload content analysis.

Payload Classification

To determine how to process a request, the system first classifies its payload. The classifyRequestPayload method performs a series of fast, ordered checks on the payload string to assign it a PayloadKind.

This classification runs on every ingested traffic event before any expensive parsing is attempted. The order of checks is critical to ensure correctness. For example, a SOAP envelope is also valid XML, so the SOAP check must occur first.

The classification logic follows this sequence:

flowchart TD
    subgraph Payload Classification Flow
        A[Start: payload string] --> B{Is payload null or empty?};
        B -- Yes --> C[Return PayloadKind.NONE];
        B -- No --> D{Does payload contain ':envelope' or '<envelope'?};
        D -- Yes --> E[Return PayloadKind.SOAP];
        D -- No --> F{Does payload start with '{' or '['?};
        F -- Yes --> G[Return PayloadKind.JSON];
        F -- No --> H{Does payload start with '<'?};
        H -- Yes --> I[Return PayloadKind.XML];
        H -- No --> J[Return PayloadKind.NONE];
    end

The resulting PayloadKind determines how the traffic-learning curator will handle the request.

Canonical Signature Generation

To group and deduplicate observed requests, the system generates stable, canonical signatures.

Service Backend Key

A service_backend_key uniquely identifies a backend service endpoint. The computeServiceBackendKey method constructs this key from a scheme, host, and port. The format is scheme://host[:port], with default ports (80 for http, 443 for https a) omitted. If the scheme is not provided, it is inferred from the port. This ensures that the same backend service is identified consistently, regardless of which traffic source observed it.

REST Request Signature

For bodyless REST calls (e.g., GET), a canonical signature is needed to group similar requests. The canonicalRestSignature method generates this signature by normalizing the HTTP method and URI.

The process discards query parameter values and the URI fragment, then sorts the parameter names alphabetically. This collapses many unique URIs into a single signature representing a distinct call shape.

The normalization process is as follows:

flowchart TD
    subgraph REST Signature Generation
        A[Start: method, uri] --> B[Normalize method to uppercase];
        B --> C[Trim URI and remove fragment (#...)];
        C --> D[Split URI into path and query string];
        D --> E{Query string exists and is not empty?};
        E -- No --> F[Result: "METHOD path"];
        E -- Yes --> G[Split query string by '&'];
        G --> H[For each pair, extract parameter name];
        H --> I[Collect unique, non-blank names];
        I --> J[Sort parameter names alphabetically];
        J --> K[Join names with '&'];
        K --> L[Result: "METHOD path?name1&name2..."];
    end

Example: REST Signature Generation

Given the following request:

• Method: get
• URI: /users/search?name=alice&sort=asc&name=bob#results

The canonicalRestSignature method would produce the following signature:

1. Method is normalized to GET.
2. The URI fragment #results is removed. The URI is now /users/search?name=alice&sort=asc&name=bob.
3. The path is /users/search.
4. The query string is name=alice&sort=asc&name=bob.
5. Unique parameter names are extracted: name, sort.
6. The names are sorted alphabetically: name, sort.
7. The final signature is constructed: GET /users/search?name&sort

Payload Content Analysis

For structured payloads like SOAP and XML, helper methods are available to extract key information.

• SOAP Operation Name: resolveSoapOperationName performs a full, namespace-aware DOM parse of a SOAP payload to find the local name of the first element inside the <soap:Body>. This is used to identify the specific SOAP operation being invoked. To prevent XXE vulnerabilities, DOCTYPE declarations are disabled during parsing.
• XML Root Name: For non-SOAP XML, resolveXmlRootName uses a fast substring scan to find the name of the root element. This is less expensive than a full DOM parse and is used for naming learned examples.
• SOAP Fault Detection: looksLikeSoapFault uses a cheap substring check to quickly determine if a response payload contains a SOAP Fault. This avoids a second expensive DOM parse on the response path.

API Reference

The TrafficDerivationCore exposes the following static methods.

classifyRequestPayload

Classifies a request payload string into a PayloadKind based on its format.

Returns: A PayloadKind enum value: SOAP, XML, JSON, or NONE.

PayloadKind Enum

This enum represents the derived shape of a request payload.

computeServiceBackendKey

Computes the canonical service_backend_key for a backend endpoint.

Returns: A string in the format scheme://host[:port], or an empty string if host is null or blank.

canonicalRestSignature

Generates a stable, canonical signature for a bodyless REST call.

Returns: A normalized string signature in the format METHOD path?param1&param2....

resolveSoapOperationName

Extracts the SOAP operation name from a payload by parsing the document and finding the first element inside the SOAP Body.

Returns: The local name of the SOAP operation element, or an empty string if parsing fails or the operation cannot be found.

resolveXmlRootName

Extracts the local name of an XML document's root element using a fast substring scan.

Returns: The name of the root XML element, or an empty string if it cannot be determined.

looksLikeSoap

Performs a quick, case-insensitive substring check to see if a payload appears to be a SOAP envelope.

Returns: true if the payload contains :envelope or <envelope, false otherwise.

looksLikeSoapFault

Performs a quick, case-insensitive substring check to see if a payload appears to be a SOAP Fault.

Returns: true if the payload contains fault-related tags (e.g., <fault>, <faultcode>), false otherwise.

isBodylessMethod

Checks if a given HTTP method is one where an empty request body is normal.

Returns: true for GET, HEAD, DELETE, and OPTIONS; false otherwise.

sha256Hex

Computes the SHA-256 hex digest of a string.

Returns: A hex-formatted string representing the SHA-256 digest.

extractLeafFields

Extracts a list of leaf-field samples from a SOAP payload. This is a pass-through to the SoapPayloadExtractor component.

Returns: A List of SoapPayloadExtractor.LeafField objects.

findChildByLocalName

Finds the first child Element of a parent Element that has a specific local name.

Returns: The matching Element, or null if not found.

Troubleshooting

See Also

• F5TrafficLearningService
• GenericTrafficLearningService
• SoapPayloadExtractor

All SOAP-to-REST docs