Traffic Derivation Core for SOAP and REST Analysis

Overview

The TrafficDerivationCore is a collection of static, side-effect-free helper methods for analyzing and normalizing observed API traffic. It provides a canonical, vendor-agnostic implementation for deriving structured information from raw SOAP and REST traffic data.

This component is used by traffic learning services to process captured events. It takes inputs such as request payloads, URIs, and backend connection details, and produces standardized outputs like service keys, operation names, and payload classifications. By centralizing this logic, it ensures that different traffic sources produce consistent, comparable analysis results.

The scope of TrafficDerivationCore is deliberately limited to pure, stateless functions. It does not access databases, instance-specific state, or perform network resolution. Its sole responsibility is to transform the provided inputs into derived, canonical forms.

How it Works

The component provides several key transformation and classification primitives.

SOAP Operation Name Resolution

A primary function is to extract the operation name from a SOAP payload. Because captured traffic can often be malformed, the process uses a two-phase, strict-then-lenient strategy.

1. Strict Parse: The resolveSoapOperationNameStrict method first attempts a full, namespace-aware XML DOM parse of the payload. It disables DOCTYPE declarations to prevent XXE vulnerabilities. If the payload is well-formed XML, it navigates to the SOAP Body element and returns the local name of its first child element.
2. Lenient Fallback: If the strict parse fails for any reason (e.g., malformed XML, non-XML payload), the resolveSoapOperationName method calls resolveSoapOperationNameLenient. This fallback uses regular expressions to find the SOAP Body tag and then extracts the name of the first element tag that follows. This approach is resilient to common formatting errors, such as missing spaces between attributes, and ensures an operation name can be recovered even from non-well-formed payloads.

This flow is depicted below:

flowchart TD
    A[SOAP Payload] --> B{Attempt Strict XML Parse};
    B --> C{Parse Successful?};
    C -- Yes --> D[Extract Operation from DOM];
    D --> F[Return Operation Name];
    C -- No --> E{Attempt Lenient Regex Scan};
    E --> G{Body/Child Tag Found?};
    G -- Yes --> H[Extract Operation from Regex];
    H --> F;
    G -- No --> I[Return Empty String];
    I --> Z[End];
    F --> Z;

Canonical REST Signature Generation

For bodyless REST calls (e.g., GET, HEAD), the canonicalRestSignature method creates a stable, unique signature that ignores transient values in query parameters. This is crucial for correctly identifying unique API operations.

The process is as follows:

1. The HTTP method is normalized to uppercase (e.g., get becomes GET).
2. The URI fragment (#...) is removed.
3. The URI is split into its path and query string components.
4. The query string is parsed to extract only the parameter names. All parameter values are discarded.
5. The list of parameter names is sorted alphabetically.
6. The final signature is constructed as METHOD path?name1&name2....

This ensures that calls like GET /items?user=123&session=abc and GET /items?session=xyz&user=456 are both mapped to the same canonical signature: GET /items?session&user.

Payload Classification

The classifyRequestPayload method performs a fast, preliminary classification of a request payload's format. It uses inexpensive string checks to categorize the payload as SOAP, XML, JSON, or NONE (for empty or unrecognizable formats). The checks are ordered to ensure correctness; for example, a payload is checked for SOAP before XML because a SOAP envelope is a specific type of XML document. This initial classification allows callers to gate more expensive parsing operations.

Examples

Computing a Service Backend Key

The computeServiceBackendKey method generates a canonical string to identify a backend service.

Generating a Canonical REST Signature

The canonicalRestSignature method normalizes a REST call into a stable signature by sorting parameter names and discarding values.

// Example Invocation
String signature = TrafficDerivationCore.canonicalRestSignature(
    "GET",
    "/users/search?region=us-west&status=active&region=us-east#results"
);
// signature is "GET /users/search?region&status"

The resulting signature is GET /users/search?region&status. Note that the fragment (#results) is removed, parameter values are discarded, and the unique parameter names (region, status) are sorted alphabetically.

API Reference

This component exposes a set of public static methods and an enum for payload classification.

PayloadKind Enum

This enum classifies the format of a request payload. It is persisted to identify the shape of a learned example.

Methods

classifyRequestPayload

static PayloadKind classifyRequestPayload(String payload)

Classifies the request payload into a PayloadKind using cheap prefix and substring tests. The order of checks is SOAP, then JSON, then XML. An empty, null, or unrecognized payload results in PayloadKind.NONE.

Returns: The PayloadKind enum member corresponding to the detected payload format.

computeServiceBackendKey

static String computeServiceBackendKey(String scheme, String host, Integer port)

Computes a canonical backend key of the form scheme://host[:port].

Returns: The canonical backend key string, or an empty string if host is null or blank.

canonicalRestSignature

static String canonicalRestSignature(String requestMethod, String uri)

Computes a stable signature for a REST call by normalizing the method and URI, discarding query parameter values, and sorting parameter names.

Returns: A canonical signature string, e.g., GET /path?param1&param2.

resolveSoapOperationName

static String resolveSoapOperationName(String payload)

Extracts the SOAP operation name from a payload. It first attempts a strict XML parse and, if that fails, falls back to a lenient regular-expression-based scan.

Returns: The local name of the first element inside the SOAP Body, or an empty string if it cannot be determined.

resolveXmlRootName

static String resolveXmlRootName(String payload)

Extracts the local name of the root element from a plain XML document using a substring scan. This is intended for non-SOAP XML payloads.

Returns: The local name of the root element, or an empty string if it cannot be determined.

looksLikeSoap

static boolean looksLikeSoap(String payload)

Performs a fast, case-insensitive substring check to determine if a payload appears to be a SOAP envelope. It checks for the presence of :envelope or <envelope.

Returns: true if the payload likely contains a SOAP envelope, false otherwise.

looksLikeSoapFault

static boolean looksLikeSoapFault(String payload)

Performs a fast, case-insensitive substring check to determine if a payload appears to be a SOAP Fault. It checks for various fault-related tags like <Fault>, <faultcode>, and their prefixed variants.

Returns: true if the payload likely contains a SOAP Fault, false otherwise.

isBodylessMethod

static boolean isBodylessMethod(String requestMethod)

Determines if an HTTP method typically does not have a request body.

Returns: true for GET, HEAD, DELETE, and OPTIONS; false otherwise.

sha256Hex

static String sha256Hex(String value)

Computes the SHA-256 hex digest of a string, encoded as UTF-8. Used for creating stable hashes for deduplication.

Returns: The lowercase hex-encoded SHA-256 hash of the input.

extractLeafFields

static List<SoapPayloadExtractor.LeafField> extractLeafFields(String payload, int maxLeaves)

Extracts a flat list of leaf-field samples from a SOAP payload. This method is a pass-through to the SoapPayloadExtractor component.

Returns: A list of LeafField objects.

Other Methods

The source also contains resolveSoapOperationNameLenient, findChildByLocalName, and resolveSoapOperationNameStrict, which are implementation details of the public-facing methods described above.

Troubleshooting

Malformed SOAP Payloads

A significant portion of observed SOAP payloads may not be well-formed XML. A common issue is a missing space between attributes in the envelope tag, often due to fragile upstream logging mechanisms.

Example of malformed XML:

<soap:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/"xmlns:wsu="..."
                                                                 ^ no space

The resolveSoapOperationName method is designed to handle this. It automatically falls back from a strict DOM parser to a more lenient regex-based scanner, allowing it to extract the operation name even from such malformed payloads. The strict parser is configured to suppress console errors for these expected failures.

Hashing Failures

The sha256Hex method relies on the standard SHA-256 MessageDigest provider in the Java environment. If this algorithm is not available for any reason, the method will throw an IllegalStateException, which is a fatal error for the calling process.

See Also

• Traffic Learning Services: The primary consumers of this component, which use its helpers to process observed traffic.
• SOAP Payload Extraction: The SoapPayloadExtractor component provides deeper payload analysis, including the leaf-field extraction used by extractLeafFields.

All SOAP-to-REST docs