Traffic Derivation Core Utilities

Overview

The TrafficDerivationCore component provides a set of shared, stateless helper methods for analyzing and canonicalizing observed API traffic. It is designed to be a vendor-agnostic utility that centralizes logic previously duplicated across different traffic-learning services, such as F5TrafficLearningService and GenericTrafficLearningService.

This component solves the problem of inconsistent traffic analysis by ensuring that all parts of the system use a single, canonical implementation for common derivation tasks. It functions as a low-level library called by higher-level services. Its inputs are raw traffic artifacts like request payloads and URIs, and its outputs are standardized identifiers like SOAP operation names, REST call signatures, and backend service keys.

The scope of TrafficDerivationCore is deliberately limited to pure, side-effect-free functions. It does not manage state, access databases, or perform network-dependent resolutions. All methods are static.

How it works

The component provides several key capabilities for processing raw traffic data into structured, canonical forms.

SOAP Operation Name Resolution

To identify the operation in a SOAP request, TrafficDerivationCore uses a two-stage process that balances strictness with resilience against malformed payloads.

flowchart TD
    A[SOAP Payload] --> B{Call resolveSoapOperationName};
    B --> C[Attempt Strict XML Parse];
    C -- Success --> D[Return Operation Name];
    C -- Failure/Empty --> E[Attempt Lenient Regex Scan];
    E -- Success --> D;
    E -- Failure/Empty --> F[Return Empty String];

1. Strict Parsing (resolveSoapOperationNameStrict): The system first attempts to parse the payload using a standard, namespace-aware XML DOM parser. This parser is configured to be secure against XML External Entity (XXE) attacks by disallowing DOCTYPE declarations. It identifies the <...:Body> element and returns the local name of its first child element.
2. Lenient Fallback (resolveSoapOperationNameLenient): If the strict parse fails or finds no operation, the system falls back to a lenient, regular-expression-based scan. This is a defensive measure to handle common payload corruptions, such as missing spaces between XML attributes, which can occur during traffic capture. The lenient scan searches for the first element tag following a <...:Body> tag without requiring the entire document to be well-formed XML.

Payload Classification

Before performing deep parsing, the classifyRequestPayload method performs a series of quick checks to determine the general type of a request payload. This allows callers to route the payload to the correct processing logic (e.g., SOAP vs. JSON). The classification follows a specific order of precedence.

flowchart TD
    A[Request Payload] --> B{Is payload blank?};
    B -- Yes --> C[PayloadKind.NONE];
    B -- No --> D{Contains ':envelope' or '<envelope'?};
    D -- Yes --> E[PayloadKind.SOAP];
    D -- No --> F{Starts with '{' or '['?};
    F -- Yes --> G[PayloadKind.JSON];
    F -- No --> H{Starts with '<'?};
    H -- Yes --> I[PayloadKind.XML];
    H -- No --> C;

This process uses inexpensive substring and prefix checks to remain efficient, as it runs for every ingested event.

Canonical REST Signature Generation

For bodyless REST calls (e.g., GET, HEAD), the canonicalRestSignature method generates a stable identifier that groups similar requests. This is necessary because raw URIs often contain unique values in query parameters (e.g., session IDs, timestamps), which would lead to an excessive number of unique "examples" for what is functionally the same API call.

The canonicalization process is as follows:

1. The HTTP method is normalized to uppercase (e.g., get becomes GET).
2. The URI fragment (#...) is removed.
3. The path is separated from the query string.
4. All query parameter values are discarded, leaving only the names.
5. The parameter names are collected, deduplicated, and sorted alphabetically.
6. A final signature is constructed in the format METHOD path?name1&name2....

For example, GET /items?user=123&session=abc and GET /items?session=xyz&user=456 would both resolve to the canonical signature GET /items?session&user.

Canonical Service Backend Key

The computeServiceBackendKey method generates a standardized key for a backend service endpoint in the format scheme://host[:port]. This allows traffic observed from different sources (e.g., F5 logs, generic syslogs) to be correctly attributed to the same backend service.

The key generation logic:

• Infers the scheme (http or https) from the port if the scheme is not provided. Ports 443 and 8443 default to https.
• Omits the port from the final string if it is the default for the scheme (80 for http, 443 for https).
• Returns an empty string if the host is null or blank.

Example: Handling Malformed SOAP

A common issue in captured traffic is malformed XML that causes standard parsers to fail. The source code notes a specific recurring corruption where a space is missing between attributes in the SOAP envelope tag.

Consider the following malformed payload snippet:

<soap:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/"xmlns:wsu="..."
                                                                   ^
                                                                   |-- Missing space

• A strict XML parser, like the one used in resolveSoapOperationNameStrict, would fail to parse this document, and no operation name would be extracted.
• The resolveSoapOperationName method detects this failure and calls the fallback resolveSoapOperationNameLenient.
• The lenient method uses regular expressions to find the <...:Body> tag and then scans for the first subsequent element tag, successfully extracting the operation name without needing to parse the invalid envelope tag.

This ensures that the system can still learn from the traffic example despite the upstream data quality issue.

Methods Reference

The following static methods are available in the TrafficDerivationCore class.

computeServiceBackendKey

Computes the canonical scheme://host[:port] key for a backend service.

Returns: A String representing the canonical backend key. Returns an empty string if host is null or blank.

resolveSoapOperationName

Extracts the SOAP operation name from a request payload. It attempts a strict XML parse and falls back to a lenient regex-based scan on failure.

Returns: A String containing the local name of the operation element, or an empty string if it cannot be determined.

canonicalRestSignature

Generates a stable signature for a REST call by normalizing the method and sorting query parameter names.

Returns: A String in the format METHOD path?param1&param2....

classifyRequestPayload

Classifies a request payload into a PayloadKind based on simple string checks.

Returns: A PayloadKind enum value: SOAP, XML, JSON, or NONE.

looksLikeSoap

Performs a quick, case-insensitive substring check to see if a payload appears to be a SOAP envelope. This is used as a gate before attempting a more expensive parse.

Returns: true if the payload contains :envelope or <envelope.

looksLikeSoapFault

Performs a quick substring scan to check if a response payload contains a SOAP Fault.

Returns: true if the payload contains fault-related tags like <Fault>, <faultcode>, or <faultstring>.

isBodylessMethod

Checks if an HTTP method is one that normally does not have a request body.

Returns: true for GET, HEAD, DELETE, and OPTIONS.

resolveXmlRootName

Extracts the local name of the root element from a plain XML document using a substring scan.

Returns: A String with the root element's local name, or an empty string on failure.

extractLeafFields

Extracts a list of leaf-field samples from a SOAP payload. This is a pass-through to the SoapPayloadExtractor component.

Returns: A List of SoapPayloadExtractor.LeafField objects.

sha256Hex

Computes the SHA-256 hex digest of a string, encoded as UTF-8.

Returns: A String containing the lowercase hex-encoded SHA-256 hash.

PayloadKind Enum

An enum that represents the shape of a learned example's payload.

Troubleshooting

SOAP Operation Name Not Found

If resolveSoapOperationName returns an empty string, it can be due to several reasons:

• Malformed Payload: The payload may be so corrupted that even the lenient regex scan cannot find a <...:Body> tag or a child element.
• Non-SOAP XML: The payload is XML but not a SOAP envelope (e.g., it lacks a Body element).
• Empty Body: The SOAP Body element exists but contains no child elements.

Misleading Bodyless Examples

The system uses the isBodylessMethod function to distinguish legitimate bodyless calls (like GET) from potentially incomplete ones (like a POST with no body). If you see a learned example for a POST or PUT request that has no body, it often indicates a logging configuration issue at the gateway, where request bodies are not being captured. The system may still store this example, but it will be an incomplete representation of the actual call.

Hashing Failure

The sha256Hex method can, in theory, throw an IllegalStateException if the SHA-256 message digest algorithm is not available in the Java Virtual Machine environment. This is extremely unlikely in any standard Java environment. If it occurs, it indicates a severe problem with the runtime environment.

See Also

• F5TrafficLearningService: A consumer of this utility for processing F5-specific traffic.
• GenericTrafficLearningService: A consumer of this utility for processing generic syslog traffic.
• SoapPayloadExtractor: A component used by TrafficDerivationCore for detailed SOAP leaf-field extraction.

All SOAP-to-REST docs