SOAP Payload Leaf Field Extractor

Overview

The SoapPayloadExtractor is a utility for parsing SOAP XML payloads from either requests or responses. Its primary function is to traverse a SOAP document, identify all text-bearing leaf elements, and extract their values into a structured format.

This component solves the problem of understanding the data types and encodings used within SOAP traffic. It produces a flat list of LeafField records. Each record contains the element's simplified XPath, its local name, its trimmed text value, a detected encoding fingerprint, and a flag indicating the presence of Hebrew characters.

The SoapPayloadExtractor is called by the traffic learning service to analyze captured traffic. The extracted field information is used to populate a data store that provides insights and hints within the API management user interface, particularly during service onboarding and configuration.

How it Works

The extraction process is initiated by the static method extractLeafFields, which takes a raw XML payload string and a processing limit as input. The process follows these steps:

1. Input Validation: The extractor first checks if the input payload is null or consists only of whitespace. If so, it immediately returns an empty list.
2. Secure XML Parsing: To handle potentially untrusted traffic from external systems, the extractor configures a secure XML parser. This involves enabling and disabling several features to prevent common XML-based vulnerabilities:
   • XMLConstants.FEATURE_SECURE_PROCESSING is enabled.
   • http://apache.org/xml/features/disallow-doctype-decl is enabled to block DOCTYPE declarations.
   • http://xml.org/sax/features/external-general-entities is disabled.
   • http://xml.org/sax/features/external-parameter-entities is disabled.
   • XInclude processing is disabled.
   • Automatic expansion of entity references is disabled.
3. DOM Tree Generation: The input string is parsed into a W3C DOM Document. Any parsing error is caught, and the method returns an empty list. This design ensures that malformed traffic does not crash the data ingestion pipeline.
4. Recursive Traversal: The extractor performs a depth-first traversal of the DOM tree, starting from the root element. It builds a simplified XPath-like string for each element (e.g., /Envelope/Body/MyOperation).
5. Leaf Element Identification: An element is considered a leaf if it contains no child elements. When a leaf element with non-empty text content is found, its value is processed.
6. Field Analysis: For each leaf field's text value, the extractor:
   • Trims leading and trailing whitespace.
   • Checks for the presence of Hebrew characters (Unicode code points U+0590 through U+05FF).
   • Generates an encodingFingerprint by applying a series of pattern checks.
7. Result Aggregation: A LeafField record is created for each qualifying element and added to a list. The process stops if the number of extracted fields reaches the maxLeaves limit.
8. Output: The final list of LeafField records is returned.

The following diagram illustrates the data flow:

flowchart TD
    A[Input: SOAP XML Payload] --> B{Payload Valid?};
    B -- No --> X[Output: Empty List];
    B -- Yes --> C[Securely Parse XML to DOM];
    C -- On Error --> X;
    C -- Success --> D[Walk DOM Tree];
    subgraph "For Each Leaf Element with Text"
        E[Extract Text Value] --> F[Analyze Value for Hebrew & Encoding];
        F --> G[Create LeafField Record];
    end
    D --> E;
    G --> H{Field Limit Reached?};
    H -- No --> D;
    H -- Yes --> I[Stop Processing];
    I --> J[Output: List of LeafField Records];
    D -- Walk Complete --> J;

API Reference

The component exposes a single static method for extracting fields.

extractLeafFields

Walks a SOAP envelope and collects every leaf element that contains non-blank text content.

static List<LeafField> extractLeafFields(String payload, int maxLeaves)

Parameters

Returns

A java.util.List of LeafField records. The list will be empty if the input payload is null, blank, or if any XML parsing error occurs.

LeafField Record

The extractLeafFields method returns a list of LeafField records, each representing a single extracted value.

Encoding Fingerprints

The encodingFingerprint is a string that categorizes the content of a field's value. The extractor checks for these categories in a specific order, and the first one to match determines the fingerprint.

Troubleshooting

Empty Result List

The extractLeafFields method is designed to fail silently by returning an empty list rather than throwing an exception. An empty list will be returned under the following conditions:

• The input payload string is null.
• The input payload string is blank (empty or contains only whitespace).
• The XML parser encounters any error while parsing the payload. This includes malformed XML or documents that violate the configured security constraints.

This behavior prevents malformed traffic from disrupting the data ingestion service.

Incomplete Results

If the returned list of fields is shorter than expected for a large document, check the maxLeaves parameter. The extraction process stops once the number of collected fields reaches this limit, which defaults to 256.

See Also

• Traffic Learning Service: This component is the primary consumer of the SoapPayloadExtractor. It uses the extracted field data to learn the structure and data formats of APIs from observed traffic.

All SOAP-to-REST docs