Automated SOAP Service Discovery from Live Traffic

Overview

Traffic-driven discovery automatically builds a complete inventory of your SOAP services by observing live traffic at your gateway edge, such as a load balancer or API gateway. This solves the common problem of not having a current, complete inventory for large SOAP estates that have evolved over many years. If a service is receiving live traffic, it will be discovered.

This component works by ingesting traffic data from one or more sources, including gateway connectors and the platform's own runtime proxy. A background process analyzes this traffic to identify distinct services, their traffic volume, and the consumers calling them. The output is a ranked, evidence-based inventory called the master discovery view. This view allows API publishers to prioritize which services to onboard for modernization based on real-world usage.

Discovery tells you what services exist and which are most active. Once a service is identified, you can use other platform features to onboard it and begin the modernization process.

How it works

The discovery process relies on a "read-path rule" where operator-facing dashboards query pre-computed data rather than raw logs, ensuring high performance regardless of traffic volume.

flowchart TD
    subgraph "Gateway Edge"
        A[API Consumer] --> B{Load Balancer / API Gateway};
    end

    subgraph "Platform"
        C[Connector] --> D[Raw Transaction Logs];
        D --> E[Background Worker];
        E --> F[Pre-computed Aggregation Tables];
        F --> G[Operator UI / Dashboards];
    end

    B --> C;

    style G fill:#d4edda,stroke:#c3e6cb

The data flow follows these steps:

1. Traffic Ingestion: A connector at the gateway edge captures SOAP request and response metadata (and optionally, bodies) and feeds it into the platform. Connectors can also enumerate configured services directly from the gateway's management API.
2. Persistence: The platform persists each observed transaction as a raw log entry.
3. Aggregation: A background worker continuously processes these raw logs, rolling them up into pre-computed aggregation tables. These tables summarize traffic metrics over different time windows.
4. Presentation: Operator-facing surfaces, such as the master discovery view and dashboards, read directly from these fast, pre-computed aggregation tables. This allows the UI to load in sub-second time, even for estates with hundreds of services and millions of transactions.

Raw logs are only queried directly for specific, targeted tasks like viewing the details of a single transaction or for background jobs that derive other data, such as learned examples.

Discovery Inputs

Discovery uses two complementary methods to build the service inventory. A given connector may support one or both.

• Traffic Ingest: The platform analyzes gateway traffic logs, including request lines, response statuses, headers, latency, and client IPs. Every distinct front-end/back-end service URL that appears in the traffic is identified as a discovered service and ranked by its traffic volume.
• Management-API Discovery: If the gateway provides a management API, the platform queries it to enumerate configured virtual servers and services. This provides a structured inventory with names, addresses, and ports, and can identify services that are configured but currently have no traffic.

The platform treats the origin of the data as an explicit dimension named source. You can filter views to a single source (e.g., a specific gateway) or view metrics summed across all sources.

The Master Discovery View

The primary user interface for discovery is the master discovery view. It presents a comprehensive list of every discovered service, ranked by request counts over three rolling time windows.

By sorting this list, API publishers can transform an unknown inventory into a prioritized worklist, focusing on the most active services first. A service that has recently gone silent will also be clearly visible. The view's fast load times are a direct result of it reading from pre-computed aggregation tables.

Consumer Attribution

Discovery identifies not only which services are busy but also who is calling them. The platform attributes each transaction to the original consumer by inspecting a standard chain of forwarded-for headers in a specific order of precedence:

1. X-Client-IP: A dedicated header for the client's IP address.
2. X-Forwarded-For: The de-facto standard proxy chain header. The platform selects the leftmost IP address that is not an internal gateway address.
3. Forwarded: The RFC 7239 standard header, using its for= parameter.

The platform uses the first valid IP address found in this sequence that does not belong to the gateway itself. If no such header is present, or if all candidate IPs belong to the gateway, attribution falls back to the gateway's IP address. The transaction is still ingested and contributes to service discovery, but with reduced per-consumer fidelity.

The Read-Path Rule: Raw Logs vs. Aggregation Tables

A core architectural principle of the platform is that time-bucketed operator surfaces read from pre-computed aggregation tables, never from raw logs at query time. This rule applies to all traffic sources and ensures that the user interface remains fast and responsive at any scale.

Surfaces driven by aggregation tables:

• Master discovery view
• Dashboards
• KPI tiles
• Timeline charts
• Per-service rollups
• SLO reports

Operations that use raw logs:

• Ingest-health monitoring: Checking the freshness of data from a source.
• Transaction drill-down: Fetching the full request/response details for a single event, identified by its correlation ID.
• Curator workers: Offline background jobs that process raw logs to build aggregation tables and learned examples.

The practical benefit of this design is that UI performance and data freshness depend on the efficient background aggregation process, not on performing slow, expensive queries over massive raw datasets with every page load.

See Also

• Connectors overview: Learn about the two functions of a connector (management-API discovery and traffic ingest) and the available integration options.
• Learned examples: See how the platform uses captured traffic payloads to generate curated examples and mapping evidence.
• Onboarding a service: Understand the next steps for turning a discovered service into a published, modern REST API.

All SOAP-to-REST docs