Traffic-Driven Service Discovery

Overview

Traffic-driven discovery automatically builds a complete and current inventory of SOAP services by observing live traffic. Many organizations lack a comprehensive catalog of their SOAP services, which makes modernization and management difficult. This feature solves that problem by creating an inventory from the traffic that already flows through an edge gateway or load balancer. If a service is actively receiving calls, it will be discovered.

The discovery process has two primary inputs, provided by components called connectors:

1. Traffic Ingest: Connectors feed gateway traffic logs—including request/response metadata, latency, and payloads—into the platform.
2. Management API Discovery: Where available, connectors query the gateway's management API to enumerate all configured services, including those that are currently idle.

The platform processes this data to produce a master discovery view, which ranks all observed services by traffic volume. This provides an evidence-based worklist, enabling API publishers to prioritize which services to onboard and modernize first. Discovery identifies what to onboard; the onboarding process then turns a discovered service into a managed, published API.

How it Works

The discovery mechanism is designed for performance and scalability. It processes raw transaction data in the background to ensure that operator-facing dashboards remain fast, regardless of the number of services or traffic volume. The process follows a clear data flow from ingestion to presentation.

flowchart TD
    subgraph Edge Tier
        A[API Gateway / Load Balancer]
    end

    subgraph Platform
        B[Connector]
        C[Platform Ingest]
        D[Raw Transaction Log]
        E[Aggregation Worker]
        F[Pre-computed Aggregations]
        G[Discovery UI & Dashboards]
    end

    A -- 1. Traffic Logs & Mgmt API Data --> B
    B -- 2. Feeds data --> C
    C -- 3. Persists transactions --> D
    E -- 4. Reads raw data offline --> D
    E -- 5. Writes aggregated data --> F
    G -- 6. Reads pre-computed data --> F

1. Data Ingestion: A connector integrates with an edge gateway. It may provide data from one or both of two sources:

   • Traffic Ingest: The connector forwards traffic logs containing details of each SOAP transaction.
   • Management API Discovery: The connector queries the gateway's management API to get a structured list of configured services, their names, and their endpoints.

2. Persistence: The platform receives this data and persists each transaction as a raw log entry. The platform's own runtime proxy traffic is also ingested and contributes to discovery. All data is tagged with its source, allowing for filtering by source or viewing a combined total.

3. Aggregation: A background worker continuously processes the raw transaction logs. It rolls up the individual entries into pre-computed aggregation tables. These tables store summary metrics, such as request counts over specific time windows.

4. Presentation: Operator-facing surfaces, including the master discovery view and performance dashboards, query the pre-computed aggregation tables. They never query the raw logs. This architectural principle, the "read-path rule," ensures that UIs load in sub-second time, even with millions of raw transactions. Raw logs are only accessed for specific tasks like viewing the details of a single transaction or by background workers.

Master Discovery View

The master discovery view is the primary user interface for discovered services. It presents a list of every service observed through traffic or management API enumeration. For each service, it displays request counts over three rolling time windows, allowing API publishers to assess both current activity and long-term relevance.

Sorting the list by traffic volume transforms the inventory into a ranked worklist for modernization. Services with high traffic that were previously unknown are immediately visible, while services that have become inactive are also clearly identified.

Consumer Attribution

Beyond identifying busy services, discovery also answers "who is calling what?" by attributing each transaction to its original API consumer. The platform inspects a standard chain of HTTP headers to find the consumer's IP address, intelligently bypassing intermediate proxy and gateway IPs.

The platform checks the following headers in order of precedence:

1. X-Client-IP: A dedicated header for the client's IP address.
2. X-Forwarded-For: The de-facto standard header for identifying originating IPs in a proxy chain. The platform selects the leftmost IP address that is not an internal gateway IP.
3. Forwarded: The RFC 7239 standard header, using its for= parameter.

The first valid IP address found that does not belong to the immediate gateway hop is used for attribution. If no such header is present, or if all candidate IPs belong to the gateway, attribution falls back to the gateway's own IP address. The transaction is still recorded, but per-consumer analytics will not be available.

The Read-Path Rule

A core architectural rule of the platform is that time-bucketed operator surfaces read from pre-computed aggregation tables, never from raw logs at query time. This rule applies to all traffic sources and is fundamental to the platform's performance at scale.

Aggregation tables are used by:

• The master discovery view
• Dashboards and KPI tiles
• Timeline charts
• Per-service traffic rollups
• SLO reports

Raw logs are reserved for only three use cases:

• Ingest-health monitoring: Checking when the last log entry arrived from a source.
• Transaction drill-down: Fetching a full request/response payload when an operator inspects a single event.
• Background workers: Curator jobs that create derived data, such as aggregation tables and learned examples, from the raw logs.

The result of this separation is that user-facing analytics surfaces remain fast and responsive, with data freshness determined by the aggregation worker's refresh interval, not by expensive, on-demand queries over raw data.

Troubleshooting

Incorrect Consumer Attribution

• Symptom: In the discovery view and analytics dashboards, all or most traffic for a service appears to come from a single IP address corresponding to the API gateway or load balancer.
• Cause: The edge gateway is not injecting a forwarded header (like X-Client-IP or X-Forwarded-For) containing the original consumer's IP address into the requests it proxies. The platform's attribution logic falls back to using the gateway's IP as the source.
• Resolution: Configure your edge gateway to add a client IP header to requests. The documentation for the specific connector for your gateway provides instructions. While discovery will function without this, per-consumer attribution will be inaccurate.

See Also

• Connectors Overview
• Learned Examples
• Onboarding a Service

All SOAP-to-REST docs