Master Discovery Aggregation

Overview

The MasterDiscoveryAggregator is a pure-function component that transforms fine-grained, per-request traffic data into a consolidated, service-centric view. It serves as a central processing step in the Master Discovery pipeline.

The primary problem this component solves is the reconciliation of raw traffic data, which is often fragmented and noisy. A single logical service may appear as dozens of distinct rows in the raw data, varying by consumer IP, specific gateway virtual server, or minor URI differences. The aggregator's purpose is to collapse these disparate rows into a single, canonical entry for each unique service within each environment.

It takes a list of flat data rows as input, where each row represents an observed traffic event with details like source IP, virtual servers, and backend pools. Its output is a refined list of aggregated service definition rows. Each output row represents a unique service, with consolidated lists of consumers, infrastructure components, and summed traffic metrics over various time windows. This aggregation is critical for presenting a clean and accurate inventory of discovered services.

How it Works

The aggregation process is executed in a multi-phase pipeline. Each phase refines the data, culminating in a list of canonical service definitions. The primary entry point is the collapseByUriAndEnvironment method.

flowchart TD
    subgraph Input
        A[Raw Traffic Rows]
    end

    subgraph Aggregation Pipeline
        B(Phase A: Per-Row Environment Detection) --> C(Phase B: Hostname-Diff Fallback)
        C --> D(Phase C: Bucket by Service Key)
        D --> E(Phase D: Merge by Backend URL)
        E --> F(Phase E: Filter Non-Service Rows)
    end

    subgraph Output
        G[Aggregated Service Rows]
    end

    A --> B
    F --> G

Phase A: Per-Row Environment Detection

The first phase processes each raw input row individually to determine its canonical URI and environment.

1. Canonical URI: The serviceUri is canonicalized by lower-casing and stripping query strings and .svc suffixes. This ensures that minor variations of a URI are treated as the same service. Rows with empty or root (/) URIs are discarded.
2. Signal Collection: Infrastructure details from the raw row, such as frontVirtualServer, backVirtualServer, dataPowerDomainName, and pool member hostnames, are collected. These act as signals for environment detection.
3. Environment Detection: The EnvironmentDetector service uses these signals to determine the most likely environment (e.g., "Prod", "Test").
4. Rollup Fallback: If the signal-based detection results in a generic default, the aggregator checks for an environment field on the raw row itself. This field, often pre-populated by upstream traffic rollup processes, is considered a more reliable source for traffic from generic gateways (e.g., HAProxy, NGINX) that may lack environment-specific naming conventions in their infrastructure signals. This fallback is skipped if the rollup environment is a placeholder value like auto.
5. Stamping: The results—_canonicalUri, _env (environment), _envSource (detection source), and _envIssues—are temporarily added to the raw row map for use in subsequent phases.

Phase B: Hostname-Diff Fallback

This phase provides a secondary environment detection mechanism for ambiguous cases. It operates on groups of rows that share the same canonical URI.

If a group of rows contains at least one row whose environment was unresolved or assigned a default, this phase attempts to find a more specific environment by comparing backend hostnames. It extracts hostnames from the serviceBackendUrl or backPoolMembers fields of all rows in the group. If distinct hostnames are found, the HostnameDiffEnvDetector analyzes their differences to infer environments. If a more specific environment is found, the row's temporary _env and _envSource fields are updated in place.

Phase C: Bucketing by Service Key

This is the core aggregation step where individual rows are collapsed into buckets. Each bucket represents a single, unique discovered service.

A composite key is generated for each row: _canonicalUri + | + _env + | + backendKey.

The backendKey is a crucial discriminator that prevents services with different backends from being incorrectly merged. It is determined using the following precedence:

1. The canonicalized serviceBackendUrl.
2. The serviceBackendKey (a pre-computed hash), if the URL is blank.
3. The frontVirtualServer name, as a final fallback.

Rows are placed into buckets based on this composite key. This ensures that two services with the same URI and in the same environment but routing to different backends (e.g., a blue/green deployment) are kept as distinct entries.

Within each bucket, traffic statistics (hits1y, hits1m, etc.) are summed, and infrastructure components (frontVirtualServers, consumers, backPoolMembers, etc.) are collected into unique sets.

Phase D: Merge by Canonical Backend

This phase acts as a safety net to merge service entries that were separated into different environment buckets in Phase C but are discovered to point to the exact same backend. This can happen during migrations or due to inconsistent environment naming in gateway configurations.

1. Grouping: Rows output from Phase C are grouped by a key of serviceUriCanonical + | + serviceBackendUrl. Rows with a blank URI or backend are not merged.
2. Merging: If multiple rows fall into the same group (same URI, same backend), they are merged.
   • Environment Winner: The environment label from the row with the higher hits1m (1-month hit count) is chosen for the final merged row.
   • Traffic Summation: Hit counts from all time windows are summed.
   • Topology Purity: To prevent creating a row with mixed-environment infrastructure, the merge logic is strict. If the merging rows came from different environments, only the infrastructure details (virtual servers, pool members) of the "winner" (the row with more traffic) are kept. If they came from the same environment, their infrastructure lists are unioned.
   • Issue Cleanup: After merging, any environment_mismatch discovery issues are removed, as the ambiguity has been resolved by this phase.

Phase E: Filter Non-Service Rows

The final phase removes "phantom" rows that represent noise rather than actual, integrable services. Two types of rows are targeted for removal:

1. Context Root Placeholders: A row is identified as a placeholder (e.g., a request to a base application path like /MyApplication instead of a service endpoint like /MyApplication/v1/service) and removed if it meets all of the following criteria:

   • It has no resolved backend URL.
   • It is a path-ancestor of at least one other discovered row that does have a resolved backend.
   • Its traffic (hits1y) is negligible compared to the total traffic of its descendant services (specifically, less than 1%).

2. Operations in Service Slots: A row is removed if it appears to be a misdirected call where a SOAP operation name was used in the service part of the URI. This occurs if all of the following conditions are met:

   • It has no resolved backend URL.
   • The final segment of its URI exactly matches a known SOAP operation name (passed in via the knownOperationNames set).
   • A sibling row under the same parent path does have a resolved backend, indicating the presence of the true service.

Rows that survive this filtering phase constitute the final list of aggregated services.

API Reference

The aggregator is exposed as a class with static methods.

MasterDiscoveryAggregator.collapseByUriAndEnvironment

This method orchestrates the entire aggregation pipeline.

Parameters

Returns

A List<Map<String, Object>> containing the aggregated and collapsed service rows. Each map in the list represents a single canonical service. See Output Row Fields for the structure of these maps.

Input Row Fields

The rawRows maps are expected to contain the following keys, which are used during aggregation. The source does not specify types or defaults for all fields.

Output Row Fields

Each map in the list returned by collapseByUriAndEnvironment conforms to the following structure, generated by the internal Bucket.toRow() method.

Key Behaviors and Troubleshooting

Preventing Mixed-Environment Topologies

A primary design goal of the aggregator is to ensure "topology purity." A service row labeled for one environment (e.g., "Prod") should never contain infrastructure components (like virtual servers or pool members) from another environment (e.g., "Test").

• Phase C (bucketBackendDiscriminator) is the first line of defense. By including a backend-specific key in the bucketing logic, it prevents rows with the same URI but different backends from being merged prematurely.
• Phase D (mergeSiblingBackendRow) enforces this rule strictly during cross-environment merges. When two rows with the same backend but different environment labels are merged, the infrastructure topology of the traffic "winner" is kept, and the loser's is discarded entirely. This prevents the final row from having a mixed list of virtual servers or pool members.

Filtering of "Phantom" Services

Operators may notice that some entries from raw traffic logs do not appear in the final discovered services list. This is often the intended result of the filtering in Phase E.

• If a service URI corresponds to a bare application context root (e.g., /DigitalRashal18), has no backend, and has negligible traffic compared to its more specific descendant services (e.g., /DigitalRashal18/UpdateMedicalStatus), it will be filtered out. This removes noise and focuses discovery on actual, callable endpoints.
• Similarly, if a URI appears to be a malformed request where a SOAP operation name was used as the service endpoint (e.g., /services/processCleanlinessLaw instead of /services/CleanlinessLawBean), it will be removed, provided the real service is also present.

Environment Resolution Precedence

The environment for a service is determined by a clear order of precedence:

1. Primary: EnvironmentDetector using specific signals from infrastructure names (e.g., a _prod_ token in a virtual server name).
2. Fallback 1: The environment field on the raw input row, if the primary detection only yielded a default. This honors upstream rollup data, which is crucial for generic gateways.
3. Fallback 2: HostnameDiffEnvDetector (Phase B), which infers the environment by comparing backend hostnames of sibling services.
4. Default: A platform-wide default environment if no other method succeeds.

See Also

• ServiceUriCanonicalizer: The component responsible for normalizing service URIs.
• EnvironmentDetector: The primary service for detecting environments from infrastructure signals.
• HostnameDiffEnvDetector: The fallback service for detecting environments based on hostname differences.
• F5LearningRepository: A typical upstream component that provides the raw traffic data for aggregation.

All SOAP-to-REST docs