API Version Publish Readiness Assessment

Overview

The Publish Readiness Assessment is a service that analyzes a draft version of an API to determine if it is ready to be activated and published. It acts as a quality gate, preventing the activation of incomplete or improperly configured API versions.

The service aggregates data from multiple sources to build its assessment:

• Operation Mappings: It checks the readiness status of each operation's request and response mappings.
• Backend Configuration: It verifies that backend endpoint settings are present and a default is configured.
• Backend Test Results: It analyzes the results of backend SOAP tests to ensure the API version has been successfully tested against the backend.
• WSDL Import History: It compares the current version's WSDL with previous versions to detect breaking changes or other significant modifications.

Based on this analysis, the service produces a comprehensive report that includes a final readiness status, a list of blocking issues, warnings, informational notes, and a set of suggested actions for the API publisher to resolve any identified problems.

How it works

The readiness assessment process is initiated by calling the assess service with a serviceId and a versionId. The service then performs a series of data gathering and analysis steps.

Data Gathering and Analysis

1. Load Context: The service first loads the context for the specified API version, including its current status (draft, published, etc.) and its associated WSDL artifact.
2. Diagnose Operations: It retrieves all operations for the version and, for each one, invokes the mapping automation service to diagnose its readiness. This yields a status for each operation, such as AUTO_READY, AUTO_READY_WITH_WARNINGS, or MANUAL_REQUIRED.
3. Check Backend Settings: It queries the service's backend profile configuration to verify that at least one profile exists and that a default profile is selected.
4. Analyze Test Results: The service aggregates the results of backend SOAP tests (previews) that are of type application or application-generated. It calculates the total number of successful and failed tests, the number of unique operations with successful or failed tests, and the timestamps of the most recent tests.
5. Compare WSDL Artifacts: The service retrieves the WSDL import artifact for the current version and compares it to the most recent comparable artifact from a previous import. This comparison identifies breaking changes, new or removed operations, and other differences.
6. Inherit Test Coverage: If the draft has no successful backend tests, the service attempts to find a previously published version with an identical contract and mapping structure. If such a version exists and has full test coverage, its test status is inherited. See Inherited Backend Test Coverage for details.

After gathering this data, the service evaluates a set of rules to generate lists of blockingIssues, warnings, and notes. These lists determine the final status, a numeric score, and a recommendation. Finally, it generates a list of suggestedActions to guide the API publisher in resolving any issues.

Architecture Flow

The following diagram illustrates the flow of data into the readiness assessment service.

flowchart TD
    subgraph "Input"
        ServiceVersion[API Service Version<br/>(serviceId, versionId)]
    end

    subgraph "Data Gathering"
        A[Load Version Context]
        B[Load Operations]
        C[Diagnose Mappings<br/>(per operation)]
        D[Load Backend Summary]
        E[Load Backend Test Summary<br/>(Previews)]
        F[Load WSDL Import &<br/>Comparison Summary]
        G{Has successful tests?}
        H[Check for Inheritable<br/>Test Coverage]
    end

    subgraph "Assessment Logic"
        I[Build PublishContext]
        J[Evaluate Rules:<br/>- Blockers<br/>- Warnings<br/>- Notes]
        K[Determine Status & Score]
        L[Generate Suggested Actions]
    end

    subgraph "Output"
        Assessment[Readiness Assessment Report<br/>- Status<br/>- Score<br/>- Issues<br/>- Actions]
    end

    ServiceVersion --> A
    ServiceVersion --> B
    ServiceVersion --> D
    ServiceVersion --> E
    ServiceVersion --> H

    A --> I
    B --> C
    C --> I
    D --> I
    E --> G
    A --> F
    F --> I

    G -- No --> H
    G -- Yes --> I
    H --> I

    I --> J
    J --> K
    J --> L
    K --> Assessment
    L --> Assessment

Inherited Backend Test Coverage

To streamline the publishing of versions with minor, non-functional changes (e.g., updating a timeout), the assessment service can inherit backend test coverage from a previously published version.

This mechanism is triggered when a draft version has zero successful backend tests. The process is as follows:

1. Fingerprint Generation: The service computes two fingerprints for the draft version:
   • Contract Fingerprint: An MD5 hash based on each operation's key, REST/SOAP configuration, and schemas. Operational settings like timeouts are excluded.
   • Mapping Fingerprint: An MD5 hash of the active request, response, and fault mappings for all operations.
2. Candidate Search: The service searches for recent, active, and previously published versions of the same API service.
3. Fingerprint Comparison: It computes the same fingerprints for each candidate version and compares them to the draft's fingerprints.
4. Inheritance: If a candidate version is found with identical contract and mapping fingerprints, and that version has complete and successful backend test coverage, the draft inherits its test status.

When coverage is inherited, the "No successful backend SOAP test" blocker is downgraded to an informational note, and the response payload includes an inheritedPreviews object detailing the source of the inherited coverage.

API Reference

The service exposes a single function to perform the assessment.

assess(serviceId, versionId)

Triggers a publish readiness assessment for a specific API service version.

Request Parameters

Response Body

The method returns a JSON object containing the full assessment report.

summary Object

inheritedPreviews Object

This object is only present if the draft inherited test coverage from a prior version.

checks Object

This object provides a quick-reference map of boolean flags for key readiness criteria.

suggestedActions Object

operations Object

Readiness Statuses

The assessment returns one of three possible statuses.

Troubleshooting Common Issues

The assessment report lists specific issues in the blockingIssues and warnings arrays. The following are common messages and their resolutions.

Blocking Issues

• "Draft version has no operations."
  • Cause: The API version was created but no operations were imported or defined.
  • Resolution: Import a WSDL or define operations for the version.
• "Backend settings are missing." or "Default backend settings are not configured."
  • Cause: The service has no backend profiles defined, or a default profile has not been selected.
  • Resolution: In the service's backend settings, create at least one backend profile and designate one as the default.
• "X operations still require manual mapping review."
  • Cause: One or more operations have a mapping status of MANUAL_REQUIRED.
  • Resolution: Review the auto-generated mappings for the indicated operations and approve them.
• "No successful backend SOAP test has been saved for this draft yet."
  • Cause: The backend test suite has not been run, or all tests failed. This blocker does not appear if test coverage was inherited.
  • Resolution: Run the "Test All Operations" or "Test This Operation" function to execute tests against the configured backend.
• "Imported WSDL is missing request/response contract snapshots for some operations."
  • Cause: An issue occurred during WSDL import that prevented the system from generating contract examples for all operations.
  • Resolution: Review the WSDL import details and consider re-importing the WSDL file.
• "WSDL re-import comparison reports breaking changes."
  • Cause: The WSDL for this version contains breaking changes compared to a previous version.
  • Resolution: Review the WSDL comparison report. If the changes are intentional, this blocker may be acceptable, but it requires careful review.

Warnings

• "Successful backend SOAP tests exist for only X/Y operations."
  • Cause: Backend tests have been run successfully, but not for every operation in the version.
  • Resolution: Run tests for the remaining untested operations to ensure full coverage.
• "WSDL import differs from the previous comparable artifact."
  • Cause: The WSDL has changed (e.g., added or changed operations) but the changes are not considered "breaking".
  • Resolution: Review the WSDL comparison details to understand the changes before activation.
• "X operations are auto-ready with warnings."
  • Cause: The mapping generation for some operations completed but produced warnings.
  • Resolution: Review the diagnostics for the indicated operations to understand the warnings.
• "Backend SOAP tests have failed for X operations."
  • Cause: Some backend tests are consistently failing.
  • Resolution: Investigate the failed tests to determine if the issue is in the API configuration, the mapping, or the backend service itself.
• Warnings related to WSDL import: Messages about readyWithReview, complexReview, routeFallbacks, or routeCollisions.
  • Cause: The WSDL import process identified complexities that require manual review.
  • Resolution: Review the generated REST routes, contract samples, and other details from the WSDL import step.

See Also

• Operation Mapping Diagnostics: The readiness assessment relies on per-operation diagnostics. Understanding how mappings are generated and diagnosed is key to resolving mapping-related issues.
• WSDL Import and Comparison: Many readiness checks are based on the results of WSDL import and comparison. Reviewing the import process helps resolve issues related to breaking changes, route generation, and contract examples.

All SOAP-to-REST docs