Automatic Regex Pattern Generation for String Fields

Overview

This document describes the system for automatically generating pattern attributes for string fields in OpenAPI specifications. This feature addresses the issue of unconstrained string fields in API definitions, which can pose data quality and security risks.

When an API specification is generated, this system can automatically add a regular expression pattern to any string field in a request body that does not already have one. This process is controlled by a system-wide setting and several configurable strategies.

The primary goals of this system are:

• To tighten API contracts by default, ensuring that string inputs conform to expected formats.
• To apply these constraints consistently across all APIs, including those onboarded through UI, API, or bulk processes.
• To provide API publishers with control over how these default patterns are applied.
• To ensure that all generated or user-provided regular expressions are safe, portable, and do not introduce performance or security vulnerabilities like Regular Expression Denial of Service (ReDoS).

The pattern generation logic is applied server-side during specification generation. This means the original, immutable API mapping is not modified; the pattern is added to the published OpenAPI specification that API consumers and gateways use.

Policy Modes

The behavior of the automatic pattern generation is governed by the conversion_defaults_string_without_regex system setting. This setting accepts one of four possible modes. The default mode is auto-apply-silent.

Pattern Generation

When the policy mode is set to auto-apply-silent, the system uses a defined strategy to derive the appropriate regular expression for each unconstrained string field.

How it Works

The pattern generation logic is encapsulated in the derivePattern function. It determines the regex based on the chosen strategy and the field's existing minLength and maxLength constraints.

The process follows these steps:

flowchart TD
    subgraph Pattern Derivation Logic
        direction LR
        Input[("Inputs: <br/>- Strategy<br/>- Generic Regex<br/>- minLength<br/>- maxLength")] --> A{Strategy?};

        A -- "generic" --> B{Generic Regex<br/>is provided?};
        B -- Yes --> C[Use provided<br/>Generic Regex];
        B -- No --> D[Use default:<br/><code>^.{0,1000}$</code>];
        C --> Output([Derived Pattern]);
        D --> Output;

        A -- "alphanumeric-length" --> E[Set character class:<br/><code>[0-9a-zA-Z]</code>];
        A -- "length-bound" (Default) --> F[Set character class:<br/><code>[\\s\\S]</code>];

        E --> G{maxLength > 1000?};
        F --> G;
        
        G -- Yes --> H[Generate unbounded pattern:<br/><code>^&lt;char-class&gt;*$</code>];
        G -- No --> I[Generate bounded pattern:<br/><code>^&lt;char-class&gt;{min,max}$</code>];

        H --> Output;
        I --> Output;
    end

Pattern Strategies

The system supports three strategies for deriving patterns. The default strategy is length-bound.

Generic Regex Safety Validation

To prevent the use of insecure or non-portable regular expressions, the system validates any custom genericRegex supplied by an API publisher. A regex is considered unsafe if it is likely to perform poorly (e.g., via catastrophic backtracking) or is incompatible with downstream components like RE2-based API gateways.

The regexSafetyProblem function performs these checks. If the regex is safe, the function returns null. If it is unsafe, it returns a string describing the problem.

The following conditions will cause a regex to be rejected:

• Is blank or empty: The regex string is required.
• Exceeds maximum length: The regex cannot be longer than 1000 characters (MAX_REGEX_LENGTH).
• Invalid syntax: The regex must compile according to Java's java.util.regex.Pattern engine.
• Uses backreferences: Patterns like \1 or \k<name> are not supported.
• Uses \Q...\E quoting: This syntax is not supported.
• Uses lookaheads or lookbehinds: (?=...), (?!...), (?<=...), and (?<!...) are not supported.
• Uses atomic groups: (?>...) is not supported.
• Uses embedded comments: The (?x) flag is not supported.
• Uses possessive quantifiers: *+, ++, ?+, }+ are not supported.
• Contains large repeat counts: Quantifiers like {n,m} where n or m is greater than 1000 (MAX_DERIVED_QUANTIFIER) are rejected. This is to maintain compatibility with gateways that have a lower quantifier limit.
• Contains nested quantifiers: A pattern where a quantified group itself contains a quantifier (e.g., (a+)+, (a*)*) is rejected. This structure is a primary cause of catastrophic backtracking.

Configuration Reference

The following table lists the key constants and default values for the pattern generation system.

Function Reference

The system exposes two core functions for its logic.

derivePattern

Derives a regular expression pattern for an unconstrained string field.

Signature: String derivePattern(String strategy, String genericRegex, Integer minLength, Integer maxLength)

Parameters:

Returns: A String containing the derived regular expression. If maxLength is null, negative, or greater than MAX_DERIVED_QUANTIFIER (1000), the derived pattern will use an unbounded quantifier (*).

regexSafetyProblem

Validates a regular expression to ensure it is safe and portable.

Signature: String regexSafetyProblem(String regex)

Parameters:

Returns: null if the regex is considered safe. Otherwise, returns a String describing the first problem found. See Generic Regex Safety Validation for a list of possible problems.

Examples

Example 1: Deriving a length-bound Pattern

Given a string field with minLength: 10 and maxLength: 200.

Invocation:

derivePattern("length-bound", null, 10, 200);

Result: The function returns the following pattern, which accepts any character within the specified length range.

^[\s\S]{10,200}$

Example 2: Deriving a Pattern with maxLength > 1000

Given a string field with maxLength: 2000.

Invocation:

derivePattern("length-bound", null, 0, 2000);

Result: Because maxLength exceeds MAX_DERIVED_QUANTIFIER, the function returns a pattern with an unbounded quantifier. The maxLength constraint is still enforced separately.

^[\s\S]*$

Example 3: Validating a Safe Generic Regex

Invocation:

regexSafetyProblem("^[a-zA-Z0-9_-]{1,50}$");

Result:

null

Example 4: Validating an Unsafe Generic Regex

Invocation:

regexSafetyProblem("^(a+)+$");

Result: The function identifies the nested quantifier, which can lead to catastrophic backtracking, and returns an error message.

"nested quantifiers can backtrack catastrophically"

All SOAP-to-REST docs