Reproducibility Contract (RNG Handling)#

Note

This document is the canonical policy for randomness in process-improve. Tracks ENG-08.

Why one contract?#

Reproducibility of a fitted model, a bootstrap interval, or an optimisation result is part of the package’s value proposition. A chemometrics or DoE user comparing two runs across a software upgrade has to be able to distinguish “the algorithm changed” from “the random seed changed”. Today the codebase mixes:

  • unseeded np.random.default_rng() calls inside production algorithms,

  • hard-coded literal seeds (np.random.default_rng(42), seed=0) inside production algorithms,

  • well-seeded paths that thread random_state through correctly,

  • @tool_spec(rng={...}) metadata that describes the contract but enforces nothing.

The result is that “this run is reproducible” is currently a vibe, not a guarantee. This document pins the guarantee.

The contract#

  1. Every public function that touches an RNG MUST accept a ``random_state`` parameter. The accepted types are int | numpy.random.Generator | None (matching the convention used by scikit-learn since 1.4).

    def bootstrap(
    self,
    X: pd.DataFrame,
    *,
    n_boot: int = 1000,
    random_state: int | np.random.Generator | None = None,
) -> ...:

  2. Resolve ``random_state`` once at function entry using the helper process_improve._random.check_random_state(). Its resolution rules match sklearn’s check_random_state but it returns a modern numpy.random.Generator:

    • None -> a fresh, unseeded np.random.default_rng().

    • int -> np.random.default_rng(int).

    • Generator -> returned as-is.

    Use the resolved Generator for all draws inside the function; never call np.random.* directly.

  3. Hard-coded literal seeds in production code are forbidden. If a function currently does

    rng = np.random.default_rng(42)  # don't

    the 42 must move to the public signature as the default, and the function must use check_random_state to resolve it:

    def find_optimum(
    ...,
    *,
    random_state: int | np.random.Generator | None = None,
):
    rng = check_random_state(random_state)

  4. Unseeded ``default_rng()`` is forbidden except where “fresh noise on every call” is the documented contract. The only such case currently is simulation.model.simulate’s noise term, which is part of the simulator’s documented behaviour and is gated behind an explicit # Fresh noise: documented behaviour, not an accident comment.

  5. Every ``@tool_spec`` that touches an RNG MUST declare its contract via the ``rng=`` metadata:

    @tool_spec(
    name="bootstrap_pca",
    ...,
    rng={
        "uses_rng": True,
        "seed_param": "random_state",
        "default_seed": 0,
    },
)

    The seed_param field is the name of the kwarg the reproducibility-check harness will exercise. Deterministic tools declare {"uses_rng": False}.

  6. ``@tool_spec`` checks the declaration in (5), not the behaviour. At decoration it rejects an rng= dict without a boolean uses_rng key, with an unknown key, with a seed_param or default_seed on a tool that declares {"uses_rng": False}, or with a default_seed but no seed_param. No test yet runs a tool to confirm that its declared seed reproduces its output; that harness is open work.

How to migrate an existing function#

Most production paths are one mechanical edit:

Before:

def my_thing(...):
    rng = np.random.default_rng()        # or np.random.default_rng(42)
    ...

After:

def my_thing(
    ...,
    *,
    random_state: int | np.random.Generator | None = None,
):
    rng = check_random_state(random_state)
    ...

The behaviour for callers that did not pass random_state is preserved (None falls through to an unseeded default_rng).

For an algorithm whose documented public behaviour was “always seeded with 42”, set default_seed=42 in the @tool_spec(rng=...) metadata and accept random_state=None as a synonym; this preserves byte-equivalence for existing callers.

What is not covered#

  • The unseeded noise in simulation.model.simulate is deliberately not reproducible; that is the simulator’s contract.

  • The hard-coded plot seeds in surfaces.py / design_quality.py (seed for plot-only jitter) are in-scope for the migration but are low priority – a plot with shifted-by-one jitter is not a correctness failure.

  • Numerical reproducibility across NumPy major versions is out of scope (NumPy ships RNG algorithm changes on a documented schedule).

Open work#

Aspects of the contract that are not yet implemented:

  • A harness that runs every @tool_spec(rng={"uses_rng": True}) tool twice with its declared default_seed and asserts byte-equal outputs, and runs every tool declaring {"uses_rng": False} twice without a seed to the same end. It needs a valid sample input for each tool, which the registry does not hold yet.

  • Migration of the existing offenders is tracked in SEC-21 sub-item 9 (Resampler), the relevant SEC-33 sub-item (#282) (optimization.py:564), and ENG-08 (#290) itself for the remaining call sites.

Cross-references#