Skip to content

Score single-class folds and refuse weightless balanced classes in microcosm-fit - #1617

Merged
MaxGhenis merged 7 commits into
mainfrom
fix/fit-fold-degenerate-classes
Nov 17, 2026
Merged

MaxGhenis merged 7 commits into
mainfrom
fix/fit-fold-degenerate-classes

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Builds on microcosm#1543, which landed on main as e0f728f. This branch has merged main (336b80c40) and been repinned. The independent review has finished (round 2 raised no P0 or P1).

microcosm#1543 made categorical_scores refuse a probability row that is not a distribution. While verifying it, that session found two fold-level cases it left alone because they were outside its brief. This PR fixes both.

What changes

Case Before This branch
out_of_fold_proba: a fold whose training rows hold a single class CategoricalClassifier.fit fits no learner there, so each learner's rows (boosted, spline_logit) in that fold stay at zero. #1543's scorer refuses them ("row(s) of proba do not sum to 1"); main scored them at the 1e-4 log-loss floor. Each learner's rows in that fold take the model's probabilities: certainty of the one training class, which the model already gave. In such a fold each learner's rows now equal the model's, so none is left at zero. A learner that no fold fitted (a one-class target, or folds that each hold one class) still stays out of the result.
qrf._class_balanced_weights: a class whose rows carry no weight labels=[0,0,1,1], weights=[1,1,0,0] gives NaN weights and a RuntimeWarning. Refuses with a ValueError naming the class and its total. A class whose total overflows to infinity still counts as carrying weight and is balanced as before (its rows weigh zero). If every class's total overflows, every row's weight is NaN and the fit fails in scikit-learn, also as before.

Why fill instead of dropping or refusing

  • Not refusing up front: a refusal would turn a stage's held-out evidence into a build failure.
  • Not dropping the learner: the UK stage-health gate compares the blend with its better learner only when both learners' log losses are present (stage_health.py, _occupation_redraw_gate: if len(learners) == 2). Dropping a learner in a degenerate fold would switch that check off without a word.
  • Filling: the model, both learners and the constant are scored on the same rows. On the filled rows the model and both learners give the same certainty, and so does the constant wherever the fold's training weights have a finite total. So every comparison the gate makes still runs, and the model-minus-learner differences come only from folds where the learner was fitted.
  • No new output for one-class targets: a learner that no fold fitted stays absent. UK APS relies on that: it pools a learner only where every major group fitted it, and a group whose donors hold one sub-major group fits neither.

Who can reach these

  • Case 1.
    • UK APS deals rows after can_deal_folds. In a major group with two sub-major groups or more, every class sits in every fold, so no training fold holds one class. In a major group with one sub-major group, every training fold holds that class, but no fold fits a learner, so nothing is filled and the output is unchanged.
    • UK SPI deals households into five folds. Each household is dealt with its rarest class (counted in households, ties to the first in sorted order). Let R be the first class in that (rarity, sorted) order and R2 the next.
      • Every household holding R is dealt in R's stratum, so R sits in every fold, and a single-class training fold could hold only R. Then R2 would have to sit wholly in the held-out fold.
      • Households holding R2 but not R are dealt round the folds in R2's own stratum (every other class they hold comes after R2 in that order), so at most one of them can sit in one fold.
      • The rest hold R and are dealt with R's households, at most ⌈n_R/K⌉ to a fold, where n_R ≤ n_R2.
      • So n_R2 − 1 ≤ ⌈n_R2/K⌉, which fails for every K ≥ 3 once can_deal_folds requires n_R2 ≥ K.
    • With two folds it can happen: households {a, b}, {a} and {b} do it for 16 of 40 seeds.
    • So neither production caller reaches this. Refuse invalid fit input that the microimpute bug-parity audit reproduced #1543 left the SPI case open; the counting argument above settles it. An exhaustive search over small household structures and every round-robin dealing finds no single-class training fold for K = 3, 4 or 5. The Hypothesis property checks the argument's first step (R sits in every fold) and its conclusion on generated households, including boundary examples that two broken dealings fail. A direct caller with two grouped folds, or with folds of its own making, can reach the case.
  • Case 2. No fit reaches the refusal. _cross_fit_gate first runs _folds_carry_every_class, which tests the same per-class totals with the same predicate (not total > 0). Any fold whose training rows carry no weight for a class sends the whole gate to the in-sample gate (in_sample_weightless_fold). That check came before the balanced gate (UK SPI stage 1 keeps 4–17% of the tape's donors; replace the removed QRF tail gate with a donor-fidelity check #1276). The refusal guards the helper for any other caller.
    • The brief said a sign weightless in one fold's training rows reaches the helper. A repro shows it gets the fallback instead, with no warning.
    • A first version also refused a class whose total overflows to infinity. The in-session pre-review found that a fit does reach that case: zero-sign rows weighted 1e307 each are finite and pass the fit's weight checks, and main completes the fit with outcome cross_fitted_class_rates. The brief says not to change any fit whose classes all carry weight, so the helper now balances that class as before, and a guard pins the fit bit for bit. That such a class's rows silently weigh zero is pre-existing and out of scope here. So is the case where every class's total overflows: every row's weight is then NaN and the fold gate's fit fails in scikit-learn (Input sample_weight contains NaN.), on main and on this branch alike.

Valid input is unchanged

A main-vs-branch differential runs two fixed workloads through each tree's microcosm.fit:

  • Refuse invalid fit input that the microimpute bug-parity audit reproduced #1543's workload: 335 arrays covering every QRF regime, chains, the classifier, out_of_fold_proba and the scorer.
  • A new one: 504 arrays covering:
    • _class_balanced_weights on 300 inputs where every class carries weight, and on 60 inputs with 1e308 weights (52 with an overflowing class total, 19 of those with every class overflowing);
    • the balanced cross-fitted gate;
    • RegimeGatedQRF with cross_fitted_balanced: weighted, grouped, the weightless-fold fallback, and three fits with an overflowing class total (two complete; one raises on every tree, and its error message is compared);
    • out_of_fold_proba with row and household folds, every boosted_weight and the no-learner cases.

Every array matches bit for bit against both main 5658fc015 and #1543's head 18924d618: 33,492 values in the first workload and 28,871 in the second.

After merging main, the branch (ea5857e16) was compared again with main 336b80c40, which includes #1543: both workloads match bit for bit (33,492 and 28,871 values). Main's only change to microcosm-fit source since 18924d618 is #1543's docstring edit to categorical_scores (AST-equal once docstrings are stripped).

Tests

The new module is packages/microcosm-fit/tests/engine_free/shared/test_fold_degenerate_classes.py, with 12 tests. Against both main and #1543's head, the 5 that check the fixes fail and the 7 guards pass. On this PR's earlier head b90f098fb, which also refused an overflowing total, the two overflow guards and the balanced-weights property fail.

  • Examples:
    • a single-class training fold gets the model's rows in each learner;
    • two grouped folds can leave one class;
    • the helper refuses a weightless class by name.
  • Guards:
    • a learner no fold fitted stays out;
    • folds that keep two classes are unchanged bit for bit;
    • an overflowing class total is balanced as before, in the helper and in a full fit;
    • a sign weighted on one row falls back to the in-sample gate with no RuntimeWarning.
  • Hypothesis properties (derandomized, with event() labels and explicit examples for the edge cases):
    • On the small tables it draws, every array out_of_fold_proba returns is a distribution in every row. Against an embedded copy of the old function, only learner rows that were all zeros move, and they equal the model's. Examples cover the boosted-only and spline-only models on a single-class fold, and folds that each hold one class.
    • Dealt by household, the rarest class sits in every fold, and with three or more folds every training fold holds two classes. Strategies push classes to the can_deal_folds bound. Swapping in a dealing by the commonest class, or a random one, fails the property.
    • The helper keeps the old formula's bits wherever no class is weightless (overflowing totals included), and refuses exactly when one is.
    • The balanced gate balances only folds that carry every class.

Attested hashes

microcosm.build.spec_engine.seeds attests microcosm.fit.qrf's source bytes. They were repinned with these tools, --check clean:

  • tools/repin_us_spec_engine_contracts.py
  • tools/spec_engine_coverage.py (41/41 inventory checks)
  • then python -m tools.graph_parity_repin fit.qrf

ruff format restores the double quotes the repin tool drops. They were repinned again after the round-1 review's docstring fixes, and again after merging main, whose only conflicts were these pin files (--check clean; 42,401/42,401 fields; the fit.qrf node keys are unchanged). At ea5857e16, 392 pin, parity, seed and microcosm-fit tests pass locally.

Review

  • In-session pre-review (supporting evidence only): found the overflow reach (fixed in f3b489a1b), a grouped-folds property that passed broken dealings (fixed), and wording slips (fixed).

  • Round 1, Subfleet hard tier (job 20261114-143433-fit-fold-r1, head f3b489a1b): REQUEST_CHANGES. Both code fixes are correct and valid input is bit-identical. The reviewer reran both differentials, the old-tree tests, the repin --check and a fuzz of stratified_folds. What it flagged:

    • P2: out_of_fold_proba's docstring promised a distribution in every row of every array, which weights whose fold total overflows float64 break. The sentence now says only what the fill guarantees: in a single-class fold each learner's rows equal the model's, so none is left at zero.
    • P2: the qrf docstring, changelog and this body said an overflowing class's rows weigh zero, but when every class overflows every row's weight is NaN. Each now says so, as before on main.
    • P3: the overflow workload count, corrected above.
    • P3: commit 94ff4a721's message says no fit reached the refusal, which was false for the overflow half that commit also refused. f3b489a1b corrects it. History is left as is, with no force-push.
    • The fixes, in 708d67e49, change docstrings, comments and the changelog only: qrf.py, held_out.py and the test module are AST-equal to f3b489a1b once docstrings are stripped.
  • Round 2, Subfleet hard tier (job 20261115-223630-fit-fold-r2, head 708d67e49, a full review): REQUEST_CHANGES, with no P0 or P1, so the review has finished under Review by class.

    • The reviewer found round-1 findings 2 and 3 resolved and finding 4 disclosed. Finding 1 was partly resolved.
    • It reran both differentials against main and Refuse invalid fit input that the microimpute bug-parity audit reproduced #1543 (bit-identical), the old-tree tests, the repin --check, coverage and the parity tests.
    • P2, fixed in the PR: out_of_fold_proba's docstring said the model, the learners and the constant differ only where a learner was fitted. In a single-class fold whose training weights overflow, the constant is NaN while the model and filled learners are certain. The clause now covers the model and the learners only (docstring only; held_out.py is not attested, so no repin).
    • P3: 94ff4a721's message, as above; left in history and disclosed.

Merge

🤖 Generated with Claude Code

MaxGhenis and others added 5 commits November 14, 2026 11:42
out_of_fold_proba left each learner's rows at zero in a fold whose
training rows hold a single class, since CategoricalClassifier fits no
learner there; categorical_scores now refuses such rows. Those rows now
take the model's probabilities, certainty of that class, which the model
and the training folds' shares already gave. A learner no fold fitted
still stays out of the result.

_class_balanced_weights divided by each class's total weight and
returned NaN weights for a class with none. It now refuses that class,
or one whose total overflows, by name. The cross-fitted gate never
reached it with such weights: it falls back to the in-sample gate
(in_sample_weightless_fold) first.

Valid input is unchanged: two differential workloads (772 arrays) match
main and microcosm#1543 bit for bit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
microcosm.build.spec_engine.seeds attests microcosm.fit.qrf's source
bytes. Repinned with tools/repin_us_spec_engine_contracts.py and
tools/spec_engine_coverage.py (41/41 inventory checks, 42,394/42,394
fields, --check clean), then python -m tools.graph_parity_repin fit.qrf,
which writes only when the local direct call's bytes are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#1543 merged main again (085672f), which moved the attested spec-engine
hashes. The four hash-only conflicts took #1543's side, then
tools/repin_us_spec_engine_contracts.py, tools/spec_engine_coverage.py and
the repin again (--check clean) repinned them for this branch's qrf.py, and
ruff format restored the double quotes the repin tool drops. No
microcosm-fit source or fit.qrf parity pin changes in the merge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An in-session pre-review found that a fit does reach
_class_balanced_weights with a class whose weight total overflows to
infinity. Each weight is finite, so the fit's weight checks take it, and
_folds_carry_every_class counts inf as weight. Main completes that fit
(cross_fitted_class_rates). 94ff4a7 made it raise, and its message, the
changelog and the PR body said no fit reached the refusal. The brief says
not to change any fit whose classes all carry weight, so the helper now
refuses only a total that is not positive: the test _folds_carry_every_class
runs on the same totals first, so no fit meets the refusal. Guards pin an
overflowing class in the helper and in a full fit against the previous
helper, bit for bit; both fail on b90f098.

Tests:
- The grouped-folds property checks that the rarest class sits in every
  fold. It draws households at the can_deal_folds bound, with boundary
  examples. A dealing by the commonest class and a random dealing now fail
  it; both passed before.
- The out_of_fold_proba property gains examples for the boosted-only and
  spline-only models on a single-class fold, and for folds that each hold
  one class.

Wording:
- The whole gate falls back, not "other folds".
- The reach claim holds for targets with two classes or more.
- Every array holds a distribution in every row a fold scores.
- The scorer's zero-row hint no longer names a cause out_of_fold_proba
  removed.

Repinned the attested fit.qrf hashes: tools/repin_us_spec_engine_contracts.py,
tools/spec_engine_coverage.py (41/41 inventory checks), the repin again,
--check clean, then python -m tools.graph_parity_repin fit.qrf. ruff format
restores the double quotes the repin tool drops.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The round-1 Subfleet review (job 20261114-143433-fit-fold-r1) found the
code fixes correct and valid input bit-identical, and flagged wording that
overstates guarantees for weights whose totals overflow float64, which the
fit accepts:

- out_of_fold_proba's docstring promised a distribution in every row of
  every array a fold scores, and "as the training folds' shares do"; a
  fold whose training weights overflow gives NaN marginal rows (and
  sometimes NaN boosted and blend rows), unchanged from main. The
  docstring now says only what the fill guarantees: in a single-class fold
  each learner's rows equal the model's, so none is left at zero.
- _class_balanced_weights' docstring, the changelog and the test module
  said an overflowing class's rows weigh zero. When every class's total
  overflows, every row's weight is NaN and the fit fails in scikit-learn
  ("Input sample_weight contains NaN."), on main as here; each now says so.

Docstrings, comments and the changelog only: qrf.py, held_out.py and the
test module are AST-equal to f3b489a with docstrings stripped.
Repinned the attested fit.qrf hashes: repin, coverage (41/41), repin,
--check clean, then graph_parity_repin fit.qrf; ruff format restores the
double quotes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…arners

The round-2 Subfleet review (job 20261115-223630-fit-fold-r2) raised no P0
or P1. Its one P2: the docstring said the model, the learners and the
constant differ only where a learner was fitted. In a single-class fold
whose training weights overflow float64, no learner is fitted, yet the
constant (training shares) is NaN while the model and the filled learners
give the class with certainty. The clause now covers the model and the
learners only, which holds by construction.

Docstring only; held_out.py's bytes are attested nowhere, so no repin.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis changed the base branch from fix/fit-input-validation-bug-parity to main November 17, 2026 00:15
…-degenerate-classes

#1543 landed on main as a merge commit, so this branch's copies of its
commits are already in main's history. The only conflicts were the
attested spec-engine hashes in four pin files. They took main's side, and
#1617's line in inventory_coverage.py's repin comment was restored. Then
tools/repin_us_spec_engine_contracts.py, tools/spec_engine_coverage.py
(42,401/42,401 fields, 41/41 inventory checks), the repin again and
--check (all exit 0), and python -m tools.graph_parity_repin fit.qrf.
The fit.qrf node keys are unchanged from 708d67e, because qrf.py is
unchanged. ruff format restores the double quotes.

Main's other change to microcosm-fit since #1617's base is #1543's
docstring edit to categorical_scores, which merged cleanly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis marked this pull request as ready for review November 17, 2026 00:34
@MaxGhenis
MaxGhenis merged commit 825843b into main Nov 17, 2026
12 checks passed
@MaxGhenis
MaxGhenis deleted the fix/fit-fold-degenerate-classes branch November 17, 2026 02:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant