From 75b587c1d7f1d1ab80a081d43bbc0f4ac0f625f5 Mon Sep 17 00:00:00 2001 From: xillig <56161060+xillig@users.noreply.github.com> Date: Wed, 11 Nov 2026 05:56:54 +0100 Subject: [PATCH] docs: show the output estimates on a fitted preprocessor The output-budget example called estimate_output_shape() and estimate_memory() on an unfitted Preprocessor, which raises NotFittedError, and the page advised running them before a fit that overflow_policy="error" would reject. Both are fitted methods: the output width is only known after fit. The example now fits first, and the page explains how to size a large dataset up front: fit on a sample (the row count is read from the data passed to the estimate), or fit with overflow_policy="warn". --- docs/core_concepts/outputs_and_inspection.md | 35 +++++++++++++++----- 1 file changed, 27 insertions(+), 8 deletions(-) diff --git a/docs/core_concepts/outputs_and_inspection.md b/docs/core_concepts/outputs_and_inspection.md index 68f1989..ee02170 100644 --- a/docs/core_concepts/outputs_and_inspection.md +++ b/docs/core_concepts/outputs_and_inspection.md @@ -84,8 +84,9 @@ pre_blocks.transform(df)["num_age"].shape ```{tip} This is why a wide expansion (a spline or feature map with a large `output_dim`, or several expanded columns) can produce a much wider matrix than the input `DataFrame` had columns. -Use `get_feature_info(verbose=True)` (below) or `estimate_output_shape(df)` to see the total -width before committing, especially with several expanded columns at once. +Use `get_feature_info(verbose=True)` (below) or `estimate_output_shape(df)` on the fitted +preprocessor to see the total width before committing, especially with several expanded columns +at once. ``` ## Output format and dtype @@ -442,16 +443,34 @@ The output budget lets you cap the blast radius and estimate cost before committ | `overflow_policy` | What to do on overflow, default `"error"`. | ```python -pre = Preprocessor(max_output_features=500, overflow_policy="error") +pre = Preprocessor(max_output_features=500, overflow_policy="error").fit(df, y) -pre.estimate_output_shape(df) # predicted (n_rows, n_cols) without transforming -pre.estimate_memory(df) # predicted dense memory in bytes +pre.estimate_output_shape(df) # (n_rows, n_cols) for df, without transforming it +pre.estimate_memory(df) # dense memory in bytes for df +``` + +Both estimates need a fitted preprocessor and raise `NotFittedError` before `fit`: the output +width is only known once the representations are fitted. The row count comes from the data you +pass, which is not transformed, so a preprocessor fitted on a sample can size the full dataset +before you transform it. + +```python +pre_sample = Preprocessor(numerical_method="bspline", output_dim=20).fit(df.head(1_000), y[:1_000]) + +pre_sample.estimate_output_shape(df) # width fitted on the sample, rows of the full df +pre_sample.estimate_memory(df) +``` + +```{note} +A width learned from the data, such as the categories of a one-hot block or an adaptive +`output_dim`, can differ between a sample and the full data, so treat a sample-based estimate +as approximate for those columns. ``` ```{warning} -With `overflow_policy="error"`, exceeding a budget raises `OutputBudgetError` at `fit`. Use -`estimate_output_shape` and `estimate_memory` first when you work with wide expansions or -large data. +With `overflow_policy="error"`, exceeding a budget raises `OutputBudgetError` at `fit`, before +the estimates are available. To explore the width of a wide expansion or a large dataset +first, fit on a sample as above, or fit with `overflow_policy="warn"` and read the estimates. ``` ## Where to go next