Skip to content

fix: Allow NaN in bounded float columns when allow_nan is set - #423

Merged
Oliver Borchert (borchero) merged 1 commit into
Quantco:mainfrom
aarab-malik:float-nan-bounds
Nov 11, 2026
Merged

Oliver Borchert (borchero) merged 1 commit into
Quantco:mainfrom
aarab-malik:float-nan-bounds

Conversation

@aarab-malik

Copy link
Copy Markdown
Contributor

Motivation

Float(allow_nan=True) still rejects NaN once a bound is set. Polars ranks NaN above every number, so NaN <= max is false and the max / max_exclusive rules fail even though NaN is allowed. NaN >= min happens to pass today, but that depends on the same ordering, so min and min_exclusive have the same bug waiting. allow_nan should be the only switch for NaN.

I also see it in sampling. test_sample_unchecked_nan draws NaNs for Float(min=0, max=10, allow_nan=True), but Schema.sample() filters those rows back out through the max rule, so the checked sampler returns no NaN at all.

Closes #416.

Changes

  • In _BaseFloat.validation_rules, when allow_nan is set, I OR expr.is_nan() into each bound rule (min, min_exclusive, max, max_exclusive). Finite values outside the bound are still rejected. NaN is still rejected when allow_nan is false, by the nan rule.
  • Added regression tests for inclusive and exclusive min and max bounds, with allow_nan=True and allow_nan=False.
  • Checked the other paths that reuse these rules. Schema sampling of a bounded float column keeps NaNs when allow_nan is set, and a list of such floats accepts an inner NaN. Unchecked sampling was already consistent and still is.

Bound comparisons reject NaN, so allow_nan was ignored once min or max
was set. Accept NaN in those rules and leave the decision to allow_nan.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

There are no active review issues for the change allowing NaN in bounded float columns when explicitly enabled.

No open issues 🎉

What changed in this PR

Fixes #416 by allowing NaN values in bounded float columns when allow_nan=True, keeping validation and checked sampling consistent.

Changes:

  • Makes all four float bound rules accept NaN when explicitly allowed.
  • Adds parameterized regression tests for bounds with NaN allowed and disallowed.
  • Covers NaN preservation in checked sampling and validation of nested float lists.
File Description
tests/​column_types/​test_float.py Adds regression coverage for bounded NaN validation, sampling, and lists.
dataframely/​columns/​float.py Allows NaN through bound rules when allow_nan=True.

Review effort: Balanced

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, thank you!

@codecov

codecov Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (58a2402) to head (e672cbf).

Additional details and impacted files
@@            Coverage Diff            @@
##              main      #423   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           46        46           
  Lines         2584      2587    +3     
=========================================
+ Hits          2584      2587    +3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@borchero
Oliver Borchert (borchero) merged commit 7f68841 into Quantco:main Nov 11, 2026
32 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Float(allow_nan=True, max=...) rejects NaN via the max rule, but min lets it through

3 participants