Repository navigation
[RF] Support negative weights in kernel estimators - #23931
Open
Dev-next-gen wants to merge 1 commit into
Open
Dev-next-gen wants to merge 1 commit into
Dev-next-gen wants to merge 1 commit into
Conversation
The kernel widths of the RooKeysPdf were computed from the square root of the event weight, so a single negative weight made the whole lookup table NaN. The adaptive widths of the RooNDKeysPdf were computed from a fractional power of the signed pilot density, which is NaN wherever the negative weights win. Negative weights are common in HEP: sWeights from the sPlot technique, or simulated samples from NLO generators. The kernel widths now only depend on the magnitude of the weights, and the events with negative weights subtract their kernels from the estimate. Datasets with a non-positive sum of weights are now rejected with an exception, because they don't describe a distribution. For datasets without negative weights, the results don't change. Closes root-project#12639. Assisted-by: claude-code:claude-opus-5-5
Dev-next-gen
requested review from
dpiparo,
guitargeek and
hageboeck
as code owners
October 30, 2026 23:52
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This Pull request:
Changes or fixes:
RooKeysPdfandRooNDKeysPdfcould not be used with datasets that contain negative event weights, which are common with sWeights from the sPlot technique and in simulated samples from NLO generators:RooKeysPdf, the width of each kernel was computed fromsqrt(w * g(x)), so a single negative weight made the whole lookup table NaN. This is the 6.28 regression reported on the forum.RooNDKeysPdf, the widths were computed from a fractional power of the signed pilot density, which is NaN wherever the negative weights win.With this PR, the kernel widths only depend on the magnitude of the weights (
|w|inRooKeysPdf, and a pilot density built from|w|and normalized by the sum of|w|inRooNDKeysPdf), while each kernel is still scaled by its signed weight, so the estimate remainssum_j w_j K_j(x) / sum_j w_j. A dataset with a non-positive sum of weights, or with a weighted covariance that is not positive definite, now raisesstd::invalid_argumentinstead of silently producing NaNs, as suggested in the issue.For datasets without negative weights,
|w| == wand nothing changes: I compared the pdf values of 46 configurations (unweighted and positively weighted data, all mirror options, 1D and 2DRooNDKeysPdfwith several options) before and after, and they are bit-identical. The construction time changes by less than 3%.I validated it on a full sPlot chain: 30 toys, each with 4000 signal and 16000 background events. The signal template in a control variable is built from the signal sWeights (68% of the events have a negative sWeight), then used to fit the signal yield of an independent sample with 2000 true signal events. The uncertainties below are on the mean over the 30 toys.
RooKeysPdffrom sWeightsRooNDKeysPdffrom sWeightsRooKeysPdffrom positive sWeights only (workaround)RooKeysPdffrom the true signal events (reference)The remaining bias of the sWeighted
RooKeysPdftemplate, +76, is about the statistical uncertainty of a single fit, compared with +875 for the workaround. Note thatRooNDKeysPdfresults also change for datasets with negative weights that did not produce NaNs before, because the adaptive widths now come from the|w|pilot density. In this study its bias goes from -47 to +41 and the integrated squared difference to the true density improves by 18%.For an NLO-like sample with 14% of events of weight -1, the integrated squared difference to the true density is 4.0e-4 for
RooKeysPdfwith this PR (NaN before), compared to 3.7e-3 when dropping the negative weights. ForRooNDKeysPdfit goes from 3.7e-4 to 3.3e-4.What remains: where the weighted estimate is negative,
RooKeysPdfstill evaluates the pdf as zero while the analytical integral includes the negative part. This was already the case with asymmetric mirroring, so I did not change it here. In a deliberately noisy sideband-subtraction example, the normalized pdf integrates to 1.007.The new test
RooKeysPdf.NegativeWeightsfails with NaN for both classes against the currentmasterlibrary and passes with this PR.RooKeysPdf.NonPositiveSumOfWeightschecks the new exception. The RooFit ctest suite (gtests, stressRooFit, RooFit tutorials) has the same results before and after, on a minimal build withroofitandhistfactoryon Linux. I did not test macOS, Windows or PyROOT.Checklist:
This PR fixes #12639
Claude Code (Anthropic's Claude Opus 5.5) wrote the code, the tests and the validation study.
Found by a defect-hunting pipeline I build and run (Dev-next-gen), using Claude Code with Anthropic's Claude Opus 5.5.
"Nothing this pipeline produces is submitted without human approval. The entire pull request — the code change, the tests, the documentation updates and this description — was reviewed by a human before the pipeline was allowed to submit it."
(@Dev-next-gen)