Skip to content

Separate correlation means from the jackknife patch layout - #416

Merged
cailmdaley merged 13 commits into
developfrom
fix/layout-independent-means
Oct 30, 2026
Merged

cailmdaley merged 13 commits into
developfrom
fix/layout-independent-means

Conversation

@cailmdaley

@cailmdaley cailmdaley commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator

Our published ξ± (and M_ap², ρ/τ) change when only the jackknife patch layout changes, by up to 0.8σ. This PR makes the published values independent of the patches. The patches are then used only for the covariance.

Why the patches leak into the means. TreeCorr groups galaxies into a tree of cells. When a pair of cells is small compared with the bin width (bin_slop), it assigns all their galaxy pairs to one bin, using the separation of the cell centres. With patches, the tree is built inside each patch, because cells never cross patch boundaries. A different patch layout gives different cells, so different galaxy pairs are approximated, and some land in neighbouring bins. Each bin's pair count, mean separation and ξ± move with the patches. Seeding the patches makes one layout reproducible, but doesn't remove the dependence.

The fix: two passes.

  • Means: measured without patches, with bin_slop tight enough (≤ 1, scaled to the bin width) that the approximation is negligible. These are the published ξ±, pair counts and separations.
  • Covariance: a second pass with patches, at TreeCorr's default bin_slop, used only for the jackknife covariance.

ξ±, M_ap² and the measured ρ/τ all use this, including cross-catalogue and tomographic pairs. The cache records both configurations, so older products, made with one shared pass, are not reused.

Effect on SP_v1.4.6.3 (1–250′, 20 bins, 100 patches): the difference in the means between two patch layouts drops from 0.32σ rms (0.81σ max) to 5×10⁻¹²σ, which is floating-point round-off.

ξ± mean difference between two patch layouts, before and after

Tests: synthetic layout tests give bit-identical means on one thread while the covariances differ, and reverting to patched means fails the ξ± and M_ap² regressions. Container suite, slow tests included: 537 passed (3 environment failures that also occur on the base).

Not covered here: how bin_slop itself affects the jackknife covariance (#401). The extra unpatched pass costs about 4 min at 8 threads on v1.4.6.3.

— Claude on behalf of Cail

🤖 Generated with Claude Code

cailmdaley and others added 10 commits October 5, 2026 15:01
A target b of 0.01 gives reporting measurements a common approximation policy while retaining bin_slop=1 on finer integration grids.

Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Tree approximations depend on the patch layout. A shared two-pass measurement keeps the full-sample means canonical while retaining patch results for covariance, including tomographic cross-pairs and aperture-mass integration. Patched text dumps cannot supply a covariance and are not reused.

Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
…changes

RhoStat and TauStat expose catalogue mappings and output tables. Measure the identical unpartitioned catalogues once, retain patched variances and joint covariance, and reuse the means across covariance draws. Record measurement settings so cached products cannot silently supply layout-dependent means.

Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
…try points

Reporting, fine-grid, and rho/tau jobs must use the configured target rather than silently falling back to a constructor default.

Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
…atched covariances

Synthetic auto/cross correlations and aperture-mass and rho/tau products retain identical single-thread means while their jackknife covariances differ. A patched-means mutation fails the layout test; fine grids retain bin_slop=1.

Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
… covariance jobs

Measurement calls retain the patch results needed for dense and derived covariances. Plotting explicitly reads the published columns and variances, at round-trip precision, so one-core figure jobs do not repeat the expensive measurement. A pinned root depth makes cached products reusable across thread counts.

Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
A direct producer rerun must retain a full jackknife covariance and patch results; cached columns are not a substitute for covariance.

Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
…he layout regression

The unpatched pass has no shared jackknife centres. Verify that it occurs once, while all three covariance draws retain distinct shared layouts and fresh catalogues.

Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Keep b_target on the published-means pass and let TreeCorr choose its standard bin_slop for patched covariance, where the tolerance does not set the reported means. Record both configurations so caches from the shared-tolerance policy are rejected.

Co-Authored-By: GPT-6 Luna <noreply@openai.com>
Apply the unpatched-means/patched-covariance policy within the current sealed-SACC measurement path. Keep SACC part reuse and derived aperture-mass covariance; do not restore unsealed xi text caches.
cailmdaley and others added 3 commits October 30, 2026 01:25
# Conflicts:
#	src/sp_validation/cosmo_val/real_space.py
#	src/sp_validation/tests/test_cosmo_val.py
#	workflow/scripts/run_2pcf.py
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JK7o9F5Ad2qR25uH6YFZZg
@cailmdaley
cailmdaley merged commit 2b5bede into develop Oct 30, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant