Skip to content

Tomographic SACC writers: one SACC per catalogue version, every statistic per bin pair - #447

Merged
cailmdaley merged 20 commits into
developfrom
feat/tomographic-sacc-376
Oct 31, 2026
Merged

cailmdaley merged 20 commits into
developfrom
feat/tomographic-sacc-376

Conversation

@cailmdaley

@cailmdaley cailmdaley commented Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator

Every tomographic statistic now lands in one SACC file per catalogue version, and blinded tomographic catalogues are concealed rather than refused. This is the last step before tomographic cosmo_val can run on the blinded catalogues.

Relation to #402. #402 brought the tomographic pseudo-Cℓ measurement. Its output was per-pair FITS files, and cosmo_val refused to run it on a blinded catalogue (_refuse_blinded_tomography). This PR adds the concealed per-pair SACC: the same measurement, sealed and blinded where it is computed.

#306 did this for ξ± and its OneCovariance covariance. This PR does the same for pseudo-Cℓ, COSEBIs, pure-E/B and ρ/τ, and the assembly writes {version}_tomo.sacc next to {version}.sacc.

What changes

  • Writers take bin pairs. Every *_to_sacc writer takes (bins, payload) pairs, as xi_pairs_to_sacc does. Each bin has one source_i tracer carrying its n(z). The data vector is pair-major in OneCovariance's order (0,0), (0,1), …; the order within a pair is as before.
  • Tomographic pseudo-Cℓ are one multi-pair part, sealed inside calculate_pseudo_cl. The per-pair FITS files and _refuse_blinded_tomography are gone, and the measurement helpers are private, so raw spectra can't be returned from outside.
  • COSEBIs and pure-E/B are derived from the concealed tomographic integration-grid ξ± part. Their covariance is K C_ξ Kᵀ over the full multi-pair ξ± covariance (OneCovariance or jackknife), so cross-pair blocks are kept.
  • ρ/τ. ρ is written once. τ is written per source bin, with tracers (source_i, psf_stars).
  • Covariance order. namaster_cl_covariance puts the iNKA FITS into the part's order, as onecov_xi_covariance does for ξ±. Both read the order from the part's own point labels.
  • Workflow. The {tomo_suffix} wildcard now reaches pseudo-Cℓ, COSEBIs, pure-E/B, ρ/τ and assembly. An empty suffix leaves every existing path unchanged.
  • OneCovariance inputs per covariance. An entry that declares shear.tomo_bin_col gives its _tomo covariance the per-bin n(z) and cov_th.tomo (per-bin n_e, sigma_e), and its whole-sample covariance the whole-sample cov_th and a bin-summed n(z) (localrule nz_bin_sum). A _tomo covariance of such an entry without cov_th.tomo is refused at DAG build, so single whole-sample values never stand in for every bin. The per-bin n(z) are summed as CosmologyValidation.get_redshift sums them, so each bin's column must integrate to its weight fraction; a file whose every column integrates to one is refused.
  • Tomographic ρ reuse. The PSF-only ρ result is shared across source bins, and it is reused only when its cache records the current means configuration (Separate correlation means from the jackknife patch layout #416's check).

Analysis choices

No ASTRA spec covers the data-vector layout yet, so the choices are recorded here.

  • COSEBIs E×B covariance is zero, across and within pairs. This keeps cosmo_numba's single-pair convention, so the non-tomographic file is unchanged. The operator itself gives a small non-zero E×B block. B-only PTEs don't depend on it; a joint E+B χ² would.
  • The pseudo-Cℓ covariance keeps EE×EE, BB×BB and EB×EB across all pairs. Cross-polarisation blocks are zeroed, as in the non-tomographic file.
  • τ covariance is zero across source bins. That is an approximation, acceptable for a validation file; nothing downstream fits tomographic τ jointly.
  • COSEBIs and pure-E/B use the same fiducial scale cut for every pair. Per-pair scale cuts are out of scope here.

Validation

  • Non-tomographic outputs are unchanged. All seven SACC files built from a fixed synthetic catalogue on develop (2badbcd) and on this branch are bit-identical: tracers, point order, tags, means, covariances. These are ξ± reporting and integration, pseudo-Cℓ, COSEBIs, pure-E/B, ρ/τ, and the assembled file.
  • Covariance order is tested independently, with 3 bins and 6 pairs, against OneCovariance ξ±, NaMaster iNKA, jackknife ξ±, and the derived COSEBIs and pure-E/B covariances. Each test builds its permutation from the written SACC's labels, never from the writer's helper. The jackknife test recomputes estimate_multi_cov in reversed pair order. One assembly test checks that every statistic's block sits at its own rows and that blocks across statistics are zero.
  • The blind leak gate (Blind leak gate: no cosmo_val run on a blinded catalogue writes an unblinded value #445) now covers the tomographic statistics: pseudo-Cℓ, COSEBIs, pure-E/B, τ and _tomo.sacc. The Blind leak gate: no cosmo_val run on a blinded catalogue writes an unblinded value #445 tests that asserted refusal now assert concealment.
  • Test results: fast suite 650 passed (the 4 failures in test_io tmpdir and test_plot_workflow also fail on develop); workflow DAG 19 passed. Two Sol reviews raised a public raw-spectrum path, a KeyError in a fresh covariance job, and stale ρ reuse; all three are fixed here. Mutation checks confirm the extended leak gate rejects unsealed tomographic signal.

Layout of the tomographic analysis file (synthetic leak-gate run, 2 bins). The gate's ξ± and pseudo-Cℓ covariance inputs are diagonal stand-ins, so this shows where each block sits; the tests above check its values.

tomographic SACC layout

Blinding shift per bin pair on the tomographic pseudo-Cℓ_EE (synthetic twin catalogues, throwaway test blind). The shift grows with the pair's redshift because each pair is shifted with its own n(z). BB and EB are unchanged (|Δ| < 10⁻¹⁵).

tomographic concealment

The white τ₅+ rows in the layout figure are negative variances, about −2×10⁻⁹, that CovTauTh itself returns for one synthetic bin. The writer copies them unchanged. Real catalogues should be checked for the same.

Closes #376

🤖 Generated with Claude Code

https://claude.ai/code/session_01JK7o9FFN33ALe6iizjiJMa

cailmdaley and others added 15 commits October 30, 2026 01:10
Co-Authored-By: GPT-6.1 Terra <noreply@openai.com>
Co-Authored-By: GPT-6.1 Terra <noreply@openai.com>
Co-Authored-By: GPT-6.1 Terra <noreply@openai.com>
Co-Authored-By: GPT-6.1 Terra <noreply@openai.com>
Co-Authored-By: GPT-6.1 Terra <noreply@openai.com>
Co-Authored-By: GPT-6.1 Terra <noreply@openai.com>
Co-Authored-By: GPT-6.1 Terra <noreply@openai.com>
Co-Authored-By: GPT-6.1 Terra <noreply@openai.com>
Co-Authored-By: GPT-6.1 Terra <noreply@openai.com>
Co-Authored-By: GPT-6.1 Terra <noreply@openai.com>
Co-Authored-By: GPT-6.1 Terra <noreply@openai.com>
Co-Authored-By: GPT-6.1 Terra <noreply@openai.com>
The PSF-only ρ shared across source bins was reused whenever its FITS and
covariance existed, bypassing the measurement metadata check, so a cache
from other means settings was republished. Drop the patched-correlation
copy fed to estimate_multi_cov: TreeCorr builds jackknife realisations from
the per-patch results, which the means pass leaves intact.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JK7o9FFN33ALe6iizjiJMa
@cailmdaley
cailmdaley marked this pull request as ready for review October 30, 2026 03:01
Cail Daley and others added 5 commits October 31, 2026 00:40
…entry

A tomographic covariance reads cov_th.tomo (per-bin n_e, sigma_e) and the
entry's multi-bin n(z). A non-tomographic covariance of an entry that
declares shear.tomo_bin_col reads the whole-sample cov_th and the
bin-summed n(z) written by nz_bin_sum. Without this, such an entry's
non-tomographic covariance was silently built on the 4-bin n(z).

From run/tomo1030 (80d2bc2, 5e5f313), without #432's pair-count inputs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JK7o9GEqa7euoocPk5aSTT
For an entry that declares shear.tomo_bin_col, a _tomo covariance without
cov_th.tomo got single whole-sample n_e and sigma_e standing in for every
bin. It now raises at DAG build.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JK7o9GEqa7euoocPk5aSTT
The whole-sample n(z) is the sum of the bins' columns, so each column
must carry its bin's weight fraction. A file whose every column
integrates to one would weight the bins equally; nz_bin_sum refuses it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JK7o9GEqa7euoocPk5aSTT
get_onecov_cat_params returns per-bin comma-separated strings for both
sigma_e and n_e.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JK7o9GEqa7euoocPk5aSTT
@cailmdaley
cailmdaley merged commit 31dd74e into develop Oct 31, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Tomographic SACC writers: one tracer and n(z) per bin

1 participant