Skip to content

Update the PHZ catalog notebook to Euclid DR1 - #409

Merged
DeekshaMohanty merged 7 commits into
Caltech-IPAC:mainfrom
xoubish:dr1-notebook4
Nov 12, 2026
Merged

DeekshaMohanty merged 7 commits into
Caltech-IPAC:mainfrom
xoubish:dr1-notebook4

Conversation

@xoubish

@xoubish xoubish commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

What changed

  • DR1 text and tables: rewrote the introduction for DR1. The notebook now describes the four DR1 PHZ tables (wide/deep photo_z and classification) and uses the deep survey PHZ and MER tables. It also notes the renamed interval columns (phz_90_int_1/_2; Q1 used phz_90_int1/2).
  • Starts from the catalog: the notebook begins with the PHZ+MER join, not the mosaic search.
  • New galaxy selection: galaxies in a 5′ box in EDF-North with flux > 0 in all four Euclid bands, phz_classification = 2, phz_flags = 0, a narrow 90% PDF interval, and phz_median between 0.85 and 1.85 (the range where Hα falls in the red grism).
  • New comparison with spectroscopic redshifts: joins the DR1 SPE catalog (spe_rank = 0) and keeps only reliable spec-z (spe_z_prob > 0.99, spe_z_rel > 0.05), then plots photo-z against spec-z.
  • Mosaic cutout: searches the DR1 deep MER mosaic collection with SIA and reads the VIS cutout straight from the cloud (S3) without downloading the full mosaic.
  • Spectrum of a bright galaxy: finds its spectra in euclid.objectid_spectrafile_association_dr1 and builds the spectrumDM URLs in the ADQL query (IRSA's recipe), because DR1 spectra won't be in SSA at release. It keeps the best red and blue grism spectra and marks Hβ, [OIII] and Hα at the spec-z. This matches the approach in notebook 3.
  • Firefly section kept. It now overlays the joined photo-z/spec-z table.
  • Gallery description updated in notebook_metadata.yml, and the DR1 PHZ paper (Tucci et al. 2026) added to the references.

Notes for reviewers

  • ⚠️ DR1 isn't public on the production services yet, so the notebook points Irsa.tap_url and Irsa.sia_url at irsadev. That cell has a TODO and must be removed when DR1 goes public. Please don't merge to production before then.
  • The PHZ paper reference is a placeholder ("in prep."), marked with a TODO to fill in the journal and arXiv details once the paper is out.

Use the DR1 deep survey PHZ and MER tables (with the renamed interval
columns), describe the four DR1 PHZ tables, and start with the catalog
rather than the mosaic search. Add a comparison of the photometric
redshifts with the DR1 SPE spectroscopic redshifts, using spe_z_prob and
spe_z_rel to identify the reliable ones, and plot the spectrum of the
brightest galaxy with a reliable spectroscopic redshift, retrieved by
position with the SSA service, with its emission lines. Read the mosaic
cutouts from the cloud and update the gallery description.
…tion table

Widen the photometric redshift range to the one where Halpha falls in the
red grism, plot only the reliable spectroscopic redshifts against the
photometric ones, and retrieve the spectra of the example galaxy through
euclid.objectid_spectrafile_association_dr1 and the spectrumDM service,
since the DR1 spectra will not be in SSA/CAOM at release.
Use the irsacloud spectrumDM endpoint with the euclid_combspec_dr1 dataset,
and build the URL in the ADQL query on the association table, as
recommended by IRSA.
@xoubish

xoubish commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

@jkrick thank you :)

@jkrick
jkrick self-requested a review November 9, 2026 20:42

@jkrick jkrick left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this one too! The DR1 PHZ notebook flows really well, and the new photo-z vs spec-z comparison is a great addition. I ran it against irsadev and it runs cleanly in about a minute; reading the cutout straight from S3 works nicely.

As with #408, a few of these go beyond the DR1 switch (marked from main), but since the code changed a lot it seemed like a good time to raise them. Several also match comments on #408, since the two notebooks share code; it'd be great to keep them consistent.

Before release (Nov 12)

  1. Please add a release-day checklist to the top of this PR conversation so one person can do the release. It should cover: removing the irsadev Irsa.tap_url / Irsa.sia_url cell (lines 128–132); confirming the DR1 table, SIA collection, and S3 bucket names on production; filling in the Tucci et al. reference (line 476); rerunning the notebook against production; rechecking the prose that describes specific results (items 3 and 4); and checking the Firefly output in section 6. Right now Firefly shows the table and overlay but fails to load the DR1 VIS mosaic ("could not retrieve image render data"), while the Q1 mosaic of the same field loads fine, so it should work once DR1 is public, but it's worth confirming that the image loads.
  2. Add numpydoc-style docstrings to all functions (bad_flux_bins and grism_name have comments instead).

Text that doesn't match the current output

  1. Section 5 text (lines 425–426) doesn't match what the notebook now shows. The brightest galaxy with a spectrum (2678621437659721934, z = 0.894) has only a blue grism spectrum. Hα is clear at 1.24 µm, but there's no visible [OIII]: it falls at the noisy blue edge, 0.95 µm. There's also no narrow spike near 1.3 µm. Please update the text to match what the plot shows, and recheck it after the release-day rerun.
  2. Line 352: "not every galaxy with a spectroscopic redshift has a spectrum listed in it yet" — the "yet" suggests a temporary state. Please confirm on production and reword or remove it.

Code

  1. Spectrum loop (lines 355–366): if no galaxy has a spectrum listed, the loop ends silently on the last galaxy and the notebook plots an empty figure. If no galaxy has a reliable redshift, galaxy is undefined. A for ... else that prints a clear message (or raises) would handle both.
  2. Spectrum plot (lines 410–422): only mark emission lines inside the plotted wavelength range (as #408's plot_spectra does). The legend in the upper left currently covers the Hβ and [OIII] labels; moving it outside the axes, as in #408, fixes that.
  3. Lines 357–359: same as #408. The ADQL query both looks up where each spectrum is stored and builds a web-service URL by gluing strings together with ||. It's easier to follow if the query returns only the plain columns and Python builds the URL. Please use the same approach in both notebooks.
  4. bad_flux_bins, grism_name, and the best-spectrum selection are copied from notebook 3. Please keep them identical to whatever #408 ends up with.
  5. Section 3 (lines 232–238) builds an IN (...) list of all 417 object IDs. For larger regions this list gets very long. A LEFT JOIN to the SPE catalog in the section 2 query (with spe_rank = 0 in the ON clause) would avoid the second query and the Python join.
  6. Small items:
    • The loop variable galaxy (line 355) could get a more specific name, to match #408.
    • tileID (line 297) is now only printed. (from main, where it was used for a search) Remove it, or
      say why it's useful.

Figures

  1. Section 4 figure (partly from main): with the wider redshift range, the cutout now has 417 red circles and it's hard to get much out of it. Could you plot only the 81 galaxies with a reliable spectroscopic redshift, as open circles colored by spe_z (with a colorbar), and mark the galaxy used in section 5? That ties sections 3–5 together. Use a colormap without a dark end (e.g. cool) so the circles stand out on the image.

Thanks again! No need to send this back to me for another review: once items 1–4 are done (the release-day checklist, docstrings, and the two text fixes). The rest are suggestions, so use your judgment on those, or open a follow-up issue for anything you'd rather not do in this PR.

@xoubish

xoubish commented Nov 10, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @jkrick! All addressed in the latest commit:

  1. Release-day checklist: added at the top of the PR description, including the result-dependent text and the Firefly check.
  2. Docstrings: all functions now have numpydoc docstrings (the spectrum helpers are now identical to notebook 3, see 8).
  3. Section 5 text: the galaxy you picked up had only a blue spectrum because IRSA's association table is missing all the red-grism files of its tile (101834041; reported to IRSA). I moved the region to the neighbouring tile 101834044, where both grisms are listed. The galaxy is now 2693942861660307299 (z_spec = 0.991, z_phot = 0.95), with Hα at 1.30 µm in both grisms, and the text describes what the plot shows. The sentence about the spike near 1.3 µm is gone.
  4. "yet": all 73 galaxies with a reliable spectroscopic redshift in the new region have both spectra listed in the association table, so I removed "yet". The loop still moves on to the next galaxy if one has no spectrum, as a safeguard.
  5. The spectrum loop now uses for ... else and raises a clear error (suggesting a larger region or a different redshift range) if none of the galaxies with a reliable redshift has a spectrum listed; section 4 also raises a clear error if no galaxy has a reliable spectroscopic redshift, so nothing is left undefined.
  6. The plot only marks lines within the wavelength range of the usable flux bins, and the legend is outside the axes, as in Update tutorial to reflect Euclid DR1 SIR 1D spectra #408.
  7. The association query returns only the plain columns, and spectrum_url(path, hdu) builds the URL in Python, with the same constants and the same sentence on spectrumDM as notebook 3.
  8. spectrum_url, find_spectra, bad_flux_bins, grism_name, and best_spectra are now identical to notebook 3, including docstrings.
  9. The section 2 query now has a LEFT OUTER JOIN to the SPE catalog, with spe_rank = 0 in the ON clause. The IN (...) query and the Python join are gone, and the counts are the same as with the two separate queries.
  10. The loop variable is bright_galaxy, and tileID is removed.
  11. The section 4 figure now shows only the 73 galaxies with a reliable spectroscopic redshift, as open circles colored by spe_z (cool colormap) with a colorbar, and a star marks the galaxy used in section 5. The red grism spectrum is drawn on top of the blue one in section 5, as it is less noisy where they overlap.

I also checked the notebook against the two DR1 PHZ papers (Hartley et al. 2026; Enia et al. 2026) and fixed a few things: the references now point to these DR1 papers instead of the Q1 one (Tucci et al. is now cited as the Q1 PHZ paper); the description of phz_mode_1/phz_mode_2 and of the unified fluxes now follows the paper; phz_classification only has star/galaxy in DR1 (no QSO or globular cluster classes); and section 3 now prints the photo-z metrics (NMAD 0.038, 8% outliers in the new region, consistent with the paper for EDF-N) and notes that only 63% of the spec-z fall within the 90% PDF interval, as the paper explains the deep-field PDFs are too narrow in DR1.

@troyraen troyraen added the content: euclid Content related issues/PRs for notebooks with Euclid relevance label Nov 12, 2026
@troyraen troyraen linked an issue Nov 12, 2026 that may be closed by this pull request
@DeekshaMohanty
DeekshaMohanty merged commit 73c2003 into Caltech-IPAC:main Nov 12, 2026
2 checks passed
github-actions Bot pushed a commit that referenced this pull request Nov 12, 2026
Update the PHZ catalog notebook to Euclid DR1 73c2003
@xoubish
xoubish deleted the dr1-notebook4 branch November 13, 2026 18:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

content: euclid Content related issues/PRs for notebooks with Euclid relevance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Update notebooks for Euclid DR1

4 participants