Skip to content

Update notebook 5 (SPE catalogs) to Euclid DR1 - #413

Merged
DeekshaMohanty merged 23 commits into
Caltech-IPAC:mainfrom
xoubish:dr1-notebook5
Nov 12, 2026
Merged

DeekshaMohanty merged 23 commits into
Caltech-IPAC:mainfrom
xoubish:dr1-notebook5

Conversation

@xoubish

@xoubish xoubish commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Release-day checklist (DR1 public, Nov 12)

  • Remove the Irsa.tap_url = 'https://irsadev.ipac.caltech.edu/TAP' cell and its TODO comment
  • Confirm the DR1 table names on production TAP (euclid_dr1_mer_catalogue_deep_survey, euclid_dr1_phz_photo_z_deep_survey, euclid_dr1_spectro_z_spe_galaxy_candidates_deep, euclid_dr1_spectro_z_spe_quality_deep, euclid_dr1_spectro_line_features_spe_line_features_deep, euclid.objectid_spectrafile_association_dr1)
  • Confirm the spectrumDM URL and dataset_id=euclid_combspec_dr1 on production
  • Add the link to Le Brun et al. 2026 (DR1 SPE paper) if available, and remove the TODO
  • Rerun the notebook end to end against production
  • Recheck the text that describes specific results against the production rerun: "about 130 galaxies" pass all of the criteria (section 3.2), the 780 valid pixels / 14 dithers and Hα S/N = 15 of the galaxy failing criterion 3 (section 5.2), and the lines named in section 6 ([SII] S/N of about 11, marginal [OIII] and [SIII]9068, [NII]/Hα of about 0.6)
  • Update the Updated date

Summary

Updates notebook 5 (SPE catalogs) from Q1 to DR1, starting from Keunho's work on his fork (his commits are kept on this branch).

  • Uses the DR1 SPE tables (deep survey), joined on IRSA's server with the MER and PHZ catalogs; nothing is downloaded in bulk.
  • Section 1 describes the SPE tables and their columns.
  • Section 3 is a worked example of filtering the SPE tables, for galaxies with a strong Hα line, applying the criteria one at a time and showing how many galaxies each one removes.
  • Explains the line quality flag spe_line_qual_gf and how to use the single-grism and combined measurements.
  • Plots the redshifts and Hα fluxes, a gallery of six spectra that pass all the criteria, and examples of galaxies that fail one criterion.
  • Looks at all the line measurements of one galaxy, with a warning about line ratios (blended Hα+[NII], doublet fluxes).
  • Spectra are read through the association table and spectrumDM, as in notebook 3.

A note on the selection criteria

@jkrick, a heads-up before you review: getting a clean sample of good SPE spectra is not straightforward, so the selection in section 3 has quite a few conditions. That is on purpose. Keunho worked hard to stay as close as possible to the recipe in the DR1 SPE paper (Le Brun et al. 2026, Sect. 4.2), which is the sample they validate against DESI and by visual inspection, and the notebook says where it differs from it. I've edited the text as much as I could to make it easier to follow: each criterion is explained, the cut flow shows what each one removes, and the text says clearly that this is one strict recipe for strong Hα emitters, not a definition of a "good" spectrum.

The notebook runs end to end against irsadev in about 3 minutes. CI will fail until the notebook is switched to production, as for the other DR1 PRs.

And this is the last PR from me before the release, I promise :D

KeunhoKim-astro and others added 19 commits November 9, 2026 09:52
Added a link for further information about Euclid Data Release 1.
Correct formatting and remove unnecessary text in the introduction.
Updated section headings and modified table names for the Euclid SPE catalog.
Updated sections on searching Euclid data and modified the structure of the document for clarity.
Added example query for selecting objects in a specific sky region and specified conditions for searching objects with spectra.
Updated search region size and conditions for spectral object search in Euclid DR1.
Updated wording for clarity regarding search conditions based on recommendations in the Euclid DR1 spectroscopic paper.
Updated the criteria for selecting objects and clarified the join operation in the catalog. Added a note about searching within the first 10000 objects to avoid timeout issues.
Revised search criteria and improved query execution time by limiting the search to the first 10,000 objects. Updated object ID for analysis of H-alpha line detection.
Updated the spectrum collection name and adjusted the query radius. Modified the spectrum reading process to handle both red and blue spectra separately, including error handling and plotting adjustments.
Updated criteria for selecting spectra in the catalog to specify valid pixel count and median dithers.
Clarified the conditions for selecting objects in the search criteria, including specific quality flags and flux measurements.
Updated the condition for selecting objects based on spectroscopic redshift probability and reliability.

@jkrick jkrick left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this one too, and for carrying Keunho's work over the line! This is a large change from the notebook on main, and a large improvement over it: it goes from finding a single Hα emitter to showing how to build and check a clean sample from the SPE tables, which is much more useful to readers. The cut-flow approach works really well: seeing how many galaxies each criterion removes, and then looking at galaxies that fail just one criterion, makes a long selection easy to follow. The heads-up about the criteria helped too. I ran it against irsadev and it runs cleanly in about 3.5 minutes.

As with #408 and #409, some of these are about code shared between the notebooks, so it'd be great to keep all three consistent.

Before release (Nov 12)

  1. Thanks for the release-day checklist! Please add two items: rechecking the prose that describes specific results against the production rerun (items 3–5 below: "about 130 galaxies", the 780 pixels / 14 dithers in section 5.2, and the lines named in section 6), and updating the Updated date. I'll add the Le Brun et al. DR1 reference to #412, which tracks the other DR1 papers that aren't out yet.
  2. Add numpydoc-style docstrings to all functions (none have one yet: values, criteria, cut_flow, query_halpha, all_criteria, read_spectra, bad_flux_bins, grism_name, best_spectra, plot_spectra, plot_galaxy, fails_only).

Text that doesn't match the current output

  1. Section 6 (line 564): "lines such as [OIII] (in the blue grism) and [SII] are detected." [SII] is clearly there (S/N 11), but [OIII]5008 has S/N 4 and nothing is visible at 0.99 µm in the plot. [SIII]9068 is also in the table (S/N 4.5) and marked in the figure, with nothing visible at 1.79 µm, but the text doesn't mention it. Please say that these two are marginal measurements that pass the S/N > 3 cut, or raise the cut so the table and plot only show lines you can see.
  2. Section 5.2 (line 538): the left galaxy's Hα is "clearly seen in the red grism spectrum." In the plot it's a small bump at 1.66 µm that is hard to pick out from the noise. Please soften this, e.g. "Hα is detected in the red grism spectrum (S/N = 15)."
  3. Blue and red grism mismatch (section 5.1, lines 515–516): in many of these spectra the blue and red grism continua don't line up where they overlap (1.2–1.37 µm). Sometimes the blue sits well below the red (e.g. 2693651229659529207, where the blue grism is negative) and sometimes well above it (e.g. 2700911240661609111). There are also unflagged spikes in the blue grism (e.g. 2699399029659465354). This is the same thing raised on #408, and it is much more noticeable here because the notebook shows many spectra side by side. I understand that you don't want the notebook to go into the SIR processing that causes this, and I agree it shouldn't. But a reader who sees two grisms disagree by a factor of two will wonder whether they did something wrong. One or two sentences would cover it: say that the blue and red grism spectra can have different continuum levels and that the blue grism can have unflagged artifacts, and point to Scodeggio et al. 2026 and the DR1 documentation for the details. The same sentence could go in notebook 3 (#408) for consistency.

Code

  1. read_spectra (lines 418–426): same as #408 item 7. Please drop the || URL building from the ADQL and use the same spectrum_url(path, hdu) function and SPECTRUMDM_URL / SPECTRUM_DATASET_ID constants that #408 now uses. Update the text on line 414 ("whose URL we build in the query") to match.
  2. bad_flux_bins, grism_name, and best_spectra are copied from notebook 3. Please keep them identical to the final #408 versions. Also, as in #408 item 4, read_spectra has no error handling: one spectrum that times out in the 20-galaxy loop in section 5.1 stops the whole figure. Catching OSError (and skipping that spectrum or galaxy) would fix that.
  3. Empty selections: rejected[0] (line 532) raises an IndexError if no galaxy fails only that criterion, and strong_halpha[0] (line 548) raises one if nothing passes all the criteria. This is fine on today's data, but it could break on production or for a reader who changes the region. A clear message would be friendlier (as in #409 item 5).
  4. Variable names: galaxy is the ADQL alias for the redshift table, the loop variable in 5.1, the parameter of plot_galaxy, and the section 6 galaxy. Please rename the loop variable (e.g. example) and the section 6 one (e.g. brightest_galaxy), to match #408 item 6.
  5. Section 6 plot (lines 570–577): the line labels come straight from the catalog (Halpha+NII6585, OIII5008, SII6718), unlike the Greek labels everywhere else, and Halpha+NII6585 collides with the spectrum. A small mapping to Hα+[NII], [OIII], … would make it consistent. The measured_lines dict comprehension can also be simplified to dict(zip(galaxy_lines['spe_line_name'], u.Quantity(galaxy_lines['spe_line_central_wl_gf']))).
  6. Small items:
    • Imports: move import warnings up with the other imports, and fix #suppress → # Suppress.
    • values() is a very generic name for a helper used all over the notebook; something like as_float would make the criteria easier to read.

Length and structure

  1. Section 5.1 gallery: 20 panels is too many. At this size I can't read the line labels or see the lines in most panels, and reading 40+ spectra adds a minute or two to the run time (the notebook's own estimate). I Suggest cutting it to about 6 panels (e.g. 2×3) at a size where the lines and labels can be read. They could still span the range of Hα S/N, and it would be nice to include at least one where Hα falls in both grisms.
  2. Overall length: the notebook is about 3–4 times the length of the version on main. I think the content is worth keeping, so I'm not asking for large cuts, but some of it could be tightened:
    • The six criteria (lines 218–223) are long paragraphs full of parenthetical justifications. A compact table (criterion | columns and thresholds | why) in the main text, with the comparisons to the paper (0.99 vs 0.94, the 98% figure, the GC SWG origin of criterion 4, the Wide Survey variant) might make the recipe much easier to scan.
    • Section 1 describes the classification table, which the notebook never uses. One line would be enough.
    • The paragraph before the criteria (lines 213–216) and the one after the cut flow (lines 368–369) repeat some of the "this is a strict recipe" caveats. Saying it once, clearly, is enough.
  3. Hide the helper function cells. Once the docstrings from item 2 are in, the function cells will get even longer and make the notebook harder to skim. The OpenUniverse2024 notebooks already collapse helper functions, with a header that works in both JupyterLab and the rendered HTML (e.g. tutorials/simulated-data/OpenUniverse2024/openuniverse2024_quickstart.md, show_gallery):
    ```{code-cell} ipython3
    ---
    jupyter:
      source_hidden: true
    tags: [hide-cell]
    ---
    def plot_spectra(...):
    
    Good candidates are the plumbing and plotting helpers: values, cut_flow, bad_flux_bins, grism_name, best_spectra, read_spectra/spectrum_url (the notebook already points to notebook 3 for these), plot_spectra, plot_galaxy, and fails_only. I'd keep criteria/all_criteria and query_halpha visible, since they are the recipe and the ADQL the notebook is teaching. Several of these cells also call the function at the end (e.g. cut_flow(criteria(candidates)), the query_halpha call). Please split those, so that only the definitions are hidden and every result is still produced by a visible cell. A short line before the first hidden cell ("the helper functions below are collapsed; click to expand") would help readers who haven't seen this before.

Figures

  1. Legends: as in #409 item 6, the "lower right" legend covers the end of the red spectrum in the section 5.2 and section 6 figures. Moving it outside the axes, as #408 does, fixes that.
  2. Optional: the section 6 table shows [NII]6585/Hα ≈ 0.6 for this galaxy, which nicely illustrates the warning below it. A sentence pointing that out would tie the warning to the data.

Thanks again! No need to send this back to me for another review: once items 1–5 and 14 are done (checklist additions, docstrings, the text fixes, the grism note, and hiding functions), it's good to go. The rest are suggestions, so use your judgment on those, or open a follow-up issue for anything you'd rather not do in this PR.

@troyraen troyraen added the content: euclid Content related issues/PRs for notebooks with Euclid relevance label Nov 12, 2026
@troyraen troyraen linked an issue Nov 12, 2026 that may be closed by this pull request
@DeekshaMohanty
DeekshaMohanty merged commit 396ac0f into Caltech-IPAC:main Nov 12, 2026
2 checks passed
github-actions Bot pushed a commit that referenced this pull request Nov 12, 2026
Update notebook 5 (SPE catalogs) to Euclid DR1 396ac0f
@xoubish
xoubish deleted the dr1-notebook5 branch November 13, 2026 18:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

content: euclid Content related issues/PRs for notebooks with Euclid relevance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Update notebooks for Euclid DR1

5 participants