Repository navigation
Update notebook 5 (SPE catalogs) to Euclid DR1 - #413
Conversation
Added a link for further information about Euclid Data Release 1.
Correct formatting and remove unnecessary text in the introduction.
Updated section headings and modified table names for the Euclid SPE catalog.
Updated sections on searching Euclid data and modified the structure of the document for clarity.
Added example query for selecting objects in a specific sky region and specified conditions for searching objects with spectra.
Updated search region size and conditions for spectral object search in Euclid DR1.
Updated wording for clarity regarding search conditions based on recommendations in the Euclid DR1 spectroscopic paper.
Updated the criteria for selecting objects and clarified the join operation in the catalog. Added a note about searching within the first 10000 objects to avoid timeout issues.
Revised search criteria and improved query execution time by limiting the search to the first 10,000 objects. Updated object ID for analysis of H-alpha line detection.
Updated the spectrum collection name and adjusted the query radius. Modified the spectrum reading process to handle both red and blue spectra separately, including error handling and plotting adjustments.
Updated criteria for selecting spectra in the catalog to specify valid pixel count and median dithers.
Clarified the conditions for selecting objects in the search criteria, including specific quality flags and flux measurements.
Updated the condition for selecting objects based on spectroscopic redshift probability and reliability.
jkrick
left a comment
There was a problem hiding this comment.
Thanks for this one too, and for carrying Keunho's work over the line! This is a large change from the notebook on main, and a large improvement over it: it goes from finding a single Hα emitter to showing how to build and check a clean sample from the SPE tables, which is much more useful to readers. The cut-flow approach works really well: seeing how many galaxies each criterion removes, and then looking at galaxies that fail just one criterion, makes a long selection easy to follow. The heads-up about the criteria helped too. I ran it against irsadev and it runs cleanly in about 3.5 minutes.
As with #408 and #409, some of these are about code shared between the notebooks, so it'd be great to keep all three consistent.
Before release (Nov 12)
- Thanks for the release-day checklist! Please add two items: rechecking the prose that describes specific results against the production rerun (items 3–5 below: "about 130 galaxies", the 780 pixels / 14 dithers in section 5.2, and the lines named in section 6), and updating the Updated date. I'll add the Le Brun et al. DR1 reference to #412, which tracks the other DR1 papers that aren't out yet.
- Add numpydoc-style docstrings to all functions (none have one yet:
values,criteria,cut_flow,query_halpha,all_criteria,read_spectra,bad_flux_bins,grism_name,best_spectra,plot_spectra,plot_galaxy,fails_only).
Text that doesn't match the current output
- Section 6 (line 564): "lines such as [OIII] (in the blue grism) and [SII] are detected." [SII] is clearly there (S/N 11), but [OIII]5008 has S/N 4 and nothing is visible at 0.99 µm in the plot. [SIII]9068 is also in the table (S/N 4.5) and marked in the figure, with nothing visible at 1.79 µm, but the text doesn't mention it. Please say that these two are marginal measurements that pass the S/N > 3 cut, or raise the cut so the table and plot only show lines you can see.
- Section 5.2 (line 538): the left galaxy's Hα is "clearly seen in the red grism spectrum." In the plot it's a small bump at 1.66 µm that is hard to pick out from the noise. Please soften this, e.g. "Hα is detected in the red grism spectrum (S/N = 15)."
- Blue and red grism mismatch (section 5.1, lines 515–516): in many of these spectra the blue and red grism continua don't line up where they overlap (1.2–1.37 µm). Sometimes the blue sits well below the red (e.g. 2693651229659529207, where the blue grism is negative) and sometimes well above it (e.g. 2700911240661609111). There are also unflagged spikes in the blue grism (e.g. 2699399029659465354). This is the same thing raised on #408, and it is much more noticeable here because the notebook shows many spectra side by side. I understand that you don't want the notebook to go into the SIR processing that causes this, and I agree it shouldn't. But a reader who sees two grisms disagree by a factor of two will wonder whether they did something wrong. One or two sentences would cover it: say that the blue and red grism spectra can have different continuum levels and that the blue grism can have unflagged artifacts, and point to Scodeggio et al. 2026 and the DR1 documentation for the details. The same sentence could go in notebook 3 (#408) for consistency.
Code
read_spectra(lines 418–426): same as #408 item 7. Please drop the||URL building from the ADQL and use the samespectrum_url(path, hdu)function andSPECTRUMDM_URL/SPECTRUM_DATASET_IDconstants that #408 now uses. Update the text on line 414 ("whose URL we build in the query") to match.bad_flux_bins,grism_name, andbest_spectraare copied from notebook 3. Please keep them identical to the final #408 versions. Also, as in #408 item 4,read_spectrahas no error handling: one spectrum that times out in the 20-galaxy loop in section 5.1 stops the whole figure. CatchingOSError(and skipping that spectrum or galaxy) would fix that.- Empty selections:
rejected[0](line 532) raises anIndexErrorif no galaxy fails only that criterion, andstrong_halpha[0](line 548) raises one if nothing passes all the criteria. This is fine on today's data, but it could break on production or for a reader who changes the region. A clear message would be friendlier (as in #409 item 5). - Variable names:
galaxyis the ADQL alias for the redshift table, the loop variable in 5.1, the parameter ofplot_galaxy, and the section 6 galaxy. Please rename the loop variable (e.g.example) and the section 6 one (e.g.brightest_galaxy), to match #408 item 6. - Section 6 plot (lines 570–577): the line labels come straight from the catalog (
Halpha+NII6585,OIII5008,SII6718), unlike the Greek labels everywhere else, andHalpha+NII6585collides with the spectrum. A small mapping toHα+[NII],[OIII], … would make it consistent. Themeasured_linesdict comprehension can also be simplified todict(zip(galaxy_lines['spe_line_name'], u.Quantity(galaxy_lines['spe_line_central_wl_gf']))). - Small items:
- Imports: move
import warningsup with the other imports, and fix#suppress→# Suppress. values()is a very generic name for a helper used all over the notebook; something likeas_floatwould make the criteria easier to read.
- Imports: move
Length and structure
- Section 5.1 gallery: 20 panels is too many. At this size I can't read the line labels or see the lines in most panels, and reading 40+ spectra adds a minute or two to the run time (the notebook's own estimate). I Suggest cutting it to about 6 panels (e.g. 2×3) at a size where the lines and labels can be read. They could still span the range of Hα S/N, and it would be nice to include at least one where Hα falls in both grisms.
- Overall length: the notebook is about 3–4 times the length of the version on main. I think the content is worth keeping, so I'm not asking for large cuts, but some of it could be tightened:
- The six criteria (lines 218–223) are long paragraphs full of parenthetical justifications. A compact table (criterion | columns and thresholds | why) in the main text, with the comparisons to the paper (0.99 vs 0.94, the 98% figure, the GC SWG origin of criterion 4, the Wide Survey variant) might make the recipe much easier to scan.
- Section 1 describes the classification table, which the notebook never uses. One line would be enough.
- The paragraph before the criteria (lines 213–216) and the one after the cut flow (lines 368–369) repeat some of the "this is a strict recipe" caveats. Saying it once, clearly, is enough.
- Hide the helper function cells. Once the docstrings from item 2 are in, the function cells will get even longer and make the notebook harder to skim. The OpenUniverse2024 notebooks already collapse helper functions, with a header that works in both JupyterLab and the rendered HTML (e.g.
tutorials/simulated-data/OpenUniverse2024/openuniverse2024_quickstart.md,show_gallery):Good candidates are the plumbing and plotting helpers:```{code-cell} ipython3 --- jupyter: source_hidden: true tags: [hide-cell] --- def plot_spectra(...):values,cut_flow,bad_flux_bins,grism_name,best_spectra,read_spectra/spectrum_url(the notebook already points to notebook 3 for these),plot_spectra,plot_galaxy, andfails_only. I'd keepcriteria/all_criteriaandquery_halphavisible, since they are the recipe and the ADQL the notebook is teaching. Several of these cells also call the function at the end (e.g.cut_flow(criteria(candidates)), thequery_halphacall). Please split those, so that only the definitions are hidden and every result is still produced by a visible cell. A short line before the first hidden cell ("the helper functions below are collapsed; click to expand") would help readers who haven't seen this before.
Figures
- Legends: as in #409 item 6, the "lower right" legend covers the end of the red spectrum in the section 5.2 and section 6 figures. Moving it outside the axes, as #408 does, fixes that.
- Optional: the section 6 table shows [NII]6585/Hα ≈ 0.6 for this galaxy, which nicely illustrates the warning below it. A sentence pointing that out would tie the warning to the data.
Thanks again! No need to send this back to me for another review: once items 1–5 and 14 are done (checklist additions, docstrings, the text fixes, the grism note, and hiding functions), it's good to go. The rest are suggestions, so use your judgment on those, or open a follow-up issue for anything you'd rather not do in this PR.
Update notebook 5 (SPE catalogs) to Euclid DR1 396ac0f
Release-day checklist (DR1 public, Nov 12)
Irsa.tap_url = 'https://irsadev.ipac.caltech.edu/TAP'cell and its TODO commenteuclid_dr1_mer_catalogue_deep_survey,euclid_dr1_phz_photo_z_deep_survey,euclid_dr1_spectro_z_spe_galaxy_candidates_deep,euclid_dr1_spectro_z_spe_quality_deep,euclid_dr1_spectro_line_features_spe_line_features_deep,euclid.objectid_spectrafile_association_dr1)dataset_id=euclid_combspec_dr1on productionSummary
Updates notebook 5 (SPE catalogs) from Q1 to DR1, starting from Keunho's work on his fork (his commits are kept on this branch).
spe_line_qual_gfand how to use the single-grism and combined measurements.A note on the selection criteria
@jkrick, a heads-up before you review: getting a clean sample of good SPE spectra is not straightforward, so the selection in section 3 has quite a few conditions. That is on purpose. Keunho worked hard to stay as close as possible to the recipe in the DR1 SPE paper (Le Brun et al. 2026, Sect. 4.2), which is the sample they validate against DESI and by visual inspection, and the notebook says where it differs from it. I've edited the text as much as I could to make it easier to follow: each criterion is explained, the cut flow shows what each one removes, and the text says clearly that this is one strict recipe for strong Hα emitters, not a definition of a "good" spectrum.
The notebook runs end to end against irsadev in about 3 minutes. CI will fail until the notebook is switched to production, as for the other DR1 PRs.
And this is the last PR from me before the release, I promise :D