Skip to content

Update tutorial to reflect Euclid DR1 SIR 1D spectra - #408

Merged
DeekshaMohanty merged 28 commits into
Caltech-IPAC:mainfrom
xoubish:dr1-notebook3
Nov 12, 2026
Merged

DeekshaMohanty merged 28 commits into
Caltech-IPAC:mainfrom
xoubish:dr1-notebook3

Conversation

@xoubish

@xoubish xoubish commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

What changed

  • DR1 text: rewrote the introduction for DR1 (EWS + EDS, ~1900 sq deg, reprocessed Q1 areas), cited the DR1 SIR paper (Scodeggio et al. 2026), and updated the learning goals.

  • New way of finding spectra: the notebook now

    1. finds the galaxy's object_id and spectroscopic redshift by joining the DR1 MER deep catalog with the SPE catalog (spe_rank = 0), and
    2. gets its spectra from euclid.objectid_spectrafile_association_dr1, building the spectrumDM URLs in the ADQL query (IRSA's recipe).

    The SSA search was removed because DR1 spectra aren't in SSA yet. A note says the notebook will switch back once they are.

  • Reaches the first plot sooner: one spectrum is plotted right after it's read in. The details come after that.

  • Red vs. blue grism: explains the two grisms (blue only in the Deep Survey), tells them apart by wavelength range, and picks the best spectrum of each grism by number of usable flux bins.

  • Plot with emission lines: the plot shows usable bins as solid lines, MASK-flagged bins as dotted lines, and the uncertainty as thin lines. Emission lines (e.g. Hα, [OIII]) are marked at the galaxy's redshift.

  • MASK column: explains which bins to discard (odd values or values ≥ 64) and plots MASK against wavelength.

  • Gallery description updated in notebook_metadata.yml.

Notes for reviewers

  • ⚠️ DR1 isn't public on the production services yet, so the notebook sets Irsa.tap_url to irsadev. That cell has a TODO and needs to be removed before (or right after) the DR1 public release. Please don't merge to production before then.

KeunhoKim-astro and others added 23 commits November 6, 2026 09:00
Removed unnecessary comment from the introduction section.
Updated text for clarity and accuracy regarding the Euclid mission and data releases.
Updated coordinates and search radius for galaxy spectrum query.
Updated comments in the code cell regarding the public release of dr1.
Updated the coordinates for the galaxy search in the 1D spectra table.
Updated coordinates for galaxy search in 1D spectra table.
Added steps for querying and plotting galaxy spectra.
Added steps to query and plot the spectrum of a specific galaxy using the IRSA SSA service. Included code cells for searching, reading, and visualizing the spectrum data.
Plot the spectrum with the most usable flux bins right after reading the
spectra, and move the grism and deep/wide tile explanation, the
best-spectrum selection, the redshift lookup and the emission-line plot
after it. Use the UNCERTAINTY column instead of sqrt(VAR), call
quantity_support in the imports cell, trim the learning goals, and note
that IRSA temporarily removed the DR1 SSA collection.
Add Euclid Collaboration: Scodeggio et al. (2026, in prep.) to the
introduction and a References section, and describe the DR1 version of
the notebook in notebook_metadata.yml.
…rvice

The DR1 SIR spectra will not be served by SSA/ObsTAP/CAOM at release, so
find the galaxy and its redshift with one TAP query on the MER and SPE
catalogs, look up its spectrum files and HDUs in
euclid.objectid_spectrafile_association_dr1, and read them through the
spectrumDM service. Drop the temporary SSA stand-in and the collection
listing, note the coming SSA access, and add the spe_z_rel reliability
cut to the random galaxy selection.
Use the irsacloud spectrumDM endpoint with the euclid_combspec_dr1 dataset,
and build the URL in the ADQL query on the association table, as
recommended by IRSA.
@xoubish

xoubish commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

@jkrick thank you :)

@jkrick
jkrick self-requested a review November 9, 2026 19:07
@jkrick

jkrick commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

Thanks for this update! The DR1 version reads really nicely, and the blue grism and emission line markers
are great additions. I ran the notebook against irsadev and it runs cleanly in about a minute.

Some of the items below go beyond the DR1 switch (two were already in the version on main, marked from
main
), but since the code changed a lot, this seemed like a good time to raise them.

Before release (Nov 12)

  1. Please add a release-day checklist to the top of this PR conversation so one person can do the release. It should cover: removing the irsadev Irsa.tap_url cell; confirming the DR1 table names on production TAP; confirming the spectrumdm URL and dataset_id; filling in the Scodeggio et al. reference (if available); and rerunning the notebook against production.
  2. Add numpydoc-style docstrings to all functions (none have one yet, e.g. find_galaxy).

Code

  1. search_radius is set but never used: call find_galaxy(coord, search_radius).
  2. get_best_spectra: catch OSError instead of HTTPError, so a timeout doesn't stop the section 7 loop.
  3. (from main) plot_spectra: mask the uncertainty on bad flux bins, as the section 3 plot already does.
  4. The section 7 loop variable galaxy overwrites the section 1 galaxy table; please rename it (e.g. random_galaxy).
  5. find_spectra: the ADQL query does two jobs: it looks up where each spectrum is stored, and it builds a
    web-service URL from that by gluing strings together with ||. It's easier to follow if the query onlyreturns the plain columns and Python builds the URL (e.g. a small spectrum_url(path, hdu) function with the service address as a named constant). The URLs should stay the same. Please add a sentence explaining what the spectrumdm service does.

Structure

  1. I think it would flow better if the MASK visualization were moved earlier, folding old section 6 into section 3 and keeping the first plot in section 3. Make the first plot a two-panel figure with a shared x-axis: the spectrum on top (bad bins dotted) and MASK against wavelength below. Show the spectrum table there too, and move the "bad bins are at the edges" sentence there. Drop the old section 6 MASK plot and renumber section 7 to 6.

Science and text

  1. Please add a caveat about the blue grism. In some random galaxies (e.g. 2709226019659198628), the blue
    grism between about 0.92 and 0.95 µm has large spikes that MASK doesn't flag. The blue spectrum also sits
    below the red one where they overlap (1.2–1.36 µm) in both that galaxy and the main one. A possible
    explanation for both should be documented.

Thanks again!

@jkrick jkrick left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't need to see it again, as long as the top two comments are addressed.

@xoubish

xoubish commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @jkrick for the careful review! All addressed in the latest commit:

  1. Release-day checklist: added at the top of the PR description.
  2. Docstrings: all functions now have numpydoc docstrings.
  3. find_galaxy(coord, search_radius) now uses search_radius.
  4. get_best_spectra catches OSError, so a timeout no longer stops the section 6 loop.
  5. plot_spectra now masks the uncertainty on bad flux bins.
  6. The loop variable is renamed random_galaxy.
  7. find_spectra now returns only the plain columns (path, hdu). The URL is built in Python by spectrum_url(path, hdu), with the service address and dataset ID as named constants (SPECTRUMDM_URL, SPECTRUM_DATASET_ID); the URLs are unchanged. I added a sentence explaining that spectrumDM reads a single HDU of a spectrum file in the cloud and returns it as a VOTable.
  8. Section 3 now shows the spectrum table, then a two-panel figure with a shared x-axis: the spectrum on top (bad bins dotted) and MASK below, followed by the "bad bins are at the edges" sentence. The old section 6 is gone, and the exploration section is now section 6.
  9. Blue grism: we'd rather not document SIR processing issues in the tutorial ourselves. The notebook follows the SIR DR1 paper (Scodeggio et al. 2026) for the MASK flags, and leaves known data limitations to that paper and the DR1 documentation.

A few other small updates:

  • The MASK note now cites the bit definitions in Scodeggio et al. (Table 2): our "odd or ≥ 64" rule corresponds to NOT_USE and ABS_FLUX.
  • The grism wavelength ranges and resolving powers follow the paper.
  • The Scodeggio et al. reference now uses the paper's current title.
  • The last section is renamed "Select galaxies and explore their spectra", since the galaxies are selected and only the final pick is random.

@troyraen troyraen added the content: euclid Content related issues/PRs for notebooks with Euclid relevance label Nov 12, 2026
@troyraen troyraen linked an issue Nov 12, 2026 that may be closed by this pull request
@DeekshaMohanty
DeekshaMohanty merged commit 386960d into Caltech-IPAC:main Nov 12, 2026
8 checks passed
github-actions Bot pushed a commit that referenced this pull request Nov 12, 2026
Update tutorial to reflect Euclid DR1 SIR 1D spectra 386960d
@xoubish
xoubish deleted the dr1-notebook3 branch November 13, 2026 18:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

content: euclid Content related issues/PRs for notebooks with Euclid relevance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Update notebooks for Euclid DR1

5 participants