From 57b596eaa43b0d02a5cb86da2ecd969fd6b9897c Mon Sep 17 00:00:00 2001 From: KeunhoKim-astro <99432679+KeunhoKim-astro@users.noreply.github.com> Date: Mon, 12 Oct 2026 20:06:59 -0700 Subject: [PATCH 01/23] updated text in incorporate the dr1 updates --- tutorials/euclid/5_Euclid_intro_SPE_catalog.md | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/tutorials/euclid/5_Euclid_intro_SPE_catalog.md b/tutorials/euclid/5_Euclid_intro_SPE_catalog.md index d79f925b..589515f8 100644 --- a/tutorials/euclid/5_Euclid_intro_SPE_catalog.md +++ b/tutorials/euclid/5_Euclid_intro_SPE_catalog.md @@ -43,18 +43,16 @@ By the end of this tutorial, you will: ## Introduction +++ - -Euclid launched in July 2023 as a European Space Agency (ESA) mission with involvement by NASA. +Euclid launched in July 2023 as a European Space Agency (ESA) mission with contributions from NASA. The primary science goals of Euclid are to better understand the composition and evolution of the dark Universe. -The Euclid mission is providing space-based imaging and spectroscopy as well as supporting ground-based imaging to achieve these primary goals. -These data will be archived by multiple global repositories, including IRSA, where they will support transformational work in many areas of astrophysics. +The Euclid mission provides space-based imaging and spectroscopy, complemented by ground-based imaging, to achieve these goals. +These data will be archived by multiple global repositories, including IRSA, where they will support a wide range of astrophysical research. -Euclid Quick Release 1 (Q1) consists of consists of ~30 TB of imaging, spectroscopy, and catalogs covering four non-contiguous fields: -Euclid Deep Field North (22.9 sq deg), Euclid Deep Field Fornax (12.1 sq deg), Euclid Deep Field South (28.1 sq deg), and LDN1641. +Euclid Data Release 1 (DR1) will include imaging, spectroscopy, and catalogs covering approximately 1,900 square degrees of the Euclid Wide Survey, along with observations of the Euclid Deep Fields. +Among the DR1 data products are multiple spectroscopic catalogs produced by the Spectroscopic Analysis (SPE) Processing Function, including spectroscopic redshifts, emission-line measurements, and other derived spectral properties. -Among the data products included in the Q1 release are multiple catalogs created by the SPE Processing Function. -This notebook provides an introduction to these SPE catalogs. +This notebook provides an introduction to the Euclid DR1 SPE catalogs. If you have questions about this notebook, please contact the [IRSA helpdesk](https://irsa.ipac.caltech.edu/docs/help_desk.html). +++ From 6e10a71f9806a6b6a8fc6b10da1d7cb0c44c9570 Mon Sep 17 00:00:00 2001 From: KeunhoKim-astro <99432679+KeunhoKim-astro@users.noreply.github.com> Date: Tue, 13 Oct 2026 14:59:16 -0700 Subject: [PATCH 02/23] Update tutorial to reference Euclid DR1 SPE catalogs --- tutorials/euclid/5_Euclid_intro_SPE_catalog.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tutorials/euclid/5_Euclid_intro_SPE_catalog.md b/tutorials/euclid/5_Euclid_intro_SPE_catalog.md index 589515f8..7067949b 100644 --- a/tutorials/euclid/5_Euclid_intro_SPE_catalog.md +++ b/tutorials/euclid/5_Euclid_intro_SPE_catalog.md @@ -33,7 +33,7 @@ authors: +++ By the end of this tutorial, you will: -- Understand the basic characteristics of Euclid Q1 SPE catalogs. +- Understand the basic characteristics of Euclid DR1 SPE catalogs. - Understand what SPE catalogs are available and how to view the columns in those catalogs. - How to query with ADQL in the SPE lines catalog to find strong H-alpha detections. - How to make a plot the detected line features over the 1D spectra. From 1351293b6eba61b376778373a4bf2d982855a492 Mon Sep 17 00:00:00 2001 From: KeunhoKim-astro <99432679+KeunhoKim-astro@users.noreply.github.com> Date: Fri, 16 Oct 2026 09:16:50 -0700 Subject: [PATCH 03/23] Rename section from 'Euclid Q1' to 'Euclid DR1' --- tutorials/euclid/5_Euclid_intro_SPE_catalog.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tutorials/euclid/5_Euclid_intro_SPE_catalog.md b/tutorials/euclid/5_Euclid_intro_SPE_catalog.md index 7067949b..3423f450 100644 --- a/tutorials/euclid/5_Euclid_intro_SPE_catalog.md +++ b/tutorials/euclid/5_Euclid_intro_SPE_catalog.md @@ -24,7 +24,7 @@ authors: - name: Jessica Krick --- -# Euclid Q1: SPE Catalogs +# Euclid DR1: SPE Catalogs +++ From f559c22642f4a0f12c16cf18070df9b583bc7854 Mon Sep 17 00:00:00 2001 From: KeunhoKim-astro <99432679+KeunhoKim-astro@users.noreply.github.com> Date: Fri, 16 Oct 2026 09:46:08 -0700 Subject: [PATCH 04/23] Update Euclid DR1 information with additional link Added a link for further information about Euclid Data Release 1. --- tutorials/euclid/5_Euclid_intro_SPE_catalog.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tutorials/euclid/5_Euclid_intro_SPE_catalog.md b/tutorials/euclid/5_Euclid_intro_SPE_catalog.md index 3423f450..5125b052 100644 --- a/tutorials/euclid/5_Euclid_intro_SPE_catalog.md +++ b/tutorials/euclid/5_Euclid_intro_SPE_catalog.md @@ -48,7 +48,7 @@ The primary science goals of Euclid are to better understand the composition and The Euclid mission provides space-based imaging and spectroscopy, complemented by ground-based imaging, to achieve these goals. These data will be archived by multiple global repositories, including IRSA, where they will support a wide range of astrophysical research. -Euclid Data Release 1 (DR1) will include imaging, spectroscopy, and catalogs covering approximately 1,900 square degrees of the Euclid Wide Survey, along with observations of the Euclid Deep Fields. +Euclid Data Release 1 (DR1) will include imaging, spectroscopy, and catalogs covering approximately 1,900 square degrees of the Euclid Wide Survey, along with observations of the Euclid Deep Fields. Further information about Euclid DR1 can be found at: https://euclid.caltech.edu/page/euclid-dr1 Among the DR1 data products are multiple spectroscopic catalogs produced by the Spectroscopic Analysis (SPE) Processing Function, including spectroscopic redshifts, emission-line measurements, and other derived spectral properties. From e8255d57534a06922fa9d2f1c40a92050b47a3d3 Mon Sep 17 00:00:00 2001 From: KeunhoKim-astro <99432679+KeunhoKim-astro@users.noreply.github.com> Date: Thu, 5 Nov 2026 12:19:47 -0800 Subject: [PATCH 05/23] Fix formatting and text in Euclid intro catalog Correct formatting and remove unnecessary text in the introduction. --- tutorials/euclid/5_Euclid_intro_SPE_catalog.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/tutorials/euclid/5_Euclid_intro_SPE_catalog.md b/tutorials/euclid/5_Euclid_intro_SPE_catalog.md index 5125b052..d2a97530 100644 --- a/tutorials/euclid/5_Euclid_intro_SPE_catalog.md +++ b/tutorials/euclid/5_Euclid_intro_SPE_catalog.md @@ -104,9 +104,11 @@ warnings.filterwarnings( ) ``` -## 1. Find the MER Tile ID that corresponds to a given RA and Dec +## 1. Find the that corresponds to a given RA and Dec -In this case, choose the coordinates from the first notebook to save time downloading the MER mosaic. Search a radius of 1.5 arcminutes around these coordinates. +In this case, choose the coordinates from the first notebook to save time + +Search a radius of 1.5 arcminutes around these coordinates. ```{code-cell} ipython3 search_radius = 10 * u.arcsec From ea8ca9ad5d4fd6fd27ce3fddddbc4a69ab0b142b Mon Sep 17 00:00:00 2001 From: KeunhoKim-astro <99432679+KeunhoKim-astro@users.noreply.github.com> Date: Thu, 5 Nov 2026 12:22:29 -0800 Subject: [PATCH 06/23] Revise Euclid SPE catalog section headings and tables Updated section headings and modified table names for the Euclid SPE catalog. --- .../euclid/5_Euclid_intro_SPE_catalog.md | 22 +++++-------------- 1 file changed, 6 insertions(+), 16 deletions(-) diff --git a/tutorials/euclid/5_Euclid_intro_SPE_catalog.md b/tutorials/euclid/5_Euclid_intro_SPE_catalog.md index d2a97530..671ff21b 100644 --- a/tutorials/euclid/5_Euclid_intro_SPE_catalog.md +++ b/tutorials/euclid/5_Euclid_intro_SPE_catalog.md @@ -104,17 +104,6 @@ warnings.filterwarnings( ) ``` -## 1. Find the that corresponds to a given RA and Dec - -In this case, choose the coordinates from the first notebook to save time - -Search a radius of 1.5 arcminutes around these coordinates. - -```{code-cell} ipython3 -search_radius = 10 * u.arcsec -coord = SkyCoord.from_name('HD 168151') -``` - ```{tip} The IRSA SIA collections can be listed using using the ``list_collections`` method, we can filter on the ones containing "euclid" in the collection name: @@ -152,7 +141,7 @@ tileID = science_images[science_images['energy_bandpassname'] == 'VIS']['obs_id' print(f'The MER tile ID for this object is : {tileID}') ``` -## 2. Download SPE catalog from IRSA directly to this notebook +## 1. Download SPE catalog from IRSA directly to this notebook Search for all Euclid catalogs and metadata tables in IRSA: @@ -166,10 +155,11 @@ Irsa.list_catalogs(filter='euclid', include_metadata_tables=True) ``` ```{code-cell} ipython3 -table_mer = 'euclid_q1_mer_catalogue' -table_galaxy_candidates = 'euclid_q1_spectro_zcatalog_spe_galaxy_candidates' -table_1dspectra = 'euclid.objectid_spectrafile_association_q1' -table_lines = 'euclid_q1_spe_lines_line_features' +table_mer = 'euclid_dr1_mer_catalogue_deep_survey' +table_galaxy_candidates = 'euclid_dr1_spectro_z_spe_galaxy_candidates_deep' +table_quality = 'euclid_dr1_spectro_z_spe_quality_deep' +table_lines = 'euclid_dr1_spectro_line_features_spe_line_features_deep' +table_photo_z = 'euclid_dr1_phz_photo_z_deep_survey' ``` ### Learn some information about the table: From 08634e3f1997d83b23122381b5107ff9c0c6b119 Mon Sep 17 00:00:00 2001 From: KeunhoKim-astro <99432679+KeunhoKim-astro@users.noreply.github.com> Date: Thu, 5 Nov 2026 14:48:41 -0800 Subject: [PATCH 07/23] Revise Euclid data search instructions and structure Updated sections on searching Euclid data and modified the structure of the document for clarity. --- .../euclid/5_Euclid_intro_SPE_catalog.md | 37 ++++--------------- 1 file changed, 7 insertions(+), 30 deletions(-) diff --git a/tutorials/euclid/5_Euclid_intro_SPE_catalog.md b/tutorials/euclid/5_Euclid_intro_SPE_catalog.md index 671ff21b..032a0eb2 100644 --- a/tutorials/euclid/5_Euclid_intro_SPE_catalog.md +++ b/tutorials/euclid/5_Euclid_intro_SPE_catalog.md @@ -110,36 +110,7 @@ The IRSA SIA collections can be listed using using the ``list_collections`` meth Irsa.list_collections(filter='euclid') ``` -+++ - -### Use IRSA to search for all Euclid data on this target - -This searches specifically in the ``euclid_DpdMerBksMosaic`` collection which is the MER images and catalogs. - -```{code-cell} ipython3 -image_table = Irsa.query_sia(pos=(coord, search_radius), collection='euclid_DpdMerBksMosaic') -``` - -This table lists all MER mosaic images available in this search position. These mosaics include the Euclid VIS, Y, J, H images, as well as ground-based telescopes which have been put on the same pixel scale. For more information, see the [Euclid documentation at IPAC](https://euclid.caltech.edu/page/euclid-faq-tech/). - -Note that there are various image types are returned as well, we filter out the `science` images from these: - -```{code-cell} ipython3 -science_images = image_table[image_table['dataproduct_subtype'] == 'science'] -science_images -``` -### Choose the VIS image and pull the Tile ID - -+++ - -Extract the tile ID from the ``obs_id`` column. The values in this column are made a combination of the 9 digit tile ID and the abbreviation of the instrument. - -```{code-cell} ipython3 -tileID = science_images[science_images['energy_bandpassname'] == 'VIS']['obs_id'][0][:9] - -print(f'The MER tile ID for this object is : {tileID}') -``` ## 1. Download SPE catalog from IRSA directly to this notebook @@ -180,7 +151,13 @@ Irsa.list_columns(catalog=table_1dspectra, full=True) columns_info ``` -## 3. Find some objects with spectra in our tileID +## 2. Find objects within a specified sky region defined by RA and Dec ranges +Euclid DR1 covers a large sky area of approximately 1,900 square degrees. To reduce query execution time and avoid potential timeout issues, we recommend restricting searches to smaller sky regions. + +As an example, let's select objects within a 0.2-square-degree region of the Euclid Deep Field North (EDF-N). + + +## 3. Find some objects with spectra with conditions We specify the following conditions on our search: - Signal to noise ratio column (_gf = gaussian fit) should be greater than 5 From ba79a225aca9907c5b5a91c43af5c3f0a61c0749 Mon Sep 17 00:00:00 2001 From: KeunhoKim-astro <99432679+KeunhoKim-astro@users.noreply.github.com> Date: Thu, 5 Nov 2026 14:49:45 -0800 Subject: [PATCH 08/23] Add example query and search conditions in catalog Added example query for selecting objects in a specific sky region and specified conditions for searching objects with spectra. --- tutorials/euclid/5_Euclid_intro_SPE_catalog.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/tutorials/euclid/5_Euclid_intro_SPE_catalog.md b/tutorials/euclid/5_Euclid_intro_SPE_catalog.md index 032a0eb2..b5e4a82e 100644 --- a/tutorials/euclid/5_Euclid_intro_SPE_catalog.md +++ b/tutorials/euclid/5_Euclid_intro_SPE_catalog.md @@ -156,6 +156,24 @@ Euclid DR1 covers a large sky area of approximately 1,900 square degrees. To red As an example, let's select objects within a 0.2-square-degree region of the Euclid Deep Field North (EDF-N). +```{code-cell} ipython3 +query_mer = f""" +SELECT DISTINCT + object_id, + ra, + dec +FROM {table_mer} +WHERE + ra BETWEEN 269.34391 AND 270.12192 +AND dec BETWEEN 65.85958 AND 66.17581 +""" + +mer = Irsa.query_tap(query_mer).to_table() + +object_ids = mer["object_id", "ra", "dec"].as_array().tolist() +object_ids = [row[0] for row in object_ids] +``` + ## 3. Find some objects with spectra with conditions From c696265e91e9c795d26a67382f949e050465ee52 Mon Sep 17 00:00:00 2001 From: KeunhoKim-astro <99432679+KeunhoKim-astro@users.noreply.github.com> Date: Thu, 5 Nov 2026 15:13:51 -0800 Subject: [PATCH 09/23] Revise object search criteria and parameters Updated search region size and conditions for spectral object search in Euclid DR1. --- tutorials/euclid/5_Euclid_intro_SPE_catalog.md | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/tutorials/euclid/5_Euclid_intro_SPE_catalog.md b/tutorials/euclid/5_Euclid_intro_SPE_catalog.md index b5e4a82e..19628442 100644 --- a/tutorials/euclid/5_Euclid_intro_SPE_catalog.md +++ b/tutorials/euclid/5_Euclid_intro_SPE_catalog.md @@ -154,7 +154,7 @@ columns_info ## 2. Find objects within a specified sky region defined by RA and Dec ranges Euclid DR1 covers a large sky area of approximately 1,900 square degrees. To reduce query execution time and avoid potential timeout issues, we recommend restricting searches to smaller sky regions. -As an example, let's select objects within a 0.2-square-degree region of the Euclid Deep Field North (EDF-N). +As an example, let's select objects within a 0.1 square-degree region of the Euclid Deep Field North (EDF-N). ```{code-cell} ipython3 query_mer = f""" @@ -177,12 +177,15 @@ object_ids = [row[0] for row in object_ids] ## 3. Find some objects with spectra with conditions -We specify the following conditions on our search: -- Signal to noise ratio column (_gf = gaussian fit) should be greater than 5 -- We want to detect H-alpha. -- We choose in which tileID to search, usign the tileID from the first notebook. -- Choose spectroscopic redshift (spe_z) between 1.4 and 1.6 and spe_z_prob greater than 0.999 +We specify the following conditions on our search, as recommended in the Euclid DR1 spectroscopic paper: +- Signal to noise ratio column (_gf = gaussian fit) should be greater than 10 +- We want to detect H-alpha +- We select objects with well-measured H-alpha flux based on the spe_line_qual_gf quality flags +- Choose spectroscopic redshift (spe_z) between 0.85 and 1.85 and spe_z_prob greater than 0.999 +- We select objects whose spectroscopic redshifts (spe_z) fall within the 95% confidence intervals of their photometric redshifts (phz_95_int) +- Choose spectrum spe_npix and spe_n_dith_med greater than 15 - H-alpha line flux should be more than 2x10^16 erg s^-1 cm^-2 +- - Join the lines and galaxy candidates tables on object_id and spe_rank Finally we sort the data by descending spe_line_snr_gf to have the largest SNR H-alpha lines detected at the top. From 08fdf4f4e0993a8c6f1d082af936c84afd06fac2 Mon Sep 17 00:00:00 2001 From: KeunhoKim-astro <99432679+KeunhoKim-astro@users.noreply.github.com> Date: Thu, 5 Nov 2026 15:26:08 -0800 Subject: [PATCH 10/23] Clarify search conditions for object spectra Updated wording for clarity regarding search conditions based on recommendations in the Euclid DR1 spectroscopic paper. --- tutorials/euclid/5_Euclid_intro_SPE_catalog.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tutorials/euclid/5_Euclid_intro_SPE_catalog.md b/tutorials/euclid/5_Euclid_intro_SPE_catalog.md index 19628442..6ba99820 100644 --- a/tutorials/euclid/5_Euclid_intro_SPE_catalog.md +++ b/tutorials/euclid/5_Euclid_intro_SPE_catalog.md @@ -177,7 +177,7 @@ object_ids = [row[0] for row in object_ids] ## 3. Find some objects with spectra with conditions -We specify the following conditions on our search, as recommended in the Euclid DR1 spectroscopic paper: +We specify the following conditions on our search, based on recommendations in the Euclid DR1 spectroscopic paper: - Signal to noise ratio column (_gf = gaussian fit) should be greater than 10 - We want to detect H-alpha - We select objects with well-measured H-alpha flux based on the spe_line_qual_gf quality flags @@ -185,7 +185,7 @@ We specify the following conditions on our search, as recommended in the Euclid - We select objects whose spectroscopic redshifts (spe_z) fall within the 95% confidence intervals of their photometric redshifts (phz_95_int) - Choose spectrum spe_npix and spe_n_dith_med greater than 15 - H-alpha line flux should be more than 2x10^16 erg s^-1 cm^-2 -- +- We exclude point-like objects (primarily stars; mer.point_like_prob < 0.05), excessively large objects (mer.kron_radius < 50), and potentially spurious detections (mer.spurious_prob < 0.5). The size criterion helps minimize contamination from galaxies at $z < 0.85$ with incorrectly measured spectroscopic redshifts. - Join the lines and galaxy candidates tables on object_id and spe_rank Finally we sort the data by descending spe_line_snr_gf to have the largest SNR H-alpha lines detected at the top. From f7fa96fbfdc4520bbc35994833aa623506dac79c Mon Sep 17 00:00:00 2001 From: KeunhoKim-astro <99432679+KeunhoKim-astro@users.noreply.github.com> Date: Fri, 6 Nov 2026 08:13:36 -0800 Subject: [PATCH 11/23] Revise object selection criteria and join operations Updated the criteria for selecting objects and clarified the join operation in the catalog. Added a note about searching within the first 10000 objects to avoid timeout issues. --- tutorials/euclid/5_Euclid_intro_SPE_catalog.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tutorials/euclid/5_Euclid_intro_SPE_catalog.md b/tutorials/euclid/5_Euclid_intro_SPE_catalog.md index 6ba99820..c768baf1 100644 --- a/tutorials/euclid/5_Euclid_intro_SPE_catalog.md +++ b/tutorials/euclid/5_Euclid_intro_SPE_catalog.md @@ -186,8 +186,9 @@ We specify the following conditions on our search, based on recommendations in t - Choose spectrum spe_npix and spe_n_dith_med greater than 15 - H-alpha line flux should be more than 2x10^16 erg s^-1 cm^-2 - We exclude point-like objects (primarily stars; mer.point_like_prob < 0.05), excessively large objects (mer.kron_radius < 50), and potentially spurious detections (mer.spurious_prob < 0.5). The size criterion helps minimize contamination from galaxies at $z < 0.85$ with incorrectly measured spectroscopic redshifts. -- Join the lines and galaxy candidates tables on object_id and spe_rank +- Join the lines, mer, photometric redshift, and galaxy candidates tables on object_id and spe_rank +We search objects within the first 10000 objects to avoid the timeout issue withe the query which happens when the search takes long time. Finally we sort the data by descending spe_line_snr_gf to have the largest SNR H-alpha lines detected at the top. ```{code-cell} ipython3 From 637a361a952b14193af036880e0f63d0f29c6dc5 Mon Sep 17 00:00:00 2001 From: KeunhoKim-astro <99432679+KeunhoKim-astro@users.noreply.github.com> Date: Fri, 6 Nov 2026 08:16:59 -0800 Subject: [PATCH 12/23] Optimize object search and update analysis ID Revised search criteria and improved query execution time by limiting the search to the first 10,000 objects. Updated object ID for analysis of H-alpha line detection. --- .../euclid/5_Euclid_intro_SPE_catalog.md | 102 ++++++++++++++---- 1 file changed, 81 insertions(+), 21 deletions(-) diff --git a/tutorials/euclid/5_Euclid_intro_SPE_catalog.md b/tutorials/euclid/5_Euclid_intro_SPE_catalog.md index c768baf1..68cc5b3a 100644 --- a/tutorials/euclid/5_Euclid_intro_SPE_catalog.md +++ b/tutorials/euclid/5_Euclid_intro_SPE_catalog.md @@ -188,35 +188,95 @@ We specify the following conditions on our search, based on recommendations in t - We exclude point-like objects (primarily stars; mer.point_like_prob < 0.05), excessively large objects (mer.kron_radius < 50), and potentially spurious detections (mer.spurious_prob < 0.5). The size criterion helps minimize contamination from galaxies at $z < 0.85$ with incorrectly measured spectroscopic redshifts. - Join the lines, mer, photometric redshift, and galaxy candidates tables on object_id and spe_rank -We search objects within the first 10000 objects to avoid the timeout issue withe the query which happens when the search takes long time. +We limit our search to the first 10,000 objects to reduce query execution time and avoid potential timeout issues. Finally we sort the data by descending spe_line_snr_gf to have the largest SNR H-alpha lines detected at the top. ```{code-cell} ipython3 -adql_query = ("SELECT DISTINCT mer.object_id,mer.ra, mer.dec, mer.tileid, mer.flux_y_templfit, " - "lines.spe_line_snr_gf,lines.spe_line_snr_di, lines.spe_line_name, lines.spe_line_central_wl_gf, " - "lines.spe_line_ew_gf, galaxy.spe_z_err, galaxy.spe_z,galaxy.spe_z_prob, " - "lines.spe_line_flux_gf, lines.spe_line_flux_err_gf " - f"FROM {table_mer} AS mer " - f"JOIN {table_lines} AS lines " - "ON mer.object_id = lines.object_id " - f"JOIN {table_galaxy_candidates} AS galaxy " - "ON lines.object_id = galaxy.object_id AND lines.spe_rank = galaxy.spe_rank " - "WHERE lines.spe_line_snr_gf >5 " - "AND lines.spe_line_name = 'Halpha' " - f"AND mer.tileid = {tileID} " - "AND galaxy.spe_z_prob > 0.99 " - "AND galaxy.spe_z BETWEEN 1.4 AND 1.6 " - "AND lines.spe_line_flux_gf > 2E-16 " - "ORDER BY lines.spe_line_snr_gf DESC ") - -# Use TAP with this ADQL string -result_table = Irsa.query_tap(adql_query).to_table() +batch_size = 500 + +line_tables = [] +for i in range(0, 10000, batch_size): + + batch = object_ids[i:i + batch_size] + + ids = ", ".join(str(int(x)) for x in batch) + + query_lines = f""" + SELECT + lines.object_id, + lines.spe_rank, + lines.spe_line_name, + lines.spe_line_qual_gf, + lines.spe_line_central_wl_gf, + lines.spe_line_snr_gf, + galaxy.object_id, + galaxy.spe_z + FROM {table_lines} AS lines + JOIN {table_galaxy_candidates} AS galaxy + ON lines.object_id = galaxy.object_id + AND lines.spe_rank = galaxy.spe_rank + JOIN {table_quality} AS quality + ON lines.object_id = quality.object_id + JOIN {table_mer} AS mer + ON lines.object_id = mer.object_id + JOIN {table_photo_z} AS phz + ON lines.object_id = phz.object_id + + WHERE + lines.object_id IN ({ids}) + AND lines.spe_line_name = 'Halpha' + AND lines.spe_line_qual_gf IN (212, 222, 252) + AND lines.spe_line_snr_gf > 10 + AND lines.spe_rank = 0 + AND lines.spe_line_flux_gf > 2E-16 + AND galaxy.spe_z_prob > 0.99 + AND galaxy.spe_z_rel > 0.05 + AND galaxy.spe_rank = 0 + AND galaxy.spe_z > 0.85 + AND galaxy.spe_z < 1.85 + AND quality.spe_npix > 800 + AND quality.spe_n_dith_med > 15 + AND (mer.point_like_prob < 0.05) + AND (mer.spurious_prob < 0.5) + AND (mer.kron_radius < 50) + AND (galaxy.spe_z > phz.phz_95_int_1) + AND (galaxy.spe_z < phz.phz_95_int_2) + ORDER BY lines.spe_line_snr_gf DESC + """ + + result = Irsa.query_tap( + query_lines, + async_job=True + ).to_table() + + line_tables.append(result) + +if line_tables: + lines = vstack(line_tables) +else: + lines = Table() + + +# -------------------------------------------------- +# Add RA and Dec from the MER catalog +# -------------------------------------------------- + +if len(lines) > 0: + lines = join( + lines, + mer["object_id", "ra", "dec"], + keys="object_id", + join_type="left" + ) + +# Sort final table by SNR (highest first) +lines.sort("spe_line_snr_gf", reverse=True) ``` ### Choose an object of interest, lets look at an object with a strong Halpha line detected with high SNR. ```{code-cell} ipython3 -obj_id = 2737659721646729968 +obj_id = 2700933273661649762 obj_row = result_table[(result_table['object_id'] == obj_id)] From a154f18d7ede8c99b6566e11264863c853702276 Mon Sep 17 00:00:00 2001 From: KeunhoKim-astro <99432679+KeunhoKim-astro@users.noreply.github.com> Date: Fri, 6 Nov 2026 08:27:41 -0800 Subject: [PATCH 13/23] Refactor spectrum query and plotting for clarity Updated the spectrum collection name and adjusted the query radius. Modified the spectrum reading process to handle both red and blue spectra separately, including error handling and plotting adjustments. --- .../euclid/5_Euclid_intro_SPE_catalog.md | 35 ++++++++++++------- 1 file changed, 22 insertions(+), 13 deletions(-) diff --git a/tutorials/euclid/5_Euclid_intro_SPE_catalog.md b/tutorials/euclid/5_Euclid_intro_SPE_catalog.md index 68cc5b3a..1d2d9c99 100644 --- a/tutorials/euclid/5_Euclid_intro_SPE_catalog.md +++ b/tutorials/euclid/5_Euclid_intro_SPE_catalog.md @@ -287,7 +287,7 @@ obj_row ```{code-cell} ipython3 # Query SSA for the 1D spectrum near this object's sky position -euclid_ssa_collection = "euclid_DpdSirCombinedSpectra" +euclid_ssa_collection = "euclid_dr1_DpdSirCombinedSpectra" # Use the object's MER coordinates from obj_row coord_obj = SkyCoord(obj_row["ra"][0], obj_row["dec"][0], unit=u.deg) @@ -295,7 +295,7 @@ coord_obj = SkyCoord(obj_row["ra"][0], obj_row["dec"][0], unit=u.deg) #complete the query ssa_result = Irsa.query_ssa( pos=coord_obj, - radius=2.0 * u.arcsec, + radius=1.0 * u.arcsec, collection=euclid_ssa_collection, ) @@ -306,12 +306,19 @@ ssa_result ```{code-cell} ipython3 # Read in the spectrum for this object from the SSA access URL -spectrum_path = ssa_result["access_url"][0] +row = ssa_result[1] +row_blue = ssa_result[0] + +spectrum_path = row["access_url"] spectrum_path + +spectrum_path_blue = row_blue["access_url"] +spectrum_path_blue ``` ```{code-cell} ipython3 spectrum = QTable.read(spectrum_path) +spectrum_blue = QTable.read(spectrum_path_blue) ``` ### Now the data are read in, plot the spectrum with the H-alpha line labeled @@ -326,16 +333,18 @@ quantity_support() ```{code-cell} ipython3 # Note that the units are missing from the lines table, we manually add Angstrom -line_wavelengths = obj_row['spe_line_central_wl_gf'] * u.angstrom -line_names = obj_row['spe_line_name'] -snr_gf = obj_row['spe_line_snr_gf'] - -plt.plot(spectrum['WAVELENGTH'].to(u.micron), spectrum['SIGNAL']) - -for wl, name, snr in zip(np.atleast_1d(line_wavelengths), np.atleast_1d(line_names), np.atleast_1d(snr_gf)): - plt.axvline(wl, color='b', linestyle='--', alpha=0.3) - plt.text(wl, .2, name+' SNR='+str(round(snr)), rotation=90, ha='center', va='bottom', fontsize=10) - +bad_mask = (spectrum['MASK'].value % 2 == 1) | (spectrum['MASK'].value >= 64) +bad_mask_blue = (spectrum_blue['MASK'].value % 2 == 1) | (spectrum_blue['MASK'].value >= 64) + +plt.plot(spectrum['WAVELENGTH'].to(u.micron), np.ma.masked_where(bad_mask, spectrum['SIGNAL']), alpha=0.6, color='red', label='Red Spectrum') +plt.plot(spectrum['WAVELENGTH'], np.ma.masked_where(~bad_mask, spectrum['SIGNAL']), color='red', label='Do not use (Red)', linestyle=':', linewidth=2.0) +plt.plot(spectrum['WAVELENGTH'], np.sqrt(spectrum['VAR']), color='grey', label='Error') +plt.plot(spectrum_blue['WAVELENGTH'].to(u.micron), np.ma.masked_where(bad_mask_blue, spectrum_blue['SIGNAL']), alpha=0.5, color='blue', label='Blue Spectrum')#, #color='black', label='Spectrum') +plt.plot(spectrum_blue['WAVELENGTH'], np.ma.masked_where(~bad_mask_blue, spectrum_blue['SIGNAL']), color='blue', label='Do not use (Blue)', linestyle=':', linewidth=2.0) +plt.plot(spectrum_blue['WAVELENGTH'], np.sqrt(spectrum_blue['VAR']), color='grey', label='Error') + +plt.legend(loc='upper right') +plt.ylim(-0.05, 0.25) plt.title(f'Object ID {obj_id}') ``` From c5bc329ca92df4acf323f856a46fafbece0391ce Mon Sep 17 00:00:00 2001 From: KeunhoKim-astro <99432679+KeunhoKim-astro@users.noreply.github.com> Date: Fri, 6 Nov 2026 09:49:33 -0800 Subject: [PATCH 14/23] Clarify spectrum selection criteria in SPE catalog Updated criteria for selecting spectra in the catalog to specify valid pixel count and median dithers. --- tutorials/euclid/5_Euclid_intro_SPE_catalog.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tutorials/euclid/5_Euclid_intro_SPE_catalog.md b/tutorials/euclid/5_Euclid_intro_SPE_catalog.md index 1d2d9c99..b79104fe 100644 --- a/tutorials/euclid/5_Euclid_intro_SPE_catalog.md +++ b/tutorials/euclid/5_Euclid_intro_SPE_catalog.md @@ -183,7 +183,7 @@ We specify the following conditions on our search, based on recommendations in t - We select objects with well-measured H-alpha flux based on the spe_line_qual_gf quality flags - Choose spectroscopic redshift (spe_z) between 0.85 and 1.85 and spe_z_prob greater than 0.999 - We select objects whose spectroscopic redshifts (spe_z) fall within the 95% confidence intervals of their photometric redshifts (phz_95_int) -- Choose spectrum spe_npix and spe_n_dith_med greater than 15 +- We select spectra with a high number of valid pixels (spe_npix > 800) and a median number of dithers greater than 15 (spe_n_dith_med > 15). - H-alpha line flux should be more than 2x10^16 erg s^-1 cm^-2 - We exclude point-like objects (primarily stars; mer.point_like_prob < 0.05), excessively large objects (mer.kron_radius < 50), and potentially spurious detections (mer.spurious_prob < 0.5). The size criterion helps minimize contamination from galaxies at $z < 0.85$ with incorrectly measured spectroscopic redshifts. - Join the lines, mer, photometric redshift, and galaxy candidates tables on object_id and spe_rank From 0105c34a4ea3e44f5b4fb453c72b9118c770b385 Mon Sep 17 00:00:00 2001 From: KeunhoKim-astro <99432679+KeunhoKim-astro@users.noreply.github.com> Date: Fri, 6 Nov 2026 09:58:41 -0800 Subject: [PATCH 15/23] Refine search criteria for H-alpha detection Clarified the conditions for selecting objects in the search criteria, including specific quality flags and flux measurements. --- tutorials/euclid/5_Euclid_intro_SPE_catalog.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/tutorials/euclid/5_Euclid_intro_SPE_catalog.md b/tutorials/euclid/5_Euclid_intro_SPE_catalog.md index b79104fe..b8487306 100644 --- a/tutorials/euclid/5_Euclid_intro_SPE_catalog.md +++ b/tutorials/euclid/5_Euclid_intro_SPE_catalog.md @@ -180,12 +180,12 @@ object_ids = [row[0] for row in object_ids] We specify the following conditions on our search, based on recommendations in the Euclid DR1 spectroscopic paper: - Signal to noise ratio column (_gf = gaussian fit) should be greater than 10 - We want to detect H-alpha -- We select objects with well-measured H-alpha flux based on the spe_line_qual_gf quality flags +- We select objects with well-measured Hα fluxes based on the spe_line_qual_gf quality flags (spe_line_qual_gf = 212, 222, or 252) - Choose spectroscopic redshift (spe_z) between 0.85 and 1.85 and spe_z_prob greater than 0.999 - We select objects whose spectroscopic redshifts (spe_z) fall within the 95% confidence intervals of their photometric redshifts (phz_95_int) -- We select spectra with a high number of valid pixels (spe_npix > 800) and a median number of dithers greater than 15 (spe_n_dith_med > 15). +- We select spectra with a high number of valid pixels (spe_npix > 800) and a median number of dithers greater than 15 (spe_n_dith_med > 15) - H-alpha line flux should be more than 2x10^16 erg s^-1 cm^-2 -- We exclude point-like objects (primarily stars; mer.point_like_prob < 0.05), excessively large objects (mer.kron_radius < 50), and potentially spurious detections (mer.spurious_prob < 0.5). The size criterion helps minimize contamination from galaxies at $z < 0.85$ with incorrectly measured spectroscopic redshifts. +- We exclude point-like objects (primarily stars; mer.point_like_prob < 0.05), excessively large objects (mer.kron_radius < 50), and potentially spurious detections (mer.spurious_prob < 0.5). The size criterion helps minimize contamination from galaxies at $z < 0.85$ with incorrectly measured spectroscopic redshifts - Join the lines, mer, photometric redshift, and galaxy candidates tables on object_id and spe_rank We limit our search to the first 10,000 objects to reduce query execution time and avoid potential timeout issues. From 6e4072edb09bd72367545ae04fa4224cd4c6f03b Mon Sep 17 00:00:00 2001 From: KeunhoKim-astro <99432679+KeunhoKim-astro@users.noreply.github.com> Date: Fri, 6 Nov 2026 10:02:11 -0800 Subject: [PATCH 16/23] Refine spectroscopic redshift selection criteria Updated the condition for selecting objects based on spectroscopic redshift probability and reliability. --- tutorials/euclid/5_Euclid_intro_SPE_catalog.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tutorials/euclid/5_Euclid_intro_SPE_catalog.md b/tutorials/euclid/5_Euclid_intro_SPE_catalog.md index b8487306..4431e6b5 100644 --- a/tutorials/euclid/5_Euclid_intro_SPE_catalog.md +++ b/tutorials/euclid/5_Euclid_intro_SPE_catalog.md @@ -181,7 +181,7 @@ We specify the following conditions on our search, based on recommendations in t - Signal to noise ratio column (_gf = gaussian fit) should be greater than 10 - We want to detect H-alpha - We select objects with well-measured Hα fluxes based on the spe_line_qual_gf quality flags (spe_line_qual_gf = 212, 222, or 252) -- Choose spectroscopic redshift (spe_z) between 0.85 and 1.85 and spe_z_prob greater than 0.999 +- Choose spectroscopic redshift (spe_z) between 0.85 and 1.85 and spe_z_prob greater than 0.99, and spe_z_rel greater than 0.05 - We select objects whose spectroscopic redshifts (spe_z) fall within the 95% confidence intervals of their photometric redshifts (phz_95_int) - We select spectra with a high number of valid pixels (spe_npix > 800) and a median number of dithers greater than 15 (spe_n_dith_med > 15) - H-alpha line flux should be more than 2x10^16 erg s^-1 cm^-2 From 5f85d43d8953ba545332d60e27d86f0ed2c969ca Mon Sep 17 00:00:00 2001 From: xoubish Date: Mon, 9 Nov 2026 12:12:56 -0800 Subject: [PATCH 17/23] working on spe --- notebook_metadata.yml | 2 +- .../euclid/5_Euclid_intro_SPE_catalog.md | 573 ++++++++++++------ 2 files changed, 405 insertions(+), 170 deletions(-) diff --git a/notebook_metadata.yml b/notebook_metadata.yml index 12a8f26f..bf477fbb 100644 --- a/notebook_metadata.yml +++ b/notebook_metadata.yml @@ -21,7 +21,7 @@ - title: SPE Catalogs file: tutorials/euclid/5_Euclid_intro_SPE_catalog.md section: Euclid - description: Join the SPE and MER catalogs and query for galaxies with H-alpha line detections, then plot the SIR spectrum of a galaxy with a high SNR H-alpha line measurement. + description: Join the DR1 SPE, MER, and PHZ catalogs to select galaxies with reliable redshifts and well-measured H-alpha lines, see what each criterion removes, and plot the spectra and line measurements of selected and rejected galaxies. - title: Early Release Observations (ERO) - Star Clusters file: tutorials/euclid/Euclid_ERO.md section: Euclid diff --git a/tutorials/euclid/5_Euclid_intro_SPE_catalog.md b/tutorials/euclid/5_Euclid_intro_SPE_catalog.md index 4431e6b5..0a4d302f 100644 --- a/tutorials/euclid/5_Euclid_intro_SPE_catalog.md +++ b/tutorials/euclid/5_Euclid_intro_SPE_catalog.md @@ -33,10 +33,12 @@ authors: +++ By the end of this tutorial, you will: -- Understand the basic characteristics of Euclid DR1 SPE catalogs. -- Understand what SPE catalogs are available and how to view the columns in those catalogs. -- How to query with ADQL in the SPE lines catalog to find strong H-alpha detections. -- How to make a plot the detected line features over the 1D spectra. +- know which Euclid DR1 SPE catalog tables are available at IRSA, and how to view their columns; +- know how to join the SPE redshift, quality, and lines tables with the MER and PHZ catalogs in ADQL; +- know how to filter the SPE tables on their redshift, quality, and line columns, with the example of selecting galaxies with a strong Hα line, and see how many galaxies each criterion removes; +- understand the line quality flags, and how to pick the combined-grism line measurements; +- know how to find and plot the spectra of galaxies, and mark their measured lines; +- know the limitations of the DR1 line measurements for line ratios. +++ @@ -72,17 +74,18 @@ We rely on ``astroquery`` features that have been recently added, so please make import matplotlib.pyplot as plt import numpy as np -from astropy.coordinates import SkyCoord -from astropy.io import fits -from astropy.table import QTable +from astropy.table import QTable, Table, join, vstack from astropy import units as u -from astropy.utils.data import conf, download_file -from astropy.visualization import ImageNormalize, PercentileInterval, AsinhStretch, quantity_support +from astropy.utils.data import conf +from astropy.visualization import quantity_support from astroquery.ipac.irsa import Irsa -# The Euclid spectrum files are large and the time it takes to read -# them can exceed astropy's default timeout limit. Increase it. +# Let matplotlib pick up the axis units from the quantities we plot +quantity_support() + +# The Euclid spectrum files can take a while to read, +# which can exceed astropy's default timeout limit. Increase it. conf.remote_timeout = 120 #suppress warnings about deprecated units and cache @@ -104,27 +107,39 @@ warnings.filterwarnings( ) ``` -```{tip} -The IRSA SIA collections can be listed using using the ``list_collections`` method, we can filter on the ones containing "euclid" in the collection name: - - Irsa.list_collections(filter='euclid') + +```{code-cell} ipython3 +# DR1 is not public yet, so we query IRSA's development server. +Irsa.tap_url = 'https://irsadev.ipac.caltech.edu/TAP' ``` +## 1. Find the SPE catalog tables at IRSA + +The SPE catalogs are large, so we never download them: we query them on IRSA's server with ADQL through the Table Access Protocol (TAP) service, and get back only the rows we ask for. +In this section we only look at which tables are available and what their columns are. +Like the MER catalog (see the [MER catalogs notebook](2_Euclid_intro_MER_catalog.md)), the DR1 SPE catalogs are split by survey, with a `_wide_survey` and a `_deep` version of each table: +* `euclid_dr1_spectro_z_spe_galaxy_candidates_*` -- the spectroscopic redshift candidates of each source fitted with galaxy templates, ranked by `spe_rank` (0 is the best candidate), with their probability `spe_z_prob` and reliability `spe_z_rel`. +* `euclid_dr1_spectro_line_features_spe_line_features_*` -- the line measurements for each redshift candidate (matched on `object_id` and `spe_rank`): every line of a reference list (from Lyα to the Paschen lines) is measured at its expected wavelength at that redshift, with its wavelength, flux, signal-to-noise ratio, and quality, from a Gaussian fit (`_gf`) and from direct integration (`_di`). Lines are measured whether or not they are detected, so most rows are non-detections (for example `spe_line_snr_gf = -99`), and a line can have several rows with different `spe_line_qual_gf` values. +* `euclid_dr1_spectro_z_spe_quality_*` -- one row per source processed by SPE, including sources without a redshift candidate, describing its spectrum and the processing: grism(s), wavelength range, number of valid pixels, number of dithers, and error and warning flags. +* `euclid_dr1_spectro_z_spe_classification_*` -- the classification of each source as a galaxy, star, or QSO from its spectrum. -## 1. Download SPE catalog from IRSA directly to this notebook +The SPE tables have no coordinates; they are joined to the MER catalog of the same survey on `object_id`. +In this notebook we work in the Euclid Deep Field North, so we use the deep tables. -Search for all Euclid catalogs and metadata tables in IRSA: +The DR1 catalogs at IRSA can be listed with the `list_catalogs` method: ```{note} -In the example below, we use ``include_metadata_tables=True``. +In the example below, we use ``include_metadata_tables=True``, as IRSA lists the PHZ and SPE tables among the metadata tables rather than the catalogs. This parameter was introduced in astroquery 0.4.11. For earlier versions, please omit this parameter to receive the combined list of available catalogs and metadata tables. ``` ```{code-cell} ipython3 -Irsa.list_catalogs(filter='euclid', include_metadata_tables=True) +Irsa.list_catalogs(filter='euclid_dr1', include_metadata_tables=True) ``` +We will use the following tables, including the MER catalog for the coordinates and morphology of the sources, and the PHZ catalog for their photometric redshifts: + ```{code-cell} ipython3 table_mer = 'euclid_dr1_mer_catalogue_deep_survey' table_galaxy_candidates = 'euclid_dr1_spectro_z_spe_galaxy_candidates_deep' @@ -133,225 +148,445 @@ table_lines = 'euclid_dr1_spectro_line_features_spe_line_features_deep' table_photo_z = 'euclid_dr1_phz_photo_z_deep_survey' ``` -### Learn some information about the table: -- How many columns are there? -- List the column names +### Explore the columns of the SPE tables + +Before writing a query, it helps to know which columns each table has and what they mean. +The `list_columns` method returns the name and description of every column of a table, without querying any data. +We look at the two tables we use most below: the lines table, where each line measurement has a set of columns from the Gaussian fit (ending in `_gf`) and the same quantities from direct integration (ending in `_di`), and the redshift table, where each row is one redshift candidate of a source: ```{code-cell} ipython3 -columns_info = Irsa.list_columns(catalog=table_mer) -print(len(columns_info)) +columns_lines = Irsa.list_columns(catalog=table_lines) +columns_galaxy_candidates = Irsa.list_columns(catalog=table_galaxy_candidates) +print(f'{len(columns_lines)} columns in the lines table, {len(columns_galaxy_candidates)} in the redshift table') +``` + +```{tip} +A few highlights of the columns we use below: +- `object_id`, the MER object ID used to join the SPE tables with each other and with the MER and PHZ catalogs; +- `spe_rank`, the rank of the redshift candidate, 0 for the best one; +- `spe_z`, `spe_z_prob`, and `spe_z_rel`, the spectroscopic redshift, its probability, and its reliability; +- `spe_line_name`, the name of the detected line, for example `'Halpha'`; +- `spe_line_central_wl_gf`, `spe_line_flux_gf`, and `spe_line_snr_gf`, the central wavelength (in Angstrom), flux (in erg s-1 cm-2), and signal-to-noise ratio of the line from the Gaussian fit; +- `spe_line_qual_gf`, the quality of the Gaussian fit measurements; +- `spe_npix` and `spe_n_dith_med` in the quality table, the number of valid pixels and the median number of dithers of the spectrum. ``` ```{code-cell} ipython3 -Irsa.list_columns(catalog=table_1dspectra, full=True) +# Full list of columns of the lines table and their description +columns_lines ``` ```{code-cell} ipython3 -# Full list of columns and their description -columns_info +# Full list of columns of the redshift table and their description +columns_galaxy_candidates ``` ## 2. Find objects within a specified sky region defined by RA and Dec ranges Euclid DR1 covers a large sky area of approximately 1,900 square degrees. To reduce query execution time and avoid potential timeout issues, we recommend restricting searches to smaller sky regions. -As an example, let's select objects within a 0.1 square-degree region of the Euclid Deep Field North (EDF-N). +As an example, let's work in a 0.1 square-degree region of the Euclid Deep Field North (EDF-N), and first count the MER sources in it: ```{code-cell} ipython3 +ra_min, ra_max = 269.34391, 270.12192 +dec_min, dec_max = 65.85958, 66.17581 + query_mer = f""" -SELECT DISTINCT - object_id, - ra, - dec +SELECT COUNT(*) AS n_sources FROM {table_mer} -WHERE - ra BETWEEN 269.34391 AND 270.12192 -AND dec BETWEEN 65.85958 AND 66.17581 +WHERE ra BETWEEN {ra_min} AND {ra_max} + AND dec BETWEEN {dec_min} AND {dec_max} +""" +n_mer = Irsa.query_tap(query_mer).to_table()['n_sources'][0] +print(f'Found {n_mer} MER sources in the region') +``` + +Even this small region contains tens of thousands of MER sources. +Only a fraction of them have an SPE redshift, and fewer still have a redshift and line measurements we can trust; finding those is the subject of the next section. + +## 3. Example: filter the SPE tables for galaxies with a strong H-alpha line + +The SPE tables can be filtered on any of their columns, and the right filters depend on the science case. +In this section, we show how to combine the SPE redshift, quality, and lines tables with the MER and PHZ catalogs, using one common case as an example: selecting galaxies with a strong, well-measured Hα line, the main target of the Euclid spectroscopic survey. + +The criteria below are one recipe for this case, following the recommendations of the Euclid SPE team. +They are deliberately strict, and are not a definition of a "good" spectrum: a galaxy at z > 1.85, for example, has its Hα line beyond the end of the red grism spectrum and fails criterion 2, but can have an excellent spectrum and redshift from its [OIII] or [OII] lines. +For another science case, drop or change the criteria that do not apply, for example the redshift range and the line in criteria 2 and 6. + +1. **High-confidence redshift**: the best redshift candidate (`spe_rank = 0`) has a high probability (`spe_z_prob > 0.99`) and reliability (`spe_z_rel > 0.05`). +2. **Hα in the spectrum**: the redshift is between 0.45 and 1.85, the range in which Hα falls in the blue grism (0.93 to 1.37 micron) or the red grism (1.2 to 1.89 micron), away from the edges of the spectra. +3. **Both grisms, full wavelength coverage and depth**: in the quality table, the source has both a blue and a red grism spectrum (`spe_grism = 'BGS+RGS'`; drop this requirement for the Euclid Wide Survey, which only has red grism spectra), its combined spectrum has at least 800 valid pixels (`spe_npix > 800`, out of at most about 850 for the combined blue and red grism spectrum), so almost none of its wavelength range is masked, and the median number of dithered exposures per pixel is greater than 15 (`spe_n_dith_med > 15`, about 20 for most sources in this region), so it reaches close to the full depth of the survey. +4. **Extended, non-spurious source**: in the MER catalog, the source is not point-like (`point_like_prob < 0.05`, which removes most stars, but also compact galaxies and quasars), is unlikely to be spurious (`spurious_prob < 0.5`), and is not excessively large (`kron_radius < 50`), which removes nearby galaxies whose lines can be mistaken for Hα. +5. **Consistent with the photometric redshift**: the spectroscopic redshift is within the 95% interval of the photometric redshift of the PHZ catalog (`phz_95_int_1` to `phz_95_int_2`), a check against a wrong line identification. +6. **Strong, well-measured Hα line**: the Gaussian fit of the Hα line is good in at least one of the two grisms, and the combined measurement of the two grisms has a signal-to-noise ratio greater than 10 and a flux greater than 2×10-16 erg s-1 cm-2. + +### 3.1 Get the redshifts, spectrum quality, and morphology of the sources + +Criteria 1 to 5 only involve one row per source, from the redshift, quality, MER, and PHZ tables, so we get all of them with a single query, which joins the tables on IRSA's server. +We use a left outer join for the PHZ catalog, to keep the sources without a photometric redshift: + +```{code-cell} ipython3 +query_candidates = f""" +SELECT mer.object_id, mer.ra, mer.dec, + mer.point_like_prob, mer.spurious_prob, mer.kron_radius, + galaxy.spe_z, galaxy.spe_z_prob, galaxy.spe_z_rel, + quality.spe_grism, quality.spe_npix, quality.spe_n_dith_med, + phz.phz_median, phz.phz_95_int_1, phz.phz_95_int_2 +FROM {table_mer} AS mer +JOIN {table_galaxy_candidates} AS galaxy ON galaxy.object_id = mer.object_id +JOIN {table_quality} AS quality ON quality.object_id = mer.object_id +LEFT OUTER JOIN {table_photo_z} AS phz ON phz.object_id = mer.object_id +WHERE mer.ra BETWEEN {ra_min} AND {ra_max} + AND mer.dec BETWEEN {dec_min} AND {dec_max} + AND galaxy.spe_rank = 0 """ +candidates = Irsa.query_tap(query_candidates, async_job=True).to_table() +print(f'{len(candidates)} of the {n_mer} MER sources have an SPE redshift') +``` -mer = Irsa.query_tap(query_mer).to_table() +We now apply the criteria one at a time and count how many galaxies are left after each of them. +Missing values (for example a missing photometric redshift) are masked in the table; we fill them with NaN, so that they fail every comparison: -object_ids = mer["object_id", "ra", "dec"].as_array().tolist() -object_ids = [row[0] for row in object_ids] +```{code-cell} ipython3 +def values(table, column): + return np.ma.filled(np.ma.asarray(table[column], dtype=float), np.nan) + + +def criteria(table): + z = values(table, 'spe_z') + return { + '1. High-confidence redshift': (values(table, 'spe_z_prob') > 0.99) & (values(table, 'spe_z_rel') > 0.05), + '2. Hα in the spectrum (0.45 < z < 1.85)': (z > 0.45) & (z < 1.85), + '3. Both grisms, full coverage and depth': ((table['spe_grism'] == 'BGS+RGS') + & (values(table, 'spe_npix') > 800) + & (values(table, 'spe_n_dith_med') > 15)), + '4. Extended, non-spurious source': ((values(table, 'point_like_prob') < 0.05) + & (values(table, 'spurious_prob') < 0.5) + & (values(table, 'kron_radius') < 50)), + '5. Consistent with the photo-z': (z > values(table, 'phz_95_int_1')) & (z < values(table, 'phz_95_int_2')), + } + + +def cut_flow(cuts): + remaining = np.ones(len(next(iter(cuts.values()))), dtype=bool) + steps, counts = ['SPE redshift'], [len(remaining)] + for name, cut in cuts.items(): + remaining &= cut + steps.append(name) + counts.append(remaining.sum()) + return Table({'criterion': steps, 'n_galaxies': counts}) + + +cut_flow(criteria(candidates)) ``` +### 3.2 Add the H-alpha measurements -## 3. Find some objects with spectra with conditions +As we saw in section 1, the lines table has several rows for each line of a source: one for each grism in which the line is measured. +The last digit of the quality flag `spe_line_qual_gf` tells which spectrum the measurement comes from, and its middle digit tells how good it is: -We specify the following conditions on our search, based on recommendations in the Euclid DR1 spectroscopic paper: -- Signal to noise ratio column (_gf = gaussian fit) should be greater than 10 -- We want to detect H-alpha -- We select objects with well-measured Hα fluxes based on the spe_line_qual_gf quality flags (spe_line_qual_gf = 212, 222, or 252) -- Choose spectroscopic redshift (spe_z) between 0.85 and 1.85 and spe_z_prob greater than 0.99, and spe_z_rel greater than 0.05 -- We select objects whose spectroscopic redshifts (spe_z) fall within the 95% confidence intervals of their photometric redshifts (phz_95_int) -- We select spectra with a high number of valid pixels (spe_npix > 800) and a median number of dithers greater than 15 (spe_n_dith_med > 15) -- H-alpha line flux should be more than 2x10^16 erg s^-1 cm^-2 -- We exclude point-like objects (primarily stars; mer.point_like_prob < 0.05), excessively large objects (mer.kron_radius < 50), and potentially spurious detections (mer.spurious_prob < 0.5). The size criterion helps minimize contamination from galaxies at $z < 0.85$ with incorrectly measured spectroscopic redshifts -- Join the lines, mer, photometric redshift, and galaxy candidates tables on object_id and spe_rank +| Digit | Value | Meaning | +|---|---|---| +| last (grism) | 1, 2, 3 | blue grism, red grism, blue and red grisms combined | +| middle (quality) | 1 | good measurement | +| | 2 | low signal-to-noise ratio (below 2) | +| | 5 | acceptable, but up to 15% of the pixels of the line are missing | +| | 8 | unreliable, more than 15% of the pixels of the line are missing | +| | 9 | the line is outside of the wavelength range (with a negative sign, for example -92) | -We limit our search to the first 10,000 objects to reduce query execution time and avoid potential timeout issues. -Finally we sort the data by descending spe_line_snr_gf to have the largest SNR H-alpha lines detected at the top. +For each galaxy, we get all of the Hα measurements in the wavelength range (`spe_line_qual_gf > 0`), and use them in two ways: +- the quality of the single-grism measurements (last digit 1 or 2) tells us whether Hα is well measured in at least one of the grisms: we require a quality digit of 1 (good) or 5 (acceptable), that is a quality flag of 211, 212, 251, or 252. Hα does not need to be well measured in both grisms: at z > 1.08 it is only in the red grism, and at z < 0.84 only in the blue grism; +- the combined measurement (last digit 3), which uses both grisms where the line falls in both, gives the flux and signal-to-noise ratio of the line. + +```{note} +The quality digit of most of the combined measurements is 4 (for example 243), which is not described in the Q1 documentation of the quality flags; we find it for lines measured in a single grism. +This is why we take the quality from the single-grism measurements. +``` + +The IRSA TAP service accepts at most 1000 values in an `IN (...)` list, so we query the Hα measurements in batches of 1000 object IDs. +We only query the galaxies that pass the first two criteria (the others are rejected anyway), which still includes the ones failing the later criteria, so that we can look at some of them in section 5: ```{code-cell} ipython3 -batch_size = 500 - -line_tables = [] -for i in range(0, 10000, batch_size): - - batch = object_ids[i:i + batch_size] - - ids = ", ".join(str(int(x)) for x in batch) - - query_lines = f""" - SELECT - lines.object_id, - lines.spe_rank, - lines.spe_line_name, - lines.spe_line_qual_gf, - lines.spe_line_central_wl_gf, - lines.spe_line_snr_gf, - galaxy.object_id, - galaxy.spe_z - FROM {table_lines} AS lines - JOIN {table_galaxy_candidates} AS galaxy - ON lines.object_id = galaxy.object_id - AND lines.spe_rank = galaxy.spe_rank - JOIN {table_quality} AS quality - ON lines.object_id = quality.object_id - JOIN {table_mer} AS mer - ON lines.object_id = mer.object_id - JOIN {table_photo_z} AS phz - ON lines.object_id = phz.object_id - - WHERE - lines.object_id IN ({ids}) - AND lines.spe_line_name = 'Halpha' - AND lines.spe_line_qual_gf IN (212, 222, 252) - AND lines.spe_line_snr_gf > 10 - AND lines.spe_rank = 0 - AND lines.spe_line_flux_gf > 2E-16 - AND galaxy.spe_z_prob > 0.99 - AND galaxy.spe_z_rel > 0.05 - AND galaxy.spe_rank = 0 - AND galaxy.spe_z > 0.85 - AND galaxy.spe_z < 1.85 - AND quality.spe_npix > 800 - AND quality.spe_n_dith_med > 15 - AND (mer.point_like_prob < 0.05) - AND (mer.spurious_prob < 0.5) - AND (mer.kron_radius < 50) - AND (galaxy.spe_z > phz.phz_95_int_1) - AND (galaxy.spe_z < phz.phz_95_int_2) - ORDER BY lines.spe_line_snr_gf DESC +def query_halpha(object_ids, batch_size=1000): + tables = [] + for start in range(0, len(object_ids), batch_size): + ids = ', '.join(str(object_id) for object_id in object_ids[start:start + batch_size]) + query = f""" + SELECT object_id, spe_line_qual_gf, spe_line_flux_gf, spe_line_snr_gf, spe_line_central_wl_gf + FROM {table_lines} + WHERE object_id IN ({ids}) + AND spe_rank = 0 + AND spe_line_name = 'Halpha' + AND spe_line_qual_gf > 0 """ + tables.append(Irsa.query_tap(query, async_job=True).to_table()) + return vstack(tables) + - result = Irsa.query_tap( - query_lines, - async_job=True - ).to_table() +cuts = criteria(candidates) +in_range = cuts['1. High-confidence redshift'] & cuts['2. Hα in the spectrum (0.45 < z < 1.85)'] +halpha_rows = query_halpha(list(candidates['object_id'][in_range])) - line_tables.append(result) +grism_digit = halpha_rows['spe_line_qual_gf'] % 10 +quality_digit = (halpha_rows['spe_line_qual_gf'] // 10) % 10 +good_single_grism = halpha_rows[(grism_digit < 3) & np.isin(quality_digit, [1, 5])] -if line_tables: - lines = vstack(line_tables) -else: - lines = Table() +# Keep the combined measurement, and flag whether Hα is good in at least one grism +halpha = halpha_rows[grism_digit == 3] +halpha['halpha_good_in_a_grism'] = np.isin(halpha['object_id'], good_single_grism['object_id']) +for column in ['qual_gf', 'flux_gf', 'snr_gf', 'central_wl_gf']: + halpha.rename_column(f'spe_line_{column}', f'halpha_{column}') +galaxies = join(candidates, halpha, keys='object_id', join_type='left') +print(f"Got the Hα measurements of {len(halpha)} of the {in_range.sum()} galaxies; " + f"Hα is good in at least one grism for {halpha['halpha_good_in_a_grism'].sum()} of them") +``` + +We add the Hα criterion to the others, and look at the number of galaxies left after each of them: -# -------------------------------------------------- -# Add RA and Dec from the MER catalog -# -------------------------------------------------- +```{code-cell} ipython3 +def all_criteria(table): + cuts = criteria(table) + cuts['6. Strong, well-measured Hα'] = (np.ma.filled(table['halpha_good_in_a_grism'], False) + & (values(table, 'halpha_snr_gf') > 10) + & (values(table, 'halpha_flux_gf') > 2e-16)) + return cuts -if len(lines) > 0: - lines = join( - lines, - mer["object_id", "ra", "dec"], - keys="object_id", - join_type="left" - ) -# Sort final table by SNR (highest first) -lines.sort("spe_line_snr_gf", reverse=True) +cuts = all_criteria(galaxies) +passes_all = np.logical_and.reduce(list(cuts.values())) +cut_flow(cuts) ``` -### Choose an object of interest, lets look at an object with a strong Halpha line detected with high SNR. +Out of the tens of thousands of MER sources in the region, about a hundred galaxies pass all the criteria of this strict recipe. +The Hα criterion removes most of the galaxies left after the first five criteria, mainly because of the flux cut: most of the Hα lines are fainter than 2×10-16 erg s-1 cm-2, the flux limit of the Euclid Wide Survey, which the deep fields go below (see section 4). +Let's sort the galaxies that pass all of the criteria by the signal-to-noise ratio of their Hα line: ```{code-cell} ipython3 -obj_id = 2700933273661649762 +strong_halpha = galaxies[passes_all] +strong_halpha.sort('halpha_snr_gf', reverse=True) +strong_halpha['object_id', 'spe_z', 'phz_median', 'halpha_flux_gf', 'halpha_snr_gf'][:10] +``` -obj_row = result_table[(result_table['object_id'] == obj_id)] +## 4. Redshifts and Hα fluxes -obj_row +Let's compare the redshifts of the galaxies that pass all of the criteria with those of all the galaxies with a high-confidence redshift in the same range, and look at their Hα fluxes as a function of redshift: + +```{code-cell} ipython3 +high_confidence = cuts['1. High-confidence redshift'] & cuts['2. Hα in the spectrum (0.45 < z < 1.85)'] +has_halpha = values(galaxies, 'halpha_flux_gf') > 0 + +fig, (ax_z, ax_flux) = plt.subplots(1, 2, figsize=(13, 4.5)) + +bins = np.arange(0.45, 1.86, 0.05) +ax_z.hist(galaxies['spe_z'][high_confidence], bins=bins, color='lightgrey', label='High-confidence redshift') +ax_z.hist(galaxies['spe_z'][passes_all], bins=bins, color='tab:red', label='All criteria') +ax_z.set_xlabel('Spectroscopic redshift') +ax_z.set_ylabel('Number of galaxies') +ax_z.legend() + +ax_flux.scatter(galaxies['spe_z'][high_confidence & has_halpha], values(galaxies, 'halpha_flux_gf')[high_confidence & has_halpha], + s=5, color='lightgrey', label='High-confidence redshift') +points = ax_flux.scatter(galaxies['spe_z'][passes_all], values(galaxies, 'halpha_flux_gf')[passes_all], + s=12, c=values(galaxies, 'halpha_snr_gf')[passes_all], cmap='viridis', vmin=10, vmax=50, + label='All criteria') +ax_flux.axhline(2e-16, color='grey', linestyle='--', linewidth=0.8) +ax_flux.set_yscale('log') +ax_flux.set_ylim(1e-17, 3e-15) +ax_flux.set_xlabel('Spectroscopic redshift') +ax_flux.set_ylabel('Hα flux (erg s$^{-1}$ cm$^{-2}$)') +ax_flux.legend(loc='upper right') +fig.colorbar(points, ax=ax_flux, label='Hα signal-to-noise ratio') +plt.show() ``` -### Pull the spectrum of this object +## 5. Look at the spectra + +To see what the criteria mean, let's look at the spectra of some of the galaxies that pass all of them, and of some galaxies that fail one of them. +We find and read in the spectra as in the [SIR 1D spectra notebook](3_Euclid_intro_1D_spectra.md): +the table `euclid.objectid_spectrafile_association_dr1` lists the file and HDU of each spectrum of a source, and IRSA's spectrumDM service converts a given HDU into a VOTable, whose URL we build in the query. +A galaxy in the Euclid Deep Fields can have spectra from both the red and the blue grism, and several spectra of the same grism, so we keep the spectrum with the most usable flux bins for each grism: ```{code-cell} ipython3 -# Query SSA for the 1D spectrum near this object's sky position -euclid_ssa_collection = "euclid_dr1_DpdSirCombinedSpectra" +def read_spectra(object_id): + query = f""" + SELECT 'https://irsacloud.ipac.caltech.edu/api/spectrumdm/convert/' || path + || '?dataset_id=euclid_combspec_dr1&hdu=' || hdu AS url + FROM euclid.objectid_spectrafile_association_dr1 + WHERE objectid = {object_id} + """ + urls = Irsa.query_tap(query).to_table()['url'] + return [QTable.read(url) for url in urls] -# Use the object's MER coordinates from obj_row -coord_obj = SkyCoord(obj_row["ra"][0], obj_row["dec"][0], unit=u.deg) -#complete the query -ssa_result = Irsa.query_ssa( - pos=coord_obj, - radius=1.0 * u.arcsec, - collection=euclid_ssa_collection, -) +def bad_flux_bins(spectrum): + # Flux bins with an odd MASK value or a MASK value of 64 or more should not be used + mask = spectrum['MASK'].value + return (mask % 2 == 1) | (mask >= 64) -ssa_result + +def grism_name(spectrum): + # Blue grism spectra start below 1.1 micron + return 'Blue grism' if spectrum['WAVELENGTH'].min() < 1.1 * u.micron else 'Red grism' + + +def best_spectra(spectra): + best = {} + for spectrum in sorted(spectra, key=lambda spectrum: np.sum(~bad_flux_bins(spectrum)), reverse=True): + best.setdefault(grism_name(spectrum), spectrum) + return best ``` -### The following steps to read in the spectrum follows the 3_Euclid_intro_1D_spectra notebook. +We plot the usable flux bins as solid lines and the flux bins not to use as dotted lines, set the y-axis range from the usable flux bins, and mark the expected wavelengths of some of the strongest emission lines at the redshift of the galaxy: ```{code-cell} ipython3 -# Read in the spectrum for this object from the SSA access URL -row = ssa_result[1] -row_blue = ssa_result[0] +grism_colors = {'Red grism': 'tab:red', 'Blue grism': 'tab:blue'} +emission_lines = {'[OII]': 3728.5 * u.AA, 'Hβ': 4862.7 * u.AA, '[OIII]': 5008.2 * u.AA, + 'Hα': 6564.6 * u.AA, '[SII]': 6725.5 * u.AA} + + +def plot_spectra(ax, spectra, title, line_wavelengths): + for name, spectrum in spectra.items(): + wavelength = spectrum['WAVELENGTH'].to(u.micron) + bad = bad_flux_bins(spectrum) + ax.plot(wavelength, np.ma.masked_where(bad, spectrum['SIGNAL']), color=grism_colors[name], + alpha=0.8, linewidth=0.8, label=name) + ax.plot(wavelength, np.ma.masked_where(~bad, spectrum['SIGNAL']), color=grism_colors[name], + alpha=0.5, linewidth=0.8, linestyle=':') + + good_flux = np.concatenate([spectrum['SIGNAL'][~bad_flux_bins(spectrum)].filled(np.nan) + for spectrum in spectra.values()]) + low, high = np.nanpercentile(good_flux, [1, 100]) + ax.set_ylim(low - 0.1 * (high - low), high + 0.3 * (high - low)) + + # Mark the lines in the wavelength range, with a single label for lines closer than 100 Angstrom + wave_min = min(spectrum['WAVELENGTH'].min() for spectrum in spectra.values()) + wave_max = max(spectrum['WAVELENGTH'].max() for spectrum in spectra.values()) + labels = [] + for line, wavelength in sorted(line_wavelengths.items(), key=lambda item: item[1]): + if wave_min < wavelength < wave_max: + ax.axvline(wavelength.to(u.micron), color='grey', linestyle='--', linewidth=0.6) + if labels and wavelength - labels[-1][1] < 100 * u.AA: + labels[-1][0] += f'+{line}' + else: + labels.append([line, wavelength]) + for label, wavelength in labels: + ax.text(wavelength.to_value(u.micron), 0.98, f' {label}', transform=ax.get_xaxis_transform(), + rotation=90, va='top', ha='right', fontsize=8) + ax.set_title(title, fontsize='small') + + +def plot_galaxy(ax, galaxy, title=None): + z = galaxy['spe_z'] + lines = {line: rest_wavelength * (1 + z) for line, rest_wavelength in emission_lines.items()} + best = best_spectra(read_spectra(galaxy['object_id'])) + description = f"Object {galaxy['object_id']}, z = {z:.3f}, Hα S/N = {galaxy['halpha_snr_gf']:.0f}" + plot_spectra(ax, best, f'{title}\n{description}' if title else description, lines) +``` -spectrum_path = row["access_url"] -spectrum_path +### 5.1 Galaxies passing all of the criteria -spectrum_path_blue = row_blue["access_url"] -spectrum_path_blue -``` +Here are 20 of the galaxies that pass all of the criteria, spread evenly from the strongest to the weakest Hα signal-to-noise ratio (reading the spectra takes a minute or two): ```{code-cell} ipython3 -spectrum = QTable.read(spectrum_path) -spectrum_blue = QTable.read(spectrum_path_blue) +n_rows, n_columns = 4, 5 +examples = strong_halpha[np.linspace(0, len(strong_halpha) - 1, n_rows * n_columns).astype(int)] + +fig, axes = plt.subplots(n_rows, n_columns, figsize=(25, 14), sharex=True) +for ax, galaxy in zip(axes.flat, examples): + plot_galaxy(ax, galaxy) +# Only label the outer axes +for ax in axes[:, 1:].flat: + ax.set_ylabel('') +for ax in axes[:-1, :].flat: + ax.set_xlabel('') +axes[0, 0].legend(loc='lower right', fontsize='small') +plt.tight_layout() +plt.show() ``` -### Now the data are read in, plot the spectrum with the H-alpha line labeled +Hα is clearly detected in all of them, down to a signal-to-noise ratio of about 10. +Where the Hα line falls in both grisms (1.2 to 1.37 micron), it is seen at the same wavelength in the red and the blue grism spectra, and in many of the galaxies other lines, such as [OIII] in the blue grism, confirm the redshift. -```{tip} -As we use astropy.visualization's ``quantity_support``, matplotlib automatically picks up the axis units from the quantities we plot. +### 5.2 Galaxies failing one of the criteria + +Now let's look at galaxies that pass all of the criteria but one: for two of the criteria, we take the galaxy with the highest Hα signal-to-noise ratio among those that fail only this criterion. + +```{code-cell} ipython3 +def fails_only(cuts, criterion): + others = np.logical_and.reduce([cut for name, cut in cuts.items() if name != criterion]) + return others & ~cuts[criterion] + + +fig, axes = plt.subplots(1, 2, figsize=(14, 4.5), sharex=True) +for ax, criterion in zip(axes, ['3. Both grisms, full coverage and depth', '5. Consistent with the photo-z']): + rejected = galaxies[fails_only(cuts, criterion)] + rejected.sort('halpha_snr_gf', reverse=True) + plot_galaxy(ax, rejected[0], f'Fails criterion {criterion}') +axes[0].legend(loc='lower right', fontsize='small') +plt.tight_layout() +plt.show() ``` +The galaxy on the left fails the grism, coverage, and depth criterion only because the median number of dithered exposures of its spectrum is 14, just below our threshold of 15: its Hα line is clearly seen in the blue grism spectrum, and the spectrum is perfectly usable for many purposes. This is a reminder that the thresholds of a strict recipe like this one also reject some good spectra. +The galaxy on the right has a strong line at 1.23 micron, which the SPE processing identified as Hα, but its redshift is outside of the 95% interval of its photometric redshift. +The line is only seen in the blue grism spectrum, although the red grism spectrum also covers this wavelength: it is most likely not a line of this galaxy, but a contamination of the blue grism spectrum by a neighboring source, and the photometric redshift criterion rightly rejects this galaxy. + +## 6. A closer look at the line measurements of one galaxy + +Hα is only one of the lines the SPE processing measures. +Let's get all of the combined-grism line measurements of the galaxy with the strongest Hα line among those that pass all of the criteria, and keep those with a signal-to-noise ratio greater than 3: + ```{code-cell} ipython3 -quantity_support() +galaxy = strong_halpha[0] + +query_lines = f""" +SELECT spe_line_name, spe_line_central_wl_gf, spe_line_flux_gf, spe_line_flux_err_gf, + spe_line_snr_gf, spe_line_ew_gf, spe_line_fwhm_gf, spe_line_qual_gf +FROM {table_lines} +WHERE object_id = {galaxy['object_id']} + AND spe_rank = 0 + AND MOD(spe_line_qual_gf, 10) = 3 + AND spe_line_snr_gf > 3 +ORDER BY spe_line_central_wl_gf +""" +galaxy_lines = Irsa.query_tap(query_lines).to_table() +galaxy_lines ``` +Besides Hα, lines such as [OIII] (in the blue grism) and [SII] are detected. +Note that the Hα and [NII]6585 lines have the same signal-to-noise ratio and FWHM: they are fitted together, as they are blended at the resolution of the spectra (see the warning below). + +We plot the spectra of this galaxy again, now marking the lines at the central wavelengths measured by the Gaussian fits: + ```{code-cell} ipython3 -# Note that the units are missing from the lines table, we manually add Angstrom -bad_mask = (spectrum['MASK'].value % 2 == 1) | (spectrum['MASK'].value >= 64) -bad_mask_blue = (spectrum_blue['MASK'].value % 2 == 1) | (spectrum_blue['MASK'].value >= 64) - -plt.plot(spectrum['WAVELENGTH'].to(u.micron), np.ma.masked_where(bad_mask, spectrum['SIGNAL']), alpha=0.6, color='red', label='Red Spectrum') -plt.plot(spectrum['WAVELENGTH'], np.ma.masked_where(~bad_mask, spectrum['SIGNAL']), color='red', label='Do not use (Red)', linestyle=':', linewidth=2.0) -plt.plot(spectrum['WAVELENGTH'], np.sqrt(spectrum['VAR']), color='grey', label='Error') -plt.plot(spectrum_blue['WAVELENGTH'].to(u.micron), np.ma.masked_where(bad_mask_blue, spectrum_blue['SIGNAL']), alpha=0.5, color='blue', label='Blue Spectrum')#, #color='black', label='Spectrum') -plt.plot(spectrum_blue['WAVELENGTH'], np.ma.masked_where(~bad_mask_blue, spectrum_blue['SIGNAL']), color='blue', label='Do not use (Blue)', linestyle=':', linewidth=2.0) -plt.plot(spectrum_blue['WAVELENGTH'], np.sqrt(spectrum_blue['VAR']), color='grey', label='Error') - -plt.legend(loc='upper right') -plt.ylim(-0.05, 0.25) -plt.title(f'Object ID {obj_id}') +best = best_spectra(read_spectra(galaxy['object_id'])) +measured_lines = {row['spe_line_name']: u.Quantity(galaxy_lines['spe_line_central_wl_gf'])[i] + for i, row in enumerate(galaxy_lines)} + +fig, ax = plt.subplots(figsize=(13, 4.5)) +plot_spectra(ax, best, f"Object {galaxy['object_id']}, z = {galaxy['spe_z']:.3f}: lines with S/N > 3", measured_lines) +ax.legend(loc='lower right', fontsize='small') +plt.show() +``` + +```{warning} +Be careful when using line ratios from the DR1 SPE lines catalog. +At the resolution of the NISP spectra, Hα is blended with the [NII] lines, and the Gaussian fits of the blend are not independent: the [NII]6585 line has the same signal-to-noise ratio and FWHM as Hα, and the [NII]/Hα ratio is poorly constrained. +Hβ and [OIII] fall in the blue grism for most of the galaxies in the Euclid Deep Fields, while Hα falls in the red grism, so ratios such as Hα/Hβ also depend on the relative calibration of the two grisms. ``` +## References + +- Euclid Collaboration et al. 2025, *Euclid Quick Data Release (Q1): Characteristics and limitations of the spectroscopic measurements*, [arXiv:2503.15308](https://arxiv.org/abs/2503.15308) +- [Euclid SPE output catalog data product description](https://euclid.esac.esa.int/dr/q1/dpdd/spedpd/dpcards/spe_spepfoutputcatalog.html), which describes the columns of the SPE catalogs and the line quality flags + + ++++ + ## About this Notebook -**Updated**: 2026-01-13 +**Updated**: 2026-11-09 **Contact:** [the IRSA Helpdesk](https://irsa.ipac.caltech.edu/docs/help_desk.html) with questions or reporting problems. -**Runtime:** As of the date above, this notebook takes about 90 seconds to run to completion on a machine with 8GB RAM and 4 CPU. +**Runtime:** As of the date above, this notebook takes about 2 to 5 minutes, depending on the load of the IRSA services, to run to completion on a machine with 8GB RAM and 4 CPU. From e28d7b78d38240e49d8d9327d56e4863809acbc2 Mon Sep 17 00:00:00 2001 From: xoubish Date: Mon, 9 Nov 2026 15:22:00 -0800 Subject: [PATCH 18/23] notebook 5 there --- notebook_metadata.yml | 2 +- .../euclid/5_Euclid_intro_SPE_catalog.md | 104 +++++++++++------- 2 files changed, 66 insertions(+), 40 deletions(-) diff --git a/notebook_metadata.yml b/notebook_metadata.yml index bf477fbb..4713807d 100644 --- a/notebook_metadata.yml +++ b/notebook_metadata.yml @@ -21,7 +21,7 @@ - title: SPE Catalogs file: tutorials/euclid/5_Euclid_intro_SPE_catalog.md section: Euclid - description: Join the DR1 SPE, MER, and PHZ catalogs to select galaxies with reliable redshifts and well-measured H-alpha lines, see what each criterion removes, and plot the spectra and line measurements of selected and rejected galaxies. + description: Join the DR1 SPE tables with the MER and PHZ catalogs, filter them for galaxies with a strong H-alpha line following the DR1 SPE paper, see what each criterion removes, and plot the spectra and line measurements of galaxies that pass or fail the criteria. - title: Early Release Observations (ERO) - Star Clusters file: tutorials/euclid/Euclid_ERO.md section: Euclid diff --git a/tutorials/euclid/5_Euclid_intro_SPE_catalog.md b/tutorials/euclid/5_Euclid_intro_SPE_catalog.md index 0a4d302f..bcc4da0d 100644 --- a/tutorials/euclid/5_Euclid_intro_SPE_catalog.md +++ b/tutorials/euclid/5_Euclid_intro_SPE_catalog.md @@ -18,6 +18,7 @@ authors: - name: Dan Masters - name: Harry Teplitz - name: Jaladh Singhal + - name: Keunho Kim - name: Shoubaneh Hemmati - name: Vandana Desai - name: Troy Raen @@ -71,6 +72,8 @@ We rely on ``astroquery`` features that have been recently added, so please make ``` ```{code-cell} ipython3 +from urllib.error import HTTPError + import matplotlib.pyplot as plt import numpy as np @@ -120,9 +123,9 @@ In this section we only look at which tables are available and what their column Like the MER catalog (see the [MER catalogs notebook](2_Euclid_intro_MER_catalog.md)), the DR1 SPE catalogs are split by survey, with a `_wide_survey` and a `_deep` version of each table: * `euclid_dr1_spectro_z_spe_galaxy_candidates_*` -- the spectroscopic redshift candidates of each source fitted with galaxy templates, ranked by `spe_rank` (0 is the best candidate), with their probability `spe_z_prob` and reliability `spe_z_rel`. -* `euclid_dr1_spectro_line_features_spe_line_features_*` -- the line measurements for each redshift candidate (matched on `object_id` and `spe_rank`): every line of a reference list (from Lyα to the Paschen lines) is measured at its expected wavelength at that redshift, with its wavelength, flux, signal-to-noise ratio, and quality, from a Gaussian fit (`_gf`) and from direct integration (`_di`). Lines are measured whether or not they are detected, so most rows are non-detections (for example `spe_line_snr_gf = -99`), and a line can have several rows with different `spe_line_qual_gf` values. +* `euclid_dr1_spectro_line_features_spe_line_features_*` -- the line measurements for each redshift candidate (matched on `object_id` and `spe_rank`): the lines are measured at their expected wavelengths at that redshift, with their wavelength, flux, signal-to-noise ratio, and quality, from a Gaussian fit (`_gf`) and from direct integration (`_di`). The 11 main lines (Lyα, [OII], Hβ, [OIII], Hα, [NII]6585, [SII]6718, [SIII]9068, [SIII]9530, Paγ, and Paβ) are measured for every candidate, whether or not they are detected, so many rows are non-detections (for example `spe_line_snr_gf = -99`); other lines are only stored when they are detected. A line can have several rows, one for each grism, with different `spe_line_qual_gf` values. * `euclid_dr1_spectro_z_spe_quality_*` -- one row per source processed by SPE, including sources without a redshift candidate, describing its spectrum and the processing: grism(s), wavelength range, number of valid pixels, number of dithers, and error and warning flags. -* `euclid_dr1_spectro_z_spe_classification_*` -- the classification of each source as a galaxy, star, or QSO from its spectrum. +* `euclid_dr1_spectro_z_spe_classification_*` -- the classification of each source as a galaxy, star, or QSO from its spectrum. The galaxy model only fits emission lines, so galaxies with a strong continuum and weak lines are often best fitted by the QSO model, which in DR1 also includes templates of quiescent, star-forming, and type 2 AGN galaxies: not all of the sources classified as QSO are quasars. The SPE tables have no coordinates; they are joined to the MER catalog of the same survey on `object_id`. In this notebook we work in the Euclid Deep Field North, so we use the deep tables. @@ -165,9 +168,10 @@ A few highlights of the columns we use below: - `object_id`, the MER object ID used to join the SPE tables with each other and with the MER and PHZ catalogs; - `spe_rank`, the rank of the redshift candidate, 0 for the best one; - `spe_z`, `spe_z_prob`, and `spe_z_rel`, the spectroscopic redshift, its probability, and its reliability; -- `spe_line_name`, the name of the detected line, for example `'Halpha'`; +- `spe_line_name`, the name of the measured line, for example `'Halpha'`; - `spe_line_central_wl_gf`, `spe_line_flux_gf`, and `spe_line_snr_gf`, the central wavelength (in Angstrom), flux (in erg s-1 cm-2), and signal-to-noise ratio of the line from the Gaussian fit; - `spe_line_qual_gf`, the quality of the Gaussian fit measurements; +- we use the Gaussian fit measurements throughout: in DR1, the direct integration fluxes (`_di`) are about 15% larger than the Gaussian fit ones, because of a too aggressive continuum subtraction (Le Brun et al. 2026); - `spe_npix` and `spe_n_dith_med` in the quality table, the number of valid pixels and the median number of dithers of the spectrum. ``` @@ -208,16 +212,17 @@ Only a fraction of them have an SPE redshift, and fewer still have a redshift an The SPE tables can be filtered on any of their columns, and the right filters depend on the science case. In this section, we show how to combine the SPE redshift, quality, and lines tables with the MER and PHZ catalogs, using one common case as an example: selecting galaxies with a strong, well-measured Hα line, the main target of the Euclid spectroscopic survey. -The criteria below are one recipe for this case, following the recommendations of the Euclid SPE team. -They are deliberately strict, and are not a definition of a "good" spectrum: a galaxy at z > 1.85, for example, has its Hα line beyond the end of the red grism spectrum and fails criterion 2, but can have an excellent spectrum and redshift from its [OIII] or [OII] lines. +The criteria below are one recipe for this case, adapted from the ones used by Le Brun et al. (2026, Sect. 4.2) to build a reference sample of Hα emitters in the EDF-N, which agrees with DESI redshifts for about 99% of the galaxies. +This is the redshift range and the line that the Euclid spectroscopic survey targets, and for which the SPE galaxy model is optimized. +The criteria are deliberately strict, and are not a definition of a "good" spectrum: a galaxy at z > 1.85, for example, has its Hα line beyond the end of the red grism spectrum and fails criterion 2, but can have an excellent spectrum and redshift from its [OIII] or [OII] lines, and a galaxy at z < 0.85 can have its Hα line well measured in the blue grism spectrum of the Euclid Deep Fields. For another science case, drop or change the criteria that do not apply, for example the redshift range and the line in criteria 2 and 6. -1. **High-confidence redshift**: the best redshift candidate (`spe_rank = 0`) has a high probability (`spe_z_prob > 0.99`) and reliability (`spe_z_rel > 0.05`). -2. **Hα in the spectrum**: the redshift is between 0.45 and 1.85, the range in which Hα falls in the blue grism (0.93 to 1.37 micron) or the red grism (1.2 to 1.89 micron), away from the edges of the spectra. -3. **Both grisms, full wavelength coverage and depth**: in the quality table, the source has both a blue and a red grism spectrum (`spe_grism = 'BGS+RGS'`; drop this requirement for the Euclid Wide Survey, which only has red grism spectra), its combined spectrum has at least 800 valid pixels (`spe_npix > 800`, out of at most about 850 for the combined blue and red grism spectrum), so almost none of its wavelength range is masked, and the median number of dithered exposures per pixel is greater than 15 (`spe_n_dith_med > 15`, about 20 for most sources in this region), so it reaches close to the full depth of the survey. -4. **Extended, non-spurious source**: in the MER catalog, the source is not point-like (`point_like_prob < 0.05`, which removes most stars, but also compact galaxies and quasars), is unlikely to be spurious (`spurious_prob < 0.5`), and is not excessively large (`kron_radius < 50`), which removes nearby galaxies whose lines can be mistaken for Hα. -5. **Consistent with the photometric redshift**: the spectroscopic redshift is within the 95% interval of the photometric redshift of the PHZ catalog (`phz_95_int_1` to `phz_95_int_2`), a check against a wrong line identification. -6. **Strong, well-measured Hα line**: the Gaussian fit of the Hα line is good in at least one of the two grisms, and the combined measurement of the two grisms has a signal-to-noise ratio greater than 10 and a flux greater than 2×10-16 erg s-1 cm-2. +1. **High-confidence redshift**: the best redshift candidate (`spe_rank = 0`) has a high probability (`spe_z_prob > 0.99`, as in Sect. 4.1 of the paper, while Sect. 4.2 uses 0.94; this is the fraction of the redshift probability distribution in its peak) and is not flagged as unreliable (`spe_z_rel > 0.05`, a reliability from a machine-learning classifier, which in DR1 is trained on simulations, so only a low threshold is recommended). +2. **Hα in the red grism spectrum**: the redshift is between 0.85 and 1.85, the range in which Hα falls in the red grism spectrum (1.21 to 1.89 micron), away from its edges. At z < 1.08, Hα is also in the blue grism spectrum (up to 1.37 micron). +3. **Both grisms, full wavelength coverage and depth**: in the quality table, the source has both a blue and a red grism spectrum (`spe_grism = 'BGS+RGS'`; drop this requirement for the Euclid Wide Survey, which only has red grism spectra), its combined spectrum has at least 800 valid pixels (`spe_npix >= 800`, out of at most about 850 for the combined blue and red grism spectrum), so almost none of its wavelength range is masked, and the median number of dithered exposures per pixel is at least 15 (`spe_n_dith_med >= 15`, about 20 for most sources in this region), so it reaches close to the full depth of the survey. +4. **Extended, non-spurious source**: in the MER catalog, the source is not point-like (`point_like_prob < 0.05`, which removes most stars, but also compact galaxies and quasars), is unlikely to be spurious (`spurious_prob < 0.5`), and is not excessively large (`kron_radius < 50`): a large galaxy is most likely nearby, and a redshift in the Hα range is then most likely wrong. These are the criteria of the Euclid Galaxy Clustering Science Working Group. +5. **Consistent with the photometric redshift**: the spectroscopic redshift is within the 95% interval of the photometric redshift of the PHZ catalog (`phz_95_int_1` to `phz_95_int_2`), a check against a wrong line identification: Le Brun et al. (2026) find that with this criterion, 98% of the redshifts are correct at 0.84 < z < 1.6 in the EDF-N. +6. **Strong, well-measured Hα line**: the Gaussian fit of the Hα line is good in at least one of the two grisms, and the combined measurement of the two grisms has a signal-to-noise ratio greater than 10 and a flux greater than 1.5×10-16 erg s-1 cm-2, as in the visually inspected sample of the paper. ### 3.1 Get the redshifts, spectrum quality, and morphology of the sources @@ -255,10 +260,10 @@ def criteria(table): z = values(table, 'spe_z') return { '1. High-confidence redshift': (values(table, 'spe_z_prob') > 0.99) & (values(table, 'spe_z_rel') > 0.05), - '2. Hα in the spectrum (0.45 < z < 1.85)': (z > 0.45) & (z < 1.85), + '2. Hα in the red grism (0.85 < z < 1.85)': (z > 0.85) & (z < 1.85), '3. Both grisms, full coverage and depth': ((table['spe_grism'] == 'BGS+RGS') - & (values(table, 'spe_npix') > 800) - & (values(table, 'spe_n_dith_med') > 15)), + & (values(table, 'spe_npix') >= 800) + & (values(table, 'spe_n_dith_med') >= 15)), '4. Extended, non-spurious source': ((values(table, 'point_like_prob') < 0.05) & (values(table, 'spurious_prob') < 0.5) & (values(table, 'kron_radius') < 50)), @@ -282,24 +287,29 @@ cut_flow(criteria(candidates)) ### 3.2 Add the H-alpha measurements As we saw in section 1, the lines table has several rows for each line of a source: one for each grism in which the line is measured. -The last digit of the quality flag `spe_line_qual_gf` tells which spectrum the measurement comes from, and its middle digit tells how good it is: +The quality flag `spe_line_qual_gf` has three digits, described in the [DR1 documentation of the SPE catalogs](http://st-dm.pages.euclid-sgs.uk/data-product-doc/dmdr1/spedpd/dpcards/spe_spepfoutputcatalog.html): the first digit is the measurement method (2 for the Gaussian fit), the last digit tells which spectrum the measurement comes from, and the middle digit tells how good it is: | Digit | Value | Meaning | |---|---|---| | last (grism) | 1, 2, 3 | blue grism, red grism, blue and red grisms combined | -| middle (quality) | 1 | good measurement | +| middle (quality) | 0 | not measured | +| | 1 | good measurement | | | 2 | low signal-to-noise ratio (below 2) | +| | 3 | positive equivalent width (probably a negative continuum) | +| | 4 | combined measurement only: the blue and red grism measurements are not both good | | | 5 | acceptable, but up to 15% of the pixels of the line are missing | +| | 7 | combined measurement only: error when combining the blue and red grism measurements | | | 8 | unreliable, more than 15% of the pixels of the line are missing | | | 9 | the line is outside of the wavelength range (with a negative sign, for example -92) | For each galaxy, we get all of the Hα measurements in the wavelength range (`spe_line_qual_gf > 0`), and use them in two ways: -- the quality of the single-grism measurements (last digit 1 or 2) tells us whether Hα is well measured in at least one of the grisms: we require a quality digit of 1 (good) or 5 (acceptable), that is a quality flag of 211, 212, 251, or 252. Hα does not need to be well measured in both grisms: at z > 1.08 it is only in the red grism, and at z < 0.84 only in the blue grism; +- the quality of the single-grism measurements (last digit 1 or 2) tells us whether Hα is well measured in at least one of the grisms: we require a quality digit of 1 (good) or 5 (acceptable), that is a quality flag of 211, 212, 251, or 252. Hα does not need to be well measured in both grisms: at z > 1.08 it is only in the red grism, and near the start of the red grism spectrum it is often only well measured in the blue grism; - the combined measurement (last digit 3), which uses both grisms where the line falls in both, gives the flux and signal-to-noise ratio of the line. ```{note} -The quality digit of most of the combined measurements is 4 (for example 243), which is not described in the Q1 documentation of the quality flags; we find it for lines measured in a single grism. -This is why we take the quality from the single-grism measurements. +The combined measurement of most lines has a quality digit of 4 (flag 243), because the line is not well measured in both grisms, most often because it is outside of the wavelength range of one of them; the combined measurement then has the same flux and signal-to-noise ratio as the single-grism one. +Only lines well measured in both grisms have a combined flag of 213. +This is why we take the quality from the single-grism measurements, and only require Hα to be well measured in one of them. ``` The IRSA TAP service accepts at most 1000 values in an `IN (...)` list, so we query the Hα measurements in batches of 1000 object IDs. @@ -323,7 +333,7 @@ def query_halpha(object_ids, batch_size=1000): cuts = criteria(candidates) -in_range = cuts['1. High-confidence redshift'] & cuts['2. Hα in the spectrum (0.45 < z < 1.85)'] +in_range = cuts['1. High-confidence redshift'] & cuts['2. Hα in the red grism (0.85 < z < 1.85)'] halpha_rows = query_halpha(list(candidates['object_id'][in_range])) grism_digit = halpha_rows['spe_line_qual_gf'] % 10 @@ -348,7 +358,7 @@ def all_criteria(table): cuts = criteria(table) cuts['6. Strong, well-measured Hα'] = (np.ma.filled(table['halpha_good_in_a_grism'], False) & (values(table, 'halpha_snr_gf') > 10) - & (values(table, 'halpha_flux_gf') > 2e-16)) + & (values(table, 'halpha_flux_gf') > 1.5e-16)) return cuts @@ -357,8 +367,8 @@ passes_all = np.logical_and.reduce(list(cuts.values())) cut_flow(cuts) ``` -Out of the tens of thousands of MER sources in the region, about a hundred galaxies pass all the criteria of this strict recipe. -The Hα criterion removes most of the galaxies left after the first five criteria, mainly because of the flux cut: most of the Hα lines are fainter than 2×10-16 erg s-1 cm-2, the flux limit of the Euclid Wide Survey, which the deep fields go below (see section 4). +Out of the tens of thousands of MER sources in the region, about 130 galaxies pass all the criteria of this strict recipe. +The Hα criterion removes most of the galaxies left after the first five criteria, mainly because of the flux cut: most of the Hα lines are fainter than 1.5×10-16 erg s-1 cm-2, close to the flux limit of the Euclid Wide Survey (2×10-16 erg s-1 cm-2), which the deep fields go below (see section 4). Let's sort the galaxies that pass all of the criteria by the signal-to-noise ratio of their Hα line: ```{code-cell} ipython3 @@ -372,12 +382,12 @@ strong_halpha['object_id', 'spe_z', 'phz_median', 'halpha_flux_gf', 'halpha_snr_ Let's compare the redshifts of the galaxies that pass all of the criteria with those of all the galaxies with a high-confidence redshift in the same range, and look at their Hα fluxes as a function of redshift: ```{code-cell} ipython3 -high_confidence = cuts['1. High-confidence redshift'] & cuts['2. Hα in the spectrum (0.45 < z < 1.85)'] +high_confidence = cuts['1. High-confidence redshift'] & cuts['2. Hα in the red grism (0.85 < z < 1.85)'] has_halpha = values(galaxies, 'halpha_flux_gf') > 0 fig, (ax_z, ax_flux) = plt.subplots(1, 2, figsize=(13, 4.5)) -bins = np.arange(0.45, 1.86, 0.05) +bins = np.arange(0.85, 1.86, 0.05) ax_z.hist(galaxies['spe_z'][high_confidence], bins=bins, color='lightgrey', label='High-confidence redshift') ax_z.hist(galaxies['spe_z'][passes_all], bins=bins, color='tab:red', label='All criteria') ax_z.set_xlabel('Spectroscopic redshift') @@ -389,7 +399,7 @@ ax_flux.scatter(galaxies['spe_z'][high_confidence & has_halpha], values(galaxies points = ax_flux.scatter(galaxies['spe_z'][passes_all], values(galaxies, 'halpha_flux_gf')[passes_all], s=12, c=values(galaxies, 'halpha_snr_gf')[passes_all], cmap='viridis', vmin=10, vmax=50, label='All criteria') -ax_flux.axhline(2e-16, color='grey', linestyle='--', linewidth=0.8) +ax_flux.axhline(1.5e-16, color='grey', linestyle='--', linewidth=0.8) ax_flux.set_yscale('log') ax_flux.set_ylim(1e-17, 3e-15) ax_flux.set_xlabel('Spectroscopic redshift') @@ -415,7 +425,15 @@ def read_spectra(object_id): WHERE objectid = {object_id} """ urls = Irsa.query_tap(query).to_table()['url'] - return [QTable.read(url) for url in urls] + spectra = [] + for url in urls: + # TODO: remove the try/except once IRSA's spectrumDM service converts all of the DR1 HDUs + # (a few of them currently return an HTTP 422 error) + try: + spectra.append(QTable.read(url)) + except HTTPError as error: + print(f'Could not read a spectrum of object {object_id}: {error}') + return spectra def bad_flux_bins(spectrum): @@ -436,7 +454,7 @@ def best_spectra(spectra): return best ``` -We plot the usable flux bins as solid lines and the flux bins not to use as dotted lines, set the y-axis range from the usable flux bins, and mark the expected wavelengths of some of the strongest emission lines at the redshift of the galaxy: +We plot the usable flux bins as solid lines and the flux bins not to use as dotted lines, set the y-axis range from the usable flux bins (ignoring the few highest ones, which are often spikes at the edges of the spectra), and mark the expected wavelengths of some of the strongest emission lines at the redshift of the galaxy: ```{code-cell} ipython3 grism_colors = {'Red grism': 'tab:red', 'Blue grism': 'tab:blue'} @@ -455,7 +473,7 @@ def plot_spectra(ax, spectra, title, line_wavelengths): good_flux = np.concatenate([spectrum['SIGNAL'][~bad_flux_bins(spectrum)].filled(np.nan) for spectrum in spectra.values()]) - low, high = np.nanpercentile(good_flux, [1, 100]) + low, high = np.nanpercentile(good_flux, [1, 99.5]) ax.set_ylim(low - 0.1 * (high - low), high + 0.3 * (high - low)) # Mark the lines in the wavelength range, with a single label for lines closer than 100 Angstrom @@ -475,10 +493,11 @@ def plot_spectra(ax, spectra, title, line_wavelengths): ax.set_title(title, fontsize='small') -def plot_galaxy(ax, galaxy, title=None): +def plot_galaxy(ax, galaxy, title=None, best=None): z = galaxy['spe_z'] lines = {line: rest_wavelength * (1 + z) for line, rest_wavelength in emission_lines.items()} - best = best_spectra(read_spectra(galaxy['object_id'])) + if best is None: + best = best_spectra(read_spectra(galaxy['object_id'])) description = f"Object {galaxy['object_id']}, z = {z:.3f}, Hα S/N = {galaxy['halpha_snr_gf']:.0f}" plot_spectra(ax, best, f'{title}\n{description}' if title else description, lines) ``` @@ -489,11 +508,16 @@ Here are 20 of the galaxies that pass all of the criteria, spread evenly from th ```{code-cell} ipython3 n_rows, n_columns = 4, 5 -examples = strong_halpha[np.linspace(0, len(strong_halpha) - 1, n_rows * n_columns).astype(int)] +positions = np.linspace(0, len(strong_halpha) - 1, n_rows * n_columns).astype(int) fig, axes = plt.subplots(n_rows, n_columns, figsize=(25, 14), sharex=True) -for ax, galaxy in zip(axes.flat, examples): - plot_galaxy(ax, galaxy) +for ax, position in zip(axes.flat, positions): + # If one of the spectra of a galaxy cannot be read, plot the next galaxy of the list instead + for galaxy in strong_halpha[position:]: + best = best_spectra(read_spectra(galaxy['object_id'])) + if len(best) == 2: + break + plot_galaxy(ax, galaxy, best=best) # Only label the outer axes for ax in axes[:, 1:].flat: ax.set_ylabel('') @@ -527,7 +551,7 @@ plt.tight_layout() plt.show() ``` -The galaxy on the left fails the grism, coverage, and depth criterion only because the median number of dithered exposures of its spectrum is 14, just below our threshold of 15: its Hα line is clearly seen in the blue grism spectrum, and the spectrum is perfectly usable for many purposes. This is a reminder that the thresholds of a strict recipe like this one also reject some good spectra. +The galaxy on the left fails the grism, coverage, and depth criterion because its spectrum has 780 valid pixels and a median of 14 dithered exposures, just below our thresholds of 800 and 15: its blue grism spectrum is noisy, but its Hα line is clearly seen in the red grism spectrum, and its redshift is consistent with its photometric redshift. This is a reminder that the thresholds of a strict recipe like this one also reject some usable spectra. The galaxy on the right has a strong line at 1.23 micron, which the SPE processing identified as Hα, but its redshift is outside of the 95% interval of its photometric redshift. The line is only seen in the blue grism spectrum, although the red grism spectrum also covers this wavelength: it is most likely not a line of this galaxy, but a contamination of the blue grism spectrum by a neighboring source, and the photometric redshift criterion rightly rejects this galaxy. @@ -571,15 +595,17 @@ plt.show() ```{warning} Be careful when using line ratios from the DR1 SPE lines catalog. -At the resolution of the NISP spectra, Hα is blended with the [NII] lines, and the Gaussian fits of the blend are not independent: the [NII]6585 line has the same signal-to-noise ratio and FWHM as Hα, and the [NII]/Hα ratio is poorly constrained. +At the resolution of the NISP spectra, Hα is blended with the [NII] doublet: they are fitted together, with the same width and fixed relative positions, so the [NII]/Hα ratio comes from deblending the two and is poorly constrained. +For doublets fitted together, such as [OIII] and [NII], the flux is the total flux of the doublet: multiply it by 2/3 to get the flux of [OIII]5008 or [NII]6585 alone (Le Brun et al. 2026). Hβ and [OIII] fall in the blue grism for most of the galaxies in the Euclid Deep Fields, while Hα falls in the red grism, so ratios such as Hα/Hβ also depend on the relative calibration of the two grisms. ``` ## References -- Euclid Collaboration et al. 2025, *Euclid Quick Data Release (Q1): Characteristics and limitations of the spectroscopic measurements*, [arXiv:2503.15308](https://arxiv.org/abs/2503.15308) -- [Euclid SPE output catalog data product description](https://euclid.esac.esa.int/dr/q1/dpdd/spedpd/dpcards/spe_spepfoutputcatalog.html), which describes the columns of the SPE catalogs and the line quality flags - +- Euclid Collaboration: Le Brun, V., Moresco, M., Bethermin, M., et al. 2026, *Euclid Data Release 1 (DR1): Performances of the redshift measurement and spectroscopic analysis* + +- Euclid Collaboration: Le Brun, V., Bethermin, M., Moresco, M., et al. 2026, A&A, 711, A7, *Euclid Quick Data Release (Q1): Characteristics and limitations of the spectroscopic measurements*, [arXiv:2503.15308](https://arxiv.org/abs/2503.15308) +- [Euclid DR1 SPE output catalog data product description](http://st-dm.pages.euclid-sgs.uk/data-product-doc/dmdr1/spedpd/dpcards/spe_spepfoutputcatalog.html), which describes the columns of the SPE catalogs and the line quality flags +++ From 8e94fd08a97892abd1584c6028668bc72354a337 Mon Sep 17 00:00:00 2001 From: xoubish Date: Mon, 9 Nov 2026 16:02:08 -0800 Subject: [PATCH 19/23] Remove the spectrumDM workaround, now fixed by IRSA --- .../euclid/5_Euclid_intro_SPE_catalog.md | 28 ++++--------------- 1 file changed, 6 insertions(+), 22 deletions(-) diff --git a/tutorials/euclid/5_Euclid_intro_SPE_catalog.md b/tutorials/euclid/5_Euclid_intro_SPE_catalog.md index bcc4da0d..a7af8914 100644 --- a/tutorials/euclid/5_Euclid_intro_SPE_catalog.md +++ b/tutorials/euclid/5_Euclid_intro_SPE_catalog.md @@ -72,8 +72,6 @@ We rely on ``astroquery`` features that have been recently added, so please make ``` ```{code-cell} ipython3 -from urllib.error import HTTPError - import matplotlib.pyplot as plt import numpy as np @@ -425,15 +423,7 @@ def read_spectra(object_id): WHERE objectid = {object_id} """ urls = Irsa.query_tap(query).to_table()['url'] - spectra = [] - for url in urls: - # TODO: remove the try/except once IRSA's spectrumDM service converts all of the DR1 HDUs - # (a few of them currently return an HTTP 422 error) - try: - spectra.append(QTable.read(url)) - except HTTPError as error: - print(f'Could not read a spectrum of object {object_id}: {error}') - return spectra + return [QTable.read(url) for url in urls] def bad_flux_bins(spectrum): @@ -493,11 +483,10 @@ def plot_spectra(ax, spectra, title, line_wavelengths): ax.set_title(title, fontsize='small') -def plot_galaxy(ax, galaxy, title=None, best=None): +def plot_galaxy(ax, galaxy, title=None): z = galaxy['spe_z'] lines = {line: rest_wavelength * (1 + z) for line, rest_wavelength in emission_lines.items()} - if best is None: - best = best_spectra(read_spectra(galaxy['object_id'])) + best = best_spectra(read_spectra(galaxy['object_id'])) description = f"Object {galaxy['object_id']}, z = {z:.3f}, Hα S/N = {galaxy['halpha_snr_gf']:.0f}" plot_spectra(ax, best, f'{title}\n{description}' if title else description, lines) ``` @@ -508,16 +497,11 @@ Here are 20 of the galaxies that pass all of the criteria, spread evenly from th ```{code-cell} ipython3 n_rows, n_columns = 4, 5 -positions = np.linspace(0, len(strong_halpha) - 1, n_rows * n_columns).astype(int) +examples = strong_halpha[np.linspace(0, len(strong_halpha) - 1, n_rows * n_columns).astype(int)] fig, axes = plt.subplots(n_rows, n_columns, figsize=(25, 14), sharex=True) -for ax, position in zip(axes.flat, positions): - # If one of the spectra of a galaxy cannot be read, plot the next galaxy of the list instead - for galaxy in strong_halpha[position:]: - best = best_spectra(read_spectra(galaxy['object_id'])) - if len(best) == 2: - break - plot_galaxy(ax, galaxy, best=best) +for ax, galaxy in zip(axes.flat, examples): + plot_galaxy(ax, galaxy) # Only label the outer axes for ax in axes[:, 1:].flat: ax.set_ylabel('') From 1517c21ecc2ef79ee1e49522c1d257bed7086a4c Mon Sep 17 00:00:00 2001 From: xoubish Date: Tue, 10 Nov 2026 14:17:32 -0800 Subject: [PATCH 20/23] Address review comments on notebook 5 --- .../euclid/5_Euclid_intro_SPE_catalog.md | 472 ++++++++++++++---- 1 file changed, 364 insertions(+), 108 deletions(-) diff --git a/tutorials/euclid/5_Euclid_intro_SPE_catalog.md b/tutorials/euclid/5_Euclid_intro_SPE_catalog.md index a7af8914..0c01648b 100644 --- a/tutorials/euclid/5_Euclid_intro_SPE_catalog.md +++ b/tutorials/euclid/5_Euclid_intro_SPE_catalog.md @@ -72,40 +72,31 @@ We rely on ``astroquery`` features that have been recently added, so please make ``` ```{code-cell} ipython3 -import matplotlib.pyplot as plt +import warnings + import numpy as np +import matplotlib.pyplot as plt +from matplotlib.lines import Line2D + from astropy.table import QTable, Table, join, vstack -from astropy import units as u from astropy.utils.data import conf from astropy.visualization import quantity_support +from astropy import units as u from astroquery.ipac.irsa import Irsa +# Suppress warnings about deprecated units in the spectrum files, and about the astropy cache directory +warnings.filterwarnings('ignore', message="The unit 'Angstrom' has been deprecated", category=u.UnitsWarning) +warnings.filterwarnings('ignore', message="The unit 'erg' has been deprecated", category=u.UnitsWarning) +warnings.filterwarnings('ignore', message='XDG_CACHE_HOME is set') + # Let matplotlib pick up the axis units from the quantities we plot quantity_support() # The Euclid spectrum files can take a while to read, # which can exceed astropy's default timeout limit. Increase it. conf.remote_timeout = 120 - -#suppress warnings about deprecated units and cache -import warnings -warnings.filterwarnings( - "ignore", - message="The unit 'Angstrom' has been deprecated", - category=u.UnitsWarning, -) - -warnings.filterwarnings( - "ignore", - message="The unit 'erg' has been deprecated", - category=u.UnitsWarning, -) -warnings.filterwarnings( - "ignore", - message="XDG_CACHE_HOME is set", -) ``` @@ -123,7 +114,7 @@ Like the MER catalog (see the [MER catalogs notebook](2_Euclid_intro_MER_catalog * `euclid_dr1_spectro_z_spe_galaxy_candidates_*` -- the spectroscopic redshift candidates of each source fitted with galaxy templates, ranked by `spe_rank` (0 is the best candidate), with their probability `spe_z_prob` and reliability `spe_z_rel`. * `euclid_dr1_spectro_line_features_spe_line_features_*` -- the line measurements for each redshift candidate (matched on `object_id` and `spe_rank`): the lines are measured at their expected wavelengths at that redshift, with their wavelength, flux, signal-to-noise ratio, and quality, from a Gaussian fit (`_gf`) and from direct integration (`_di`). The 11 main lines (Lyα, [OII], Hβ, [OIII], Hα, [NII]6585, [SII]6718, [SIII]9068, [SIII]9530, Paγ, and Paβ) are measured for every candidate, whether or not they are detected, so many rows are non-detections (for example `spe_line_snr_gf = -99`); other lines are only stored when they are detected. A line can have several rows, one for each grism, with different `spe_line_qual_gf` values. * `euclid_dr1_spectro_z_spe_quality_*` -- one row per source processed by SPE, including sources without a redshift candidate, describing its spectrum and the processing: grism(s), wavelength range, number of valid pixels, number of dithers, and error and warning flags. -* `euclid_dr1_spectro_z_spe_classification_*` -- the classification of each source as a galaxy, star, or QSO from its spectrum. The galaxy model only fits emission lines, so galaxies with a strong continuum and weak lines are often best fitted by the QSO model, which in DR1 also includes templates of quiescent, star-forming, and type 2 AGN galaxies: not all of the sources classified as QSO are quasars. +* `euclid_dr1_spectro_z_spe_classification_*` -- the classification of each source as a galaxy, star, or QSO from its spectrum (not used in this notebook). The SPE tables have no coordinates; they are joined to the MER catalog of the same survey on `object_id`. In this notebook we work in the Euclid Deep Field North, so we use the deep tables. @@ -212,15 +203,26 @@ In this section, we show how to combine the SPE redshift, quality, and lines tab The criteria below are one recipe for this case, adapted from the ones used by Le Brun et al. (2026, Sect. 4.2) to build a reference sample of Hα emitters in the EDF-N, which agrees with DESI redshifts for about 99% of the galaxies. This is the redshift range and the line that the Euclid spectroscopic survey targets, and for which the SPE galaxy model is optimized. -The criteria are deliberately strict, and are not a definition of a "good" spectrum: a galaxy at z > 1.85, for example, has its Hα line beyond the end of the red grism spectrum and fails criterion 2, but can have an excellent spectrum and redshift from its [OIII] or [OII] lines, and a galaxy at z < 0.85 can have its Hα line well measured in the blue grism spectrum of the Euclid Deep Fields. +The recipe is deliberately strict, and is not a definition of a "good" spectrum: a galaxy at z > 1.85, for example, fails criterion 2, but can have an excellent spectrum and redshift from its [OIII] or [OII] lines. For another science case, drop or change the criteria that do not apply, for example the redshift range and the line in criteria 2 and 6. -1. **High-confidence redshift**: the best redshift candidate (`spe_rank = 0`) has a high probability (`spe_z_prob > 0.99`, as in Sect. 4.1 of the paper, while Sect. 4.2 uses 0.94; this is the fraction of the redshift probability distribution in its peak) and is not flagged as unreliable (`spe_z_rel > 0.05`, a reliability from a machine-learning classifier, which in DR1 is trained on simulations, so only a low threshold is recommended). -2. **Hα in the red grism spectrum**: the redshift is between 0.85 and 1.85, the range in which Hα falls in the red grism spectrum (1.21 to 1.89 micron), away from its edges. At z < 1.08, Hα is also in the blue grism spectrum (up to 1.37 micron). -3. **Both grisms, full wavelength coverage and depth**: in the quality table, the source has both a blue and a red grism spectrum (`spe_grism = 'BGS+RGS'`; drop this requirement for the Euclid Wide Survey, which only has red grism spectra), its combined spectrum has at least 800 valid pixels (`spe_npix >= 800`, out of at most about 850 for the combined blue and red grism spectrum), so almost none of its wavelength range is masked, and the median number of dithered exposures per pixel is at least 15 (`spe_n_dith_med >= 15`, about 20 for most sources in this region), so it reaches close to the full depth of the survey. -4. **Extended, non-spurious source**: in the MER catalog, the source is not point-like (`point_like_prob < 0.05`, which removes most stars, but also compact galaxies and quasars), is unlikely to be spurious (`spurious_prob < 0.5`), and is not excessively large (`kron_radius < 50`): a large galaxy is most likely nearby, and a redshift in the Hα range is then most likely wrong. These are the criteria of the Euclid Galaxy Clustering Science Working Group. -5. **Consistent with the photometric redshift**: the spectroscopic redshift is within the 95% interval of the photometric redshift of the PHZ catalog (`phz_95_int_1` to `phz_95_int_2`), a check against a wrong line identification: Le Brun et al. (2026) find that with this criterion, 98% of the redshifts are correct at 0.84 < z < 1.6 in the EDF-N. -6. **Strong, well-measured Hα line**: the Gaussian fit of the Hα line is good in at least one of the two grisms, and the combined measurement of the two grisms has a signal-to-noise ratio greater than 10 and a flux greater than 1.5×10-16 erg s-1 cm-2, as in the visually inspected sample of the paper. +| | Criterion | Columns and thresholds | Why | +|---|---|---|---| +| 1 | High-confidence redshift | `spe_rank = 0`, `spe_z_prob > 0.99`, `spe_z_rel > 0.05` | The best redshift candidate has a high probability (the fraction of the redshift probability distribution in its peak) and is not flagged as unreliable | +| 2 | Hα in the red grism spectrum | `0.85 < spe_z < 1.85` | Hα falls in the red grism spectrum (1.21 to 1.89 micron), away from its edges | +| 3 | Both grisms, full wavelength coverage and depth | `spe_grism = 'BGS+RGS'`, `spe_npix >= 800`, `spe_n_dith_med >= 15` (quality table) | The source has a blue and a red grism spectrum, with almost none of the wavelength range masked and close to the full depth of the survey | +| 4 | Extended, non-spurious source | `point_like_prob < 0.05`, `spurious_prob < 0.5`, `kron_radius < 50` (MER catalog) | Removes stars (and also compact galaxies and quasars), spurious detections, and very large galaxies, which are most likely nearby | +| 5 | Consistent with the photometric redshift | `phz_95_int_1 < spe_z < phz_95_int_2` (PHZ catalog) | A check against a wrong line identification | +| 6 | Strong, well-measured Hα line | Gaussian fit good in at least one grism; combined signal-to-noise ratio > 10 and flux > 1.5×10-16 erg s-1 cm-2 (lines table, see section 3.2) | The line is clearly detected, as in the visually inspected sample of the paper | + +```{note} +How the criteria compare with Le Brun et al. (2026), and when to change them: +- Criterion 1: `spe_z_prob > 0.99` is the threshold of Sect. 4.1 of the paper, while Sect. 4.2 uses 0.94. The reliability `spe_z_rel` comes from a machine-learning classifier, which in DR1 is trained on simulations, so only a low threshold is recommended. +- Criterion 2: at z < 1.08, Hα is also in the blue grism spectrum (up to 1.37 micron). +- Criterion 3: the combined blue and red grism spectrum has at most about 850 valid pixels, and most sources in this region have about 20 dithered exposures. Drop the `BGS+RGS` requirement for the Euclid Wide Survey, which only has red grism spectra. +- Criterion 4: these are the criteria of the Euclid Galaxy Clustering Science Working Group. +- Criterion 5: the paper finds that with this criterion, 98% of the redshifts are correct at 0.84 < z < 1.6 in the EDF-N. +``` ### 3.1 Get the redshifts, spectrum quality, and morphology of the sources @@ -231,45 +233,63 @@ We use a left outer join for the PHZ catalog, to keep the sources without a phot query_candidates = f""" SELECT mer.object_id, mer.ra, mer.dec, mer.point_like_prob, mer.spurious_prob, mer.kron_radius, - galaxy.spe_z, galaxy.spe_z_prob, galaxy.spe_z_rel, + spe.spe_z, spe.spe_z_prob, spe.spe_z_rel, quality.spe_grism, quality.spe_npix, quality.spe_n_dith_med, phz.phz_median, phz.phz_95_int_1, phz.phz_95_int_2 FROM {table_mer} AS mer -JOIN {table_galaxy_candidates} AS galaxy ON galaxy.object_id = mer.object_id +JOIN {table_galaxy_candidates} AS spe ON spe.object_id = mer.object_id JOIN {table_quality} AS quality ON quality.object_id = mer.object_id LEFT OUTER JOIN {table_photo_z} AS phz ON phz.object_id = mer.object_id WHERE mer.ra BETWEEN {ra_min} AND {ra_max} AND mer.dec BETWEEN {dec_min} AND {dec_max} - AND galaxy.spe_rank = 0 + AND spe.spe_rank = 0 """ candidates = Irsa.query_tap(query_candidates, async_job=True).to_table() print(f'{len(candidates)} of the {n_mer} MER sources have an SPE redshift') ``` We now apply the criteria one at a time and count how many galaxies are left after each of them. -Missing values (for example a missing photometric redshift) are masked in the table; we fill them with NaN, so that they fail every comparison: +Missing values (for example a missing photometric redshift) are masked in the table; we fill them with NaN, so that they fail every comparison. + +The helper functions of this notebook, like the two below, are in collapsed cells: click on a cell to expand it. ```{code-cell} ipython3 -def values(table, column): +--- +jupyter: + source_hidden: true +tags: [hide-cell] +--- +def as_float(table, column): + """Get a column of a table as floats, with NaN for the missing values. + + Parameters + ---------- + table : astropy.table.Table + Table to read the column from. + column : str + Name of the column. + + Returns + ------- + numpy.ndarray + Values of the column as floats, with the masked (missing) values set to NaN. + """ return np.ma.filled(np.ma.asarray(table[column], dtype=float), np.nan) -def criteria(table): - z = values(table, 'spe_z') - return { - '1. High-confidence redshift': (values(table, 'spe_z_prob') > 0.99) & (values(table, 'spe_z_rel') > 0.05), - '2. Hα in the red grism (0.85 < z < 1.85)': (z > 0.85) & (z < 1.85), - '3. Both grisms, full coverage and depth': ((table['spe_grism'] == 'BGS+RGS') - & (values(table, 'spe_npix') >= 800) - & (values(table, 'spe_n_dith_med') >= 15)), - '4. Extended, non-spurious source': ((values(table, 'point_like_prob') < 0.05) - & (values(table, 'spurious_prob') < 0.5) - & (values(table, 'kron_radius') < 50)), - '5. Consistent with the photo-z': (z > values(table, 'phz_95_int_1')) & (z < values(table, 'phz_95_int_2')), - } +def cut_flow(cuts): + """Count the galaxies left after applying each criterion in turn. + Parameters + ---------- + cuts : dict + Boolean array of each criterion (True for the galaxies that pass it), keyed by the name of the criterion. -def cut_flow(cuts): + Returns + ------- + astropy.table.Table + One row per criterion, with the number of galaxies that pass it and all of the criteria before it. + """ remaining = np.ones(len(next(iter(cuts.values()))), dtype=bool) steps, counts = ['SPE redshift'], [len(remaining)] for name, cut in cuts.items(): @@ -277,8 +297,37 @@ def cut_flow(cuts): steps.append(name) counts.append(remaining.sum()) return Table({'criterion': steps, 'n_galaxies': counts}) +``` + +```{code-cell} ipython3 +def criteria(table): + """Apply criteria 1 to 5 to a table of sources. + Parameters + ---------- + table : astropy.table.Table + Sources, with the columns of the redshift, quality, MER, and PHZ tables used by the criteria. + Returns + ------- + dict + Boolean array of each criterion (True for the sources that pass it), keyed by the name of the criterion. + """ + z = as_float(table, 'spe_z') + return { + '1. High-confidence redshift': (as_float(table, 'spe_z_prob') > 0.99) & (as_float(table, 'spe_z_rel') > 0.05), + '2. Hα in the red grism (0.85 < z < 1.85)': (z > 0.85) & (z < 1.85), + '3. Both grisms, full coverage and depth': ((table['spe_grism'] == 'BGS+RGS') + & (as_float(table, 'spe_npix') >= 800) + & (as_float(table, 'spe_n_dith_med') >= 15)), + '4. Extended, non-spurious source': ((as_float(table, 'point_like_prob') < 0.05) + & (as_float(table, 'spurious_prob') < 0.5) + & (as_float(table, 'kron_radius') < 50)), + '5. Consistent with the photo-z': (z > as_float(table, 'phz_95_int_1')) & (z < as_float(table, 'phz_95_int_2')), + } +``` + +```{code-cell} ipython3 cut_flow(criteria(candidates)) ``` @@ -315,6 +364,21 @@ We only query the galaxies that pass the first two criteria (the others are reje ```{code-cell} ipython3 def query_halpha(object_ids, batch_size=1000): + """Get the Hα measurements of a list of sources from the lines table. + + Parameters + ---------- + object_ids : list of int + MER object IDs of the sources. + batch_size : int, optional + Number of object IDs in each query (the IRSA TAP service accepts at most 1000). + + Returns + ------- + astropy.table.Table + One row per Hα measurement in the wavelength range, for the best (rank 0) redshift of each source: + one for each grism the line falls in, and one for the two grisms combined. + """ tables = [] for start in range(0, len(object_ids), batch_size): ids = ', '.join(str(object_id) for object_id in object_ids[start:start + batch_size]) @@ -328,8 +392,9 @@ def query_halpha(object_ids, batch_size=1000): """ tables.append(Irsa.query_tap(query, async_job=True).to_table()) return vstack(tables) +``` - +```{code-cell} ipython3 cuts = criteria(candidates) in_range = cuts['1. High-confidence redshift'] & cuts['2. Hα in the red grism (0.85 < z < 1.85)'] halpha_rows = query_halpha(list(candidates['object_id'][in_range])) @@ -353,24 +418,39 @@ We add the Hα criterion to the others, and look at the number of galaxies left ```{code-cell} ipython3 def all_criteria(table): + """Apply criteria 1 to 6 to a table of sources. + + Parameters + ---------- + table : astropy.table.Table + Sources, with the columns used by `criteria` and their Hα measurements. + + Returns + ------- + dict + Boolean array of each criterion (True for the sources that pass it), keyed by the name of the criterion. + """ cuts = criteria(table) cuts['6. Strong, well-measured Hα'] = (np.ma.filled(table['halpha_good_in_a_grism'], False) - & (values(table, 'halpha_snr_gf') > 10) - & (values(table, 'halpha_flux_gf') > 1.5e-16)) + & (as_float(table, 'halpha_snr_gf') > 10) + & (as_float(table, 'halpha_flux_gf') > 1.5e-16)) return cuts +``` - +```{code-cell} ipython3 cuts = all_criteria(galaxies) passes_all = np.logical_and.reduce(list(cuts.values())) cut_flow(cuts) ``` -Out of the tens of thousands of MER sources in the region, about 130 galaxies pass all the criteria of this strict recipe. +Out of the tens of thousands of MER sources in the region, about 130 galaxies pass all of the criteria. The Hα criterion removes most of the galaxies left after the first five criteria, mainly because of the flux cut: most of the Hα lines are fainter than 1.5×10-16 erg s-1 cm-2, close to the flux limit of the Euclid Wide Survey (2×10-16 erg s-1 cm-2), which the deep fields go below (see section 4). Let's sort the galaxies that pass all of the criteria by the signal-to-noise ratio of their Hα line: ```{code-cell} ipython3 strong_halpha = galaxies[passes_all] +if len(strong_halpha) == 0: + raise ValueError('No galaxy passes all of the criteria; try a larger region or relax some of the criteria.') strong_halpha.sort('halpha_snr_gf', reverse=True) strong_halpha['object_id', 'spe_z', 'phz_median', 'halpha_flux_gf', 'halpha_snr_gf'][:10] ``` @@ -381,7 +461,7 @@ Let's compare the redshifts of the galaxies that pass all of the criteria with t ```{code-cell} ipython3 high_confidence = cuts['1. High-confidence redshift'] & cuts['2. Hα in the red grism (0.85 < z < 1.85)'] -has_halpha = values(galaxies, 'halpha_flux_gf') > 0 +has_halpha = as_float(galaxies, 'halpha_flux_gf') > 0 fig, (ax_z, ax_flux) = plt.subplots(1, 2, figsize=(13, 4.5)) @@ -392,10 +472,10 @@ ax_z.set_xlabel('Spectroscopic redshift') ax_z.set_ylabel('Number of galaxies') ax_z.legend() -ax_flux.scatter(galaxies['spe_z'][high_confidence & has_halpha], values(galaxies, 'halpha_flux_gf')[high_confidence & has_halpha], +ax_flux.scatter(galaxies['spe_z'][high_confidence & has_halpha], as_float(galaxies, 'halpha_flux_gf')[high_confidence & has_halpha], s=5, color='lightgrey', label='High-confidence redshift') -points = ax_flux.scatter(galaxies['spe_z'][passes_all], values(galaxies, 'halpha_flux_gf')[passes_all], - s=12, c=values(galaxies, 'halpha_snr_gf')[passes_all], cmap='viridis', vmin=10, vmax=50, +points = ax_flux.scatter(galaxies['spe_z'][passes_all], as_float(galaxies, 'halpha_flux_gf')[passes_all], + s=12, c=as_float(galaxies, 'halpha_snr_gf')[passes_all], cmap='viridis', vmin=10, vmax=50, label='All criteria') ax_flux.axhline(1.5e-16, color='grey', linestyle='--', linewidth=0.8) ax_flux.set_yscale('log') @@ -410,54 +490,171 @@ plt.show() ## 5. Look at the spectra To see what the criteria mean, let's look at the spectra of some of the galaxies that pass all of them, and of some galaxies that fail one of them. -We find and read in the spectra as in the [SIR 1D spectra notebook](3_Euclid_intro_1D_spectra.md): -the table `euclid.objectid_spectrafile_association_dr1` lists the file and HDU of each spectrum of a source, and IRSA's spectrumDM service converts a given HDU into a VOTable, whose URL we build in the query. -A galaxy in the Euclid Deep Fields can have spectra from both the red and the blue grism, and several spectra of the same grism, so we keep the spectrum with the most usable flux bins for each grism: +We find and read in the spectra with the functions of the [SIR 1D spectra notebook](3_Euclid_intro_1D_spectra.md), which explains them step by step: +the table `euclid.objectid_spectrafile_association_dr1` lists the file and HDU of each spectrum of a source, and IRSA's spectrumDM service returns a given HDU as a VOTable, at a URL built from the file path and the HDU number. +A galaxy in the Euclid Deep Fields can have spectra from both the red and the blue grism, and several spectra of the same grism, so we keep the spectrum with the most usable flux bins for each grism. +If a spectrum cannot be read (for example, because of a network error or a timeout), we skip it and use the other spectra of the galaxy. ```{code-cell} ipython3 -def read_spectra(object_id): +--- +jupyter: + source_hidden: true +tags: [hide-cell] +--- +SPECTRUMDM_URL = 'https://irsacloud.ipac.caltech.edu/api/spectrumdm/convert/' +SPECTRUM_DATASET_ID = 'euclid_combspec_dr1' + + +def spectrum_url(path, hdu): + """Build the spectrumDM URL of one spectrum. + + Parameters + ---------- + path : str + Path of the spectrum file, from the ``path`` column of the association table. + hdu : int + Number of the HDU of the spectrum in the file. + + Returns + ------- + str + URL that returns the spectrum as a VOTable. + """ + return f'{SPECTRUMDM_URL}{path}?dataset_id={SPECTRUM_DATASET_ID}&hdu={hdu}' + + +def find_spectra(object_id): + """Find the spectra of an object in the spectrum association table. + + Parameters + ---------- + object_id : int + MER object ID of the source. + + Returns + ------- + astropy.table.Table + One row per spectrum, with the file path and HDU number of the spectrum, and its spectrumDM URL. + """ query = f""" - SELECT 'https://irsacloud.ipac.caltech.edu/api/spectrumdm/convert/' || path - || '?dataset_id=euclid_combspec_dr1&hdu=' || hdu AS url + SELECT objectid, tileid, path, hdu FROM euclid.objectid_spectrafile_association_dr1 WHERE objectid = {object_id} """ - urls = Irsa.query_tap(query).to_table()['url'] - return [QTable.read(url) for url in urls] + spectrum_files = Irsa.query_tap(query).to_table() + spectrum_files['url'] = [spectrum_url(path, hdu) for path, hdu in zip(spectrum_files['path'], spectrum_files['hdu'])] + return spectrum_files def bad_flux_bins(spectrum): - # Flux bins with an odd MASK value or a MASK value of 64 or more should not be used + """Find the flux bins of a spectrum that should not be used. + + Parameters + ---------- + spectrum : astropy.table.QTable + Spectrum, with a MASK column. + + Returns + ------- + numpy.ndarray + Boolean array, True for the flux bins with an odd MASK value or a MASK value of 64 or more. + """ mask = spectrum['MASK'].value return (mask % 2 == 1) | (mask >= 64) def grism_name(spectrum): - # Blue grism spectra start below 1.1 micron + """Tell which kind of grism a spectrum comes from. + + Parameters + ---------- + spectrum : astropy.table.QTable + Spectrum, with a WAVELENGTH column. + + Returns + ------- + str + ``'Blue grism'`` if the spectrum starts below 1.1 micron, ``'Red grism'`` otherwise. + """ return 'Blue grism' if spectrum['WAVELENGTH'].min() < 1.1 * u.micron else 'Red grism' def best_spectra(spectra): + """Pick the spectrum with the most usable flux bins for each kind of grism. + + Parameters + ---------- + spectra : list of astropy.table.QTable + Spectra of one object. + + Returns + ------- + dict + Best spectrum of each kind of grism, keyed by the name of the grism. + """ best = {} for spectrum in sorted(spectra, key=lambda spectrum: np.sum(~bad_flux_bins(spectrum)), reverse=True): best.setdefault(grism_name(spectrum), spectrum) return best + + +def get_best_spectra(object_id): + """Read the spectra of an object and pick the best one of each kind of grism. + + Spectra that cannot be read are skipped, with a message. + + Parameters + ---------- + object_id : int + MER object ID of the source. + + Returns + ------- + dict + Best spectrum of each kind of grism, keyed by the name of the grism; empty if no spectrum could be read. + """ + spectra = [] + for url in find_spectra(object_id)['url']: + try: + spectra.append(QTable.read(url)) + except OSError as error: + print(f'Skipping a spectrum that could not be read ({error}): {url}') + return best_spectra(spectra) ``` We plot the usable flux bins as solid lines and the flux bins not to use as dotted lines, set the y-axis range from the usable flux bins (ignoring the few highest ones, which are often spikes at the edges of the spectra), and mark the expected wavelengths of some of the strongest emission lines at the redshift of the galaxy: ```{code-cell} ipython3 +--- +jupyter: + source_hidden: true +tags: [hide-cell] +--- grism_colors = {'Red grism': 'tab:red', 'Blue grism': 'tab:blue'} emission_lines = {'[OII]': 3728.5 * u.AA, 'Hβ': 4862.7 * u.AA, '[OIII]': 5008.2 * u.AA, 'Hα': 6564.6 * u.AA, '[SII]': 6725.5 * u.AA} def plot_spectra(ax, spectra, title, line_wavelengths): + """Plot the spectra of an object on an axis, and mark its emission lines. + + Parameters + ---------- + ax : matplotlib.axes.Axes + Axis to plot on. + spectra : dict + Spectra to plot, keyed by the name of their grism, as returned by `best_spectra`. + title : str + Title of the plot. + line_wavelengths : dict + Observed wavelength of each line to mark, keyed by the label of the line. + Only the lines in the wavelength range of the spectra are marked. + """ for name, spectrum in spectra.items(): wavelength = spectrum['WAVELENGTH'].to(u.micron) bad = bad_flux_bins(spectrum) ax.plot(wavelength, np.ma.masked_where(bad, spectrum['SIGNAL']), color=grism_colors[name], - alpha=0.8, linewidth=0.8, label=name) + alpha=0.8, linewidth=0.8) ax.plot(wavelength, np.ma.masked_where(~bad, spectrum['SIGNAL']), color=grism_colors[name], alpha=0.5, linewidth=0.8, linestyle=':') @@ -479,63 +676,115 @@ def plot_spectra(ax, spectra, title, line_wavelengths): labels.append([line, wavelength]) for label, wavelength in labels: ax.text(wavelength.to_value(u.micron), 0.98, f' {label}', transform=ax.get_xaxis_transform(), - rotation=90, va='top', ha='right', fontsize=8) - ax.set_title(title, fontsize='small') + rotation=90, va='top', ha='right', fontsize=9) + ax.set_title(title, fontsize='medium') def plot_galaxy(ax, galaxy, title=None): + """Read and plot the best spectra of a galaxy, with the expected wavelengths of its emission lines. + + Parameters + ---------- + ax : matplotlib.axes.Axes + Axis to plot on. + galaxy : astropy.table.Row + Row of the galaxy, with its object ID, redshift, and Hα signal-to-noise ratio. + title : str, optional + First line of the title of the plot, above the description of the galaxy. + """ z = galaxy['spe_z'] - lines = {line: rest_wavelength * (1 + z) for line, rest_wavelength in emission_lines.items()} - best = best_spectra(read_spectra(galaxy['object_id'])) description = f"Object {galaxy['object_id']}, z = {z:.3f}, Hα S/N = {galaxy['halpha_snr_gf']:.0f}" - plot_spectra(ax, best, f'{title}\n{description}' if title else description, lines) + spectra = get_best_spectra(galaxy['object_id']) + if not spectra: + ax.set_title(f'{description}\nNo spectrum could be read', fontsize='medium') + return + lines = {line: rest_wavelength * (1 + z) for line, rest_wavelength in emission_lines.items()} + plot_spectra(ax, spectra, f'{title}\n{description}' if title else description, lines) + + +def add_legend(fig): + """Add a legend of the grism colors and line styles to a figure, outside of its axes. + + Parameters + ---------- + fig : matplotlib.figure.Figure + Figure to add the legend to; it should use the constrained layout. + """ + handles = [Line2D([], [], color=color, label=name) for name, color in grism_colors.items()] + handles.append(Line2D([], [], color='grey', linestyle=':', label='Do not use')) + handles.append(Line2D([], [], color='grey', linestyle='--', linewidth=0.6, label='Emission line')) + fig.legend(handles=handles, loc='outside right upper', fontsize='small') + + +def fails_only(cuts, criterion): + """Find the galaxies that fail one criterion and pass all of the others. + + Parameters + ---------- + cuts : dict + Boolean array of each criterion, keyed by the name of the criterion, as returned by `all_criteria`. + criterion : str + Name of the criterion to fail. + + Returns + ------- + numpy.ndarray + Boolean array, True for the galaxies that fail only this criterion. + """ + others = np.logical_and.reduce([cut for name, cut in cuts.items() if name != criterion]) + return others & ~cuts[criterion] ``` ### 5.1 Galaxies passing all of the criteria -Here are 20 of the galaxies that pass all of the criteria, spread evenly from the strongest to the weakest Hα signal-to-noise ratio (reading the spectra takes a minute or two): +Here are six of the galaxies that pass all of the criteria: five spread evenly from the strongest to the weakest Hα signal-to-noise ratio, and the galaxy with the strongest Hα line among those at z < 1.08, where Hα falls in both grisms: ```{code-cell} ipython3 -n_rows, n_columns = 4, 5 -examples = strong_halpha[np.linspace(0, len(strong_halpha) - 1, n_rows * n_columns).astype(int)] - -fig, axes = plt.subplots(n_rows, n_columns, figsize=(25, 14), sharex=True) -for ax, galaxy in zip(axes.flat, examples): - plot_galaxy(ax, galaxy) -# Only label the outer axes +examples = strong_halpha[np.linspace(0, len(strong_halpha) - 1, 5).astype(int)] +in_both_grisms = strong_halpha[(strong_halpha['spe_z'] < 1.08) & ~np.isin(strong_halpha['object_id'], examples['object_id'])] +examples = vstack([examples, in_both_grisms[:1]]) + +fig, axes = plt.subplots(3, 2, figsize=(14, 11), sharex=True, layout='constrained') +for ax, example in zip(axes.flat, examples): + plot_galaxy(ax, example) +# Only label the outer axes, and hide the unused ones for ax in axes[:, 1:].flat: ax.set_ylabel('') for ax in axes[:-1, :].flat: ax.set_xlabel('') -axes[0, 0].legend(loc='lower right', fontsize='small') -plt.tight_layout() +for ax in axes.flat[len(examples):]: + ax.set_visible(False) +add_legend(fig) plt.show() ``` -Hα is clearly detected in all of them, down to a signal-to-noise ratio of about 10. -Where the Hα line falls in both grisms (1.2 to 1.37 micron), it is seen at the same wavelength in the red and the blue grism spectra, and in many of the galaxies other lines, such as [OIII] in the blue grism, confirm the redshift. +Hα is detected in all of them, down to a signal-to-noise ratio of about 10. +Where the Hα line falls in both grisms (1.21 to 1.37 micron), it is seen at the same wavelength in the red and the blue grism spectra, and in some of the galaxies other lines, such as [OIII] or [OII] in the blue grism, confirm the redshift. + +```{note} +The blue and red grism spectra of a galaxy can have different continuum levels where they overlap (1.21 to 1.37 micron), sometimes by a factor of two or more, and the blue grism spectra can have spikes and other artifacts that are not flagged in the MASK column. +This is a property of the DR1 spectra, not a problem with how they are read in here: see Scodeggio et al. (2026) and the [Euclid DR1 documentation](https://euclid.caltech.edu/page/euclid-dr1) for the details. +``` ### 5.2 Galaxies failing one of the criteria Now let's look at galaxies that pass all of the criteria but one: for two of the criteria, we take the galaxy with the highest Hα signal-to-noise ratio among those that fail only this criterion. ```{code-cell} ipython3 -def fails_only(cuts, criterion): - others = np.logical_and.reduce([cut for name, cut in cuts.items() if name != criterion]) - return others & ~cuts[criterion] - - -fig, axes = plt.subplots(1, 2, figsize=(14, 4.5), sharex=True) +fig, axes = plt.subplots(1, 2, figsize=(14, 4.5), sharex=True, layout='constrained') for ax, criterion in zip(axes, ['3. Both grisms, full coverage and depth', '5. Consistent with the photo-z']): rejected = galaxies[fails_only(cuts, criterion)] + if len(rejected) == 0: + ax.set_title(f'No galaxy fails only criterion {criterion}', fontsize='medium') + continue rejected.sort('halpha_snr_gf', reverse=True) plot_galaxy(ax, rejected[0], f'Fails criterion {criterion}') -axes[0].legend(loc='lower right', fontsize='small') -plt.tight_layout() +axes[1].set_ylabel('') +add_legend(fig) plt.show() ``` -The galaxy on the left fails the grism, coverage, and depth criterion because its spectrum has 780 valid pixels and a median of 14 dithered exposures, just below our thresholds of 800 and 15: its blue grism spectrum is noisy, but its Hα line is clearly seen in the red grism spectrum, and its redshift is consistent with its photometric redshift. This is a reminder that the thresholds of a strict recipe like this one also reject some usable spectra. +The galaxy on the left fails the grism, coverage, and depth criterion because its spectrum has 780 valid pixels and a median of 14 dithered exposures, just below our thresholds of 800 and 15: its blue grism spectrum is noisy, but Hα is detected in the red grism spectrum (S/N = 15), and its redshift is consistent with its photometric redshift. Thresholds like these also reject some usable spectra. The galaxy on the right has a strong line at 1.23 micron, which the SPE processing identified as Hα, but its redshift is outside of the 95% interval of its photometric redshift. The line is only seen in the blue grism spectrum, although the red grism spectrum also covers this wavelength: it is most likely not a line of this galaxy, but a contamination of the blue grism spectrum by a neighboring source, and the photometric redshift criterion rightly rejects this galaxy. @@ -545,13 +794,13 @@ Hα is only one of the lines the SPE processing measures. Let's get all of the combined-grism line measurements of the galaxy with the strongest Hα line among those that pass all of the criteria, and keep those with a signal-to-noise ratio greater than 3: ```{code-cell} ipython3 -galaxy = strong_halpha[0] +strongest_galaxy = strong_halpha[0] query_lines = f""" SELECT spe_line_name, spe_line_central_wl_gf, spe_line_flux_gf, spe_line_flux_err_gf, spe_line_snr_gf, spe_line_ew_gf, spe_line_fwhm_gf, spe_line_qual_gf FROM {table_lines} -WHERE object_id = {galaxy['object_id']} +WHERE object_id = {strongest_galaxy['object_id']} AND spe_rank = 0 AND MOD(spe_line_qual_gf, 10) = 3 AND spe_line_snr_gf > 3 @@ -561,19 +810,24 @@ galaxy_lines = Irsa.query_tap(query_lines).to_table() galaxy_lines ``` -Besides Hα, lines such as [OIII] (in the blue grism) and [SII] are detected. -Note that the Hα and [NII]6585 lines have the same signal-to-noise ratio and FWHM: they are fitted together, as they are blended at the resolution of the spectra (see the warning below). +Besides Hα, the [SII] doublet is well detected, with a signal-to-noise ratio of about 11. +[OIII] and [SIII]9068 also pass our cut, but with a signal-to-noise ratio between 4 and 5: these are marginal measurements, and the lines cannot be seen by eye in the spectrum below. +Note that the Hα and [NII]6585 lines have the same signal-to-noise ratio and FWHM: they are fitted together, as they are blended at the resolution of the spectra. +The table gives an [NII]/Hα flux ratio of about 0.6 for this galaxy, but this ratio comes from deblending the two lines and is poorly constrained (see the warning below). -We plot the spectra of this galaxy again, now marking the lines at the central wavelengths measured by the Gaussian fits: +We plot the spectra of this galaxy again, now marking the lines at the central wavelengths measured by the Gaussian fits, with the same labels as above for the main lines of the catalog: ```{code-cell} ipython3 -best = best_spectra(read_spectra(galaxy['object_id'])) -measured_lines = {row['spe_line_name']: u.Quantity(galaxy_lines['spe_line_central_wl_gf'])[i] - for i, row in enumerate(galaxy_lines)} - -fig, ax = plt.subplots(figsize=(13, 4.5)) -plot_spectra(ax, best, f"Object {galaxy['object_id']}, z = {galaxy['spe_z']:.3f}: lines with S/N > 3", measured_lines) -ax.legend(loc='lower right', fontsize='small') +line_labels = {'LyA': 'Lyα', 'OII3727': '[OII]', 'Hbeta': 'Hβ', 'OIII5008': '[OIII]', 'Halpha': 'Hα', + 'NII6585': '[NII]', 'SII6718': '[SII]', 'SIII9068': '[SIII]9068', 'SIII9530': '[SIII]9530', + 'PaGamma': 'Paγ', 'PaBeta': 'Paβ'} +measured_lines = {line_labels.get(name, name): wavelength for name, wavelength + in zip(galaxy_lines['spe_line_name'], u.Quantity(galaxy_lines['spe_line_central_wl_gf']))} + +fig, ax = plt.subplots(figsize=(13, 4.5), layout='constrained') +plot_spectra(ax, get_best_spectra(strongest_galaxy['object_id']), + f"Object {strongest_galaxy['object_id']}, z = {strongest_galaxy['spe_z']:.3f}: lines with S/N > 3", measured_lines) +add_legend(fig) plt.show() ``` @@ -581,13 +835,15 @@ plt.show() Be careful when using line ratios from the DR1 SPE lines catalog. At the resolution of the NISP spectra, Hα is blended with the [NII] doublet: they are fitted together, with the same width and fixed relative positions, so the [NII]/Hα ratio comes from deblending the two and is poorly constrained. For doublets fitted together, such as [OIII] and [NII], the flux is the total flux of the doublet: multiply it by 2/3 to get the flux of [OIII]5008 or [NII]6585 alone (Le Brun et al. 2026). -Hβ and [OIII] fall in the blue grism for most of the galaxies in the Euclid Deep Fields, while Hα falls in the red grism, so ratios such as Hα/Hβ also depend on the relative calibration of the two grisms. +Which grism a line falls in depends on the redshift. In the Euclid Deep Fields, Hβ is in the blue grism spectrum at about 0.89 < z < 1.82 and [OIII] at about 0.84 < z < 1.74, while Hα is only in the red grism spectrum at z > 1.08. For the galaxies selected here (0.85 < z < 1.85) that are at z > 1.08, ratios such as Hα/Hβ therefore compare a line in the red grism with one in the blue grism, and also depend on the relative calibration of the two grisms. ``` ## References - Euclid Collaboration: Le Brun, V., Moresco, M., Bethermin, M., et al. 2026, *Euclid Data Release 1 (DR1): Performances of the redshift measurement and spectroscopic analysis* + +- Euclid Collaboration: Scodeggio, M., Copin, Y., et al. 2026, *Euclid Data Release 1 (DR1): From spectrograms to spectra: The SIR spectroscopic processing function*, A&A, in prep. - Euclid Collaboration: Le Brun, V., Bethermin, M., Moresco, M., et al. 2026, A&A, 711, A7, *Euclid Quick Data Release (Q1): Characteristics and limitations of the spectroscopic measurements*, [arXiv:2503.15308](https://arxiv.org/abs/2503.15308) - [Euclid DR1 SPE output catalog data product description](http://st-dm.pages.euclid-sgs.uk/data-product-doc/dmdr1/spedpd/dpcards/spe_spepfoutputcatalog.html), which describes the columns of the SPE catalogs and the line quality flags @@ -595,8 +851,8 @@ Hβ and [OIII] fall in the blue grism for most of the galaxies in the Euclid Dee ## About this Notebook -**Updated**: 2026-11-09 +**Updated**: 2026-11-12 **Contact:** [the IRSA Helpdesk](https://irsa.ipac.caltech.edu/docs/help_desk.html) with questions or reporting problems. -**Runtime:** As of the date above, this notebook takes about 2 to 5 minutes, depending on the load of the IRSA services, to run to completion on a machine with 8GB RAM and 4 CPU. +**Runtime:** As of the date above, this notebook takes about 2 to 3 minutes, depending on the load of the IRSA services, to run to completion on a machine with 8GB RAM and 4 CPU. From 795269583db2878ee481e68d686dcbf18fb438d4 Mon Sep 17 00:00:00 2001 From: xoubish Date: Tue, 10 Nov 2026 14:22:41 -0800 Subject: [PATCH 21/23] Reword the grism note --- tutorials/euclid/5_Euclid_intro_SPE_catalog.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tutorials/euclid/5_Euclid_intro_SPE_catalog.md b/tutorials/euclid/5_Euclid_intro_SPE_catalog.md index 0c01648b..9c825098 100644 --- a/tutorials/euclid/5_Euclid_intro_SPE_catalog.md +++ b/tutorials/euclid/5_Euclid_intro_SPE_catalog.md @@ -763,7 +763,7 @@ Where the Hα line falls in both grisms (1.21 to 1.37 micron), it is seen at the ```{note} The blue and red grism spectra of a galaxy can have different continuum levels where they overlap (1.21 to 1.37 micron), sometimes by a factor of two or more, and the blue grism spectra can have spikes and other artifacts that are not flagged in the MASK column. -This is a property of the DR1 spectra, not a problem with how they are read in here: see Scodeggio et al. (2026) and the [Euclid DR1 documentation](https://euclid.caltech.edu/page/euclid-dr1) for the details. +This is a limitation of the DR1 spectra, not a problem with how they are read in here: see Scodeggio et al. (2026) and the [Euclid DR1 documentation](https://euclid.caltech.edu/page/euclid-dr1) for the details. ``` ### 5.2 Galaxies failing one of the criteria From 971d4a46d2888db77142aec532c177277e0c2c19 Mon Sep 17 00:00:00 2001 From: xoubish Date: Wed, 11 Nov 2026 08:14:24 -0800 Subject: [PATCH 22/23] Point the notebook to the production IRSA services for DR1 --- tutorials/euclid/5_Euclid_intro_SPE_catalog.md | 6 ------ 1 file changed, 6 deletions(-) diff --git a/tutorials/euclid/5_Euclid_intro_SPE_catalog.md b/tutorials/euclid/5_Euclid_intro_SPE_catalog.md index 9c825098..6d7cd804 100644 --- a/tutorials/euclid/5_Euclid_intro_SPE_catalog.md +++ b/tutorials/euclid/5_Euclid_intro_SPE_catalog.md @@ -99,12 +99,6 @@ quantity_support() conf.remote_timeout = 120 ``` - -```{code-cell} ipython3 -# DR1 is not public yet, so we query IRSA's development server. -Irsa.tap_url = 'https://irsadev.ipac.caltech.edu/TAP' -``` - ## 1. Find the SPE catalog tables at IRSA The SPE catalogs are large, so we never download them: we query them on IRSA's server with ADQL through the Table Access Protocol (TAP) service, and get back only the rows we ask for. From 45c80283e62c70262a4cd382749e5e47084ca7ca Mon Sep 17 00:00:00 2001 From: xoubish Date: Wed, 11 Nov 2026 12:32:07 -0800 Subject: [PATCH 23/23] Require astroquery 0.4.11 for include_metadata_tables --- tutorials/euclid/5_Euclid_intro_SPE_catalog.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tutorials/euclid/5_Euclid_intro_SPE_catalog.md b/tutorials/euclid/5_Euclid_intro_SPE_catalog.md index 6d7cd804..800dbbf2 100644 --- a/tutorials/euclid/5_Euclid_intro_SPE_catalog.md +++ b/tutorials/euclid/5_Euclid_intro_SPE_catalog.md @@ -63,7 +63,7 @@ If you have questions about this notebook, please contact the [IRSA helpdesk](ht ## Imports ```{important} -We rely on ``astroquery`` features that have been recently added, so please make sure you have version v0.4.10 or newer installed. +We rely on ``astroquery`` features that have been recently added, so please make sure you have version v0.4.11 or newer installed. ``` ```{code-cell} ipython3