From 0a56307e5d45f8e9cc7ea3bc3c6b054887f6ea96 Mon Sep 17 00:00:00 2001 From: xoubish Date: Fri, 6 Nov 2026 10:46:47 -0800 Subject: [PATCH 1/7] Update the PHZ catalog notebook to Euclid DR1 Use the DR1 deep survey PHZ and MER tables (with the renamed interval columns), describe the four DR1 PHZ tables, and start with the catalog rather than the mosaic search. Add a comparison of the photometric redshifts with the DR1 SPE spectroscopic redshifts, using spe_z_prob and spe_z_rel to identify the reliable ones, and plot the spectrum of the brightest galaxy with a reliable spectroscopic redshift, retrieved by position with the SSA service, with its emission lines. Read the mosaic cutouts from the cloud and update the gallery description. --- notebook_metadata.yml | 2 +- .../euclid/4_Euclid_intro_PHZ_catalog.md | 467 ++++++++++-------- 2 files changed, 269 insertions(+), 200 deletions(-) diff --git a/notebook_metadata.yml b/notebook_metadata.yml index 12a8f26f..574ae4da 100644 --- a/notebook_metadata.yml +++ b/notebook_metadata.yml @@ -17,7 +17,7 @@ - title: PHZ Catalogs file: tutorials/euclid/4_Euclid_intro_PHZ_catalog.md section: Euclid - description: Join the PHZ and MER catalogs to query galaxies with quality redshifts in a box region, create MER mosaic cutouts with catalog overlays, and plot the brightest galaxy's SIR spectrum. + description: Join the DR1 PHZ and MER catalogs to select galaxies with good photometric redshifts in a region, overlay them on a MER mosaic cutout, and compare the photometric redshift of one of them with its SIR spectrum and spectroscopic redshift. - title: SPE Catalogs file: tutorials/euclid/5_Euclid_intro_SPE_catalog.md section: Euclid diff --git a/tutorials/euclid/4_Euclid_intro_PHZ_catalog.md b/tutorials/euclid/4_Euclid_intro_PHZ_catalog.md index 87618aee..2de96b70 100644 --- a/tutorials/euclid/4_Euclid_intro_PHZ_catalog.md +++ b/tutorials/euclid/4_Euclid_intro_PHZ_catalog.md @@ -4,11 +4,11 @@ jupytext: extension: .md format_name: myst format_version: 0.13 - jupytext_version: 1.16.7 + jupytext_version: 1.18.1 kernelspec: - display_name: Python 3 (ipykernel) - language: python name: python3 + display_name: python3 + language: python authors: - name: Tiffany Meshkat - name: Anahita Alavi @@ -21,9 +21,10 @@ authors: - name: Shoubaneh Hemmati - name: Vandana Desai - name: Troy Raen + - name: Jessica Krick --- -# Euclid Q1: PHZ Catalogs +# Euclid DR1: PHZ Catalogs +++ @@ -32,12 +33,12 @@ authors: +++ By the end of this tutorial, you will: -- Understand the basic characteristics of Euclid Q1 photo-z catalog and how to match it with MER mosaics. -- Understand what PHZ catalogs are available and how to view the columns in those catalogs. -- How to query with ADQL in the PHZ catalog to find galaxies between a redshift of 1.4 and 1.6. -- Pull and plot a spectrum of one of the galaxies in that catalog. -- Cutout an image of the galaxy to view it close up. -- Learn how to upload images and catalogs to Firefly to inspect individual sources in greater detail. +- know which Euclid DR1 PHZ (photometric redshift) catalog tables are available at IRSA, and how to view their columns; +- know how to join the PHZ and MER catalogs with ADQL to select galaxies with good photometric redshifts in a redshift range; +- know how to compare the photometric redshifts with the spectroscopic redshifts of the DR1 SPE catalog; +- know how to make a cutout of a MER mosaic and overplot the selected galaxies on it; +- know how to retrieve and plot the spectrum of one of the galaxies, and compare its photometric and spectroscopic redshifts; +- know how to load the image and catalog into Firefly to inspect individual sources interactively. +++ @@ -48,17 +49,37 @@ By the end of this tutorial, you will: Euclid launched in July 2023 as a European Space Agency (ESA) mission with involvement by NASA. The primary science goals of Euclid are to better understand the composition and evolution of the dark Universe. The Euclid mission is providing space-based imaging and spectroscopy as well as supporting ground-based imaging to achieve these primary goals. -These data will be archived by multiple global repositories, including IRSA, where they will support transformational work in many areas of astrophysics. +These data are archived by multiple global repositories, including IRSA, where they will support transformational work in many areas of astrophysics. -Euclid Quick Release 1 (Q1) consists of consists of ~30 TB of imaging, spectroscopy, and catalogs covering four non-contiguous fields: -Euclid Deep Field North (22.9 sq deg), Euclid Deep Field Fornax (12.1 sq deg), Euclid Deep Field South (28.1 sq deg), and LDN1641. +Euclid Data Release 1 (DR1) comprises about 1900 sq deg of imaging, spectroscopy, and catalogs from the Euclid Wide Survey (EWS), which will eventually cover about 14,000 sq deg, and from the Euclid Deep Survey (EDS), which covers about 53 sq deg in the three Euclid Deep Fields (North, South, and Fornax). +DR1 also includes several photometric redshift and instrument calibration fields, such as COSMOS, CDFS, and AEGIS. +It follows Quick Release 1 (Q1), which covered about 63 sq deg with a single pass over each of the three Euclid Deep Fields and the dark cloud LDN 1641; all of the DR1 data, including the areas observed in Q1, have been reprocessed with updated pipelines and calibrations. -Among the data products included in the Q1 release are multiple catalogs created by the PHZ Processing Function. -This notebook provides an introduction to the main PHZ catalog, which contains 61 columns describing the photometric redshift probability distribution, fluxes, and classification for each source. +Among the data products included in DR1 are the catalogs created by the PHZ processing function, which estimates the photometric redshift and the classification (star, galaxy, QSO) of every source in the MER catalog from its photometry. +This notebook provides an introduction to the main PHZ catalog, which contains the photometric redshift probability distribution, the unified fluxes, and the classification for each source. +We follow a typical path through the catalog: we select galaxies with good photometric redshifts in a redshift range, look at them on a MER mosaic, and compare the photometric redshift of one of them with its spectrum and spectroscopic redshift. +The PHZ processing for DR1 is described in detail in Euclid Collaboration: Tucci et al. (2026), see the [References](#references) section below. If you have questions about this notebook, please contact the [IRSA helpdesk](https://irsa.ipac.caltech.edu/docs/help_desk.html). +++ +## The PHZ catalog tables + +Like the MER catalog (see the [MER catalogs notebook](2_Euclid_intro_MER_catalog.md)), the DR1 PHZ catalogs are split by survey: + +* `euclid_dr1_phz_photo_z_wide` -- the main PHZ catalog of the **Euclid Wide Survey**: photometric redshifts, unified fluxes, and classification. +* `euclid_dr1_phz_photo_z_deep_survey` -- the same for the **Euclid Deep Survey** (the deep tiles of the Euclid Deep Fields North, South, and Fornax). +* `euclid_dr1_phz_classification_wide` and `euclid_dr1_phz_classification_deep_survey` -- the detailed classification probabilities (star, galaxy, QSO, globular cluster) behind the `phz_classification` column of the main catalogs. + +The PHZ tables have no coordinates; they are joined to the MER catalog of the same survey on `object_id`. +In this notebook we work in the Euclid Deep Field North, so we use the deep survey tables. + +```{note} +Some column names differ from the Q1 PHZ catalog: for example the redshift intervals are `phz_90_int_1` and `phz_90_int_2` in DR1 (`phz_90_int1` and `phz_90_int2` in Q1). +``` + ++++ + ## Imports ```{important} @@ -67,14 +88,13 @@ We rely on ``astroquery`` features that have been recently added, so please make ```{code-cell} ipython3 # Uncomment the next line to install dependencies if needed. -# !pip install matplotlib 'astropy>=5.3' 'astroquery>=0.4.11' fsspec firefly_client +# !pip install numpy matplotlib 'astropy>=5.3' 'astroquery>=0.4.10' fsspec s3fs firefly_client ``` ```{code-cell} ipython3 -import os -import re import io -import urllib +import json +import warnings import numpy as np import matplotlib.pyplot as plt @@ -82,101 +102,98 @@ import matplotlib.pyplot as plt from astropy.coordinates import SkyCoord from astropy.io import fits from astropy.nddata import Cutout2D -from astropy.table import QTable +from astropy.table import QTable, join from astropy import units as u -from astropy.utils.data import conf, download_file +from astropy.utils.data import conf from astropy.visualization import ImageNormalize, PercentileInterval, AsinhStretch, LogStretch, quantity_support from astropy.wcs import WCS from firefly_client import FireflyClient from astroquery.ipac.irsa import Irsa -# The Euclid spectrum files are large and the time it takes to read -# them can exceed astropy's default timeout limit. Increase it. -conf.remote_timeout = 120 -``` - -## 1. Find the MER Tile ID that corresponds to a given RA and Dec +# Suppress warnings about deprecated units in the spectrum files +warnings.filterwarnings('ignore', message="The unit 'Angstrom' has been deprecated", category=u.UnitsWarning) +warnings.filterwarnings('ignore', message="The unit 'erg' has been deprecated", category=u.UnitsWarning) -In this case, choose random coordinates to show a different MER mosaic image. Search a radius around these coordinates. - -```{code-cell} ipython3 -ra = 268 -dec = 66 -search_radius= 10 * u.arcsec +# Let matplotlib pick up the axis units from the quantities we plot +quantity_support() -coord = SkyCoord(ra, dec, unit='deg', frame='icrs') +# The Euclid spectrum files can take a while to read, +# which can exceed astropy's default timeout limit. Increase it. +conf.remote_timeout = 120 ``` -### Use IRSA to search for all Euclid data on this target - -This searches specifically in the euclid_DpdMerBksMosaic "collection" which is the MER images and catalogs. - -+++ - -```{note} -This table lists all MER mosaic images available in this position. These mosaics include the Euclid VIS, Y, J, H images, as well as ground-based telescopes which have been put on the same pixel scale. For more information, see the [Euclid documentation at IPAC](https://euclid.caltech.edu/page/euclid-faq-tech/). We use the ``facility`` argument below to query for Euclid images only. -``` + ```{code-cell} ipython3 -image_table = Irsa.query_sia(pos=(coord, search_radius), collection='euclid_DpdMerBksMosaic', facility='Euclid') +# DR1 is not public yet, so we query IRSA's development server. +Irsa.tap_url = 'https://irsadev.ipac.caltech.edu/TAP' +Irsa.sia_url = 'https://irsadev.ipac.caltech.edu/SIA' +Irsa.ssa_url = 'https://irsadev.ipac.caltech.edu/SSA' ``` -Note that there are various image types are returned as well, we filter out the science images from these: + ```{code-cell} ipython3 -science_images = image_table[image_table['dataproduct_subtype'] == 'science'] -science_images -``` +# TEMPORARY: the SSA service does not return DR1 spectra yet, so we emulate +# Irsa.query_ssa with a TAP query on IRSA's CAOM tables, which already have them. +from astropy.table import Table -Choose the VIS image and pull the filename and tileID -```{code-cell} ipython3 -filename = science_images[science_images['energy_bandpassname'] == 'VIS']['access_url'][0] -tileID = science_images[science_images['energy_bandpassname'] == 'VIS']['obs_id'][0][:9] +def query_ssa_from_caom(*, pos, radius, collection): + query = f""" + SELECT a.uri + FROM euclid.plane_euclid_dr1 AS p + JOIN euclid.observation_euclid_dr1 AS o ON o.obsid = p.obsid + JOIN euclid.artifact_euclid_dr1 AS a ON a.planeid = p.planeid + WHERE o.collection = '{collection}' + AND CONTAINS(p.pt, CIRCLE('ICRS', {pos.ra.deg}, {pos.dec.deg}, {radius.to(u.deg).value})) = 1 + """ + uris = Irsa.query_tap(query, async_job=True).to_table()['uri'] + return Table({'s_ra': [pos.ra.deg] * len(uris), 's_dec': [pos.dec.deg] * len(uris), + 'access_url': [f'https://irsadev.ipac.caltech.edu/{uri}' for uri in uris]}) -print(f'The MER tile ID for this object is : {tileID}') -``` -## 2. Download PHZ catalog from IRSA +Irsa.query_ssa = query_ssa_from_caom +``` -Use IRSA's TAP to search catalogs and metadata tables associated with Euclid: +## 1. Find the PHZ catalog tables at IRSA +The Euclid catalogs at IRSA can be listed with the `list_catalogs` method; here we filter on the DR1 tables. ```{note} -In the example below, we use ``include_metadata_tables=True``. +In the example below, we use ``include_metadata_tables=True``, as IRSA lists the PHZ and SPE tables among the metadata tables rather than the catalogs. This parameter was introduced in astroquery 0.4.11. For earlier versions, please omit this parameter to receive the combined list of available catalogs and metadata tables. ``` - ```{code-cell} ipython3 -Irsa.list_catalogs(filter='euclid', include_metadata_tables=True) +Irsa.list_catalogs(filter='euclid_dr1', include_metadata_tables=True) ``` ```{code-cell} ipython3 -table_mer = 'euclid_q1_mer_catalogue' -table_phz = 'euclid_q1_phz_photo_z' -table_1dspectra = 'euclid.objectid_spectrafile_association_q1' +table_mer = 'euclid_dr1_mer_catalogue_deep_survey' +table_phz = 'euclid_dr1_phz_photo_z_deep_survey' ``` -### Learn some information about the photo-z catalog: +### Learn some information about the photo-z catalog -- How many columns are there? -- List the column names +Let's look at the number of columns and their descriptions: ```{code-cell} ipython3 columns_info = Irsa.list_columns(catalog=table_phz) -print(len(columns_info)) +print(f'{len(columns_info)} columns') ``` ```{tip} -The PHZ catalog contains 67 columns, below are a few highlights: +A few highlights of the PHZ catalog columns: -- object_id -- flux_vis_unif, flux_y_unif, flux_j_unif, flux_h_unif -- median redshift (phz_median) -- phz_classification -- phz_90_int1, phz_90_int2 (The phz PDF interval containing 90% of the probability, upper and lower values) +- `object_id`, the MER object ID used to join the PHZ catalog with the MER catalog; +- `phz_median`, the median of the photometric redshift probability distribution (PDF); +- `phz_90_int_1` and `phz_90_int_2`, the lower and upper values of the PDF interval containing 90% of the probability; +- `phz_mode_1` and `phz_mode_2`, the two highest modes of the PDF; +- `phz_classification`: 1 for a star, 2 for a galaxy, 4 for a QSO, 8 for a globular cluster (a sum, for example 3, when the source is accepted as several classes); +- `phz_flags`, the PHZ processing flags, 0 when all went well; +- `flux_vis_unif`, `flux_y_unif`, `flux_j_unif`, `flux_h_unif` and the ground-based bands: the "unified" fluxes, recomputed from the MER fluxes after correction for Galactic extinction and filter shifts, which are the fluxes used for the photometric redshifts. ``` ```{code-cell} ipython3 @@ -184,106 +201,150 @@ The PHZ catalog contains 67 columns, below are a few highlights: columns_info ``` -```{note} -The phz_catalog on IRSA has more columns than it does on the ESA archive. -This is because the ESA catalog stores some information in one column (for example, phz_90_int is stored as [lower, upper], rather than in two separate columns). +## 2. Select galaxies between z = 1.4 and 1.6 with good photometric redshifts -The fluxes are different from the fluxes derived in the MER catalog. -The _unif fluxes are: "Unified flux recomputed after correction from galactic extinction and filter shifts". -``` - -+++ - -### Find some galaxies between 1.4 and 1.6 at a selected RA and Dec - -We specify the following conditions on our search: -- We select just the galaxies where the flux is greater than zero, to ensure the appear in all four of the Euclid MER images. -- Select only objects in a circle (search radius selected below) around our selected RA and Dec -- `phz_classification = 2` means we select only galaxies -- Using the `phz_90_int1` and `phz_90_int2`, we select just the galaxies where the error on the photometric redshift is less than 20% -- Select just the galaxies between a median redshift of 1.4 and 1.6 -- We search just a 5 arcminute box around an RA and Dec - -+++ - -Search based on ``tileID``: +We join the PHZ catalog with the MER catalog, which has the coordinates, and select galaxies in a 5 arcmin box around a position in the Euclid Deep Field North, with the following conditions: +- the unified fluxes in the four Euclid bands are greater than zero, so the galaxies are detected in all four Euclid MER mosaics; +- `phz_classification = 2`, so the source is accepted as a galaxy only; +- `phz_flags = 0`, so the PHZ processing had no problems; +- the 90% interval of the photometric redshift PDF is narrower than 20% of (1 + z), so the photometric redshift is well constrained; +- the median photometric redshift is between 1.4 and 1.6. ```{code-cell} ipython3 ######################## User defined section ############################ -# Set the image cutout size -im_cutout= 5 * u.arcmin - -# Set the center of the cutout +# Set the center and size of the region to select galaxies from (and of the image cutout below) ra_cutout = 267.8 -dec_cutout = 66 +dec_cutout = 66 +im_cutout = 5 * u.arcmin + +# Redshift range +z_min, z_max = 1.4, 1.6 +########################################################################## coords_cutout = SkyCoord(ra_cutout, dec_cutout, unit='deg', frame='icrs') size_cutout = im_cutout.to(u.deg).value ``` ```{code-cell} ipython3 -adql = ("SELECT DISTINCT mer.object_id, mer.ra, mer.dec, " - "phz.flux_vis_unif, phz.flux_y_unif, phz.flux_j_unif, phz.flux_h_unif, " - "phz.phz_classification, phz.phz_median, phz.phz_90_int1, phz.phz_90_int2 " - f"FROM {table_mer} AS mer " - f"JOIN {table_phz} as phz " - "ON mer.object_id = phz.object_id " - "WHERE 1 = CONTAINS(POINT('ICRS', mer.ra, mer.dec), " - f"BOX('ICRS', {ra_cutout}, {dec_cutout}, {size_cutout}, {size_cutout})) " - "AND phz.flux_vis_unif> 0 " - "AND phz.flux_y_unif > 0 " - "AND phz.flux_j_unif > 0 " - "AND phz.flux_h_unif > 0 " - "AND phz.phz_classification = 2 " - "AND ((phz.phz_90_int2 - phz.phz_90_int1) / (1 + phz.phz_median)) < 0.20 " - "AND phz.phz_median BETWEEN 1.4 AND 1.6") - - -# Use TAP with this ADQL string -result_galaxies = Irsa.query_tap(adql).to_table() +adql = f""" + SELECT mer.object_id, mer.ra, mer.dec, + phz.flux_vis_unif, phz.flux_y_unif, phz.flux_j_unif, phz.flux_h_unif, + phz.phz_classification, phz.phz_median, phz.phz_90_int_1, phz.phz_90_int_2 + FROM {table_mer} AS mer + JOIN {table_phz} AS phz ON mer.object_id = phz.object_id + WHERE CONTAINS(POINT('ICRS', mer.ra, mer.dec), + BOX('ICRS', {ra_cutout}, {dec_cutout}, {size_cutout}, {size_cutout})) = 1 + AND phz.flux_vis_unif > 0 + AND phz.flux_y_unif > 0 + AND phz.flux_j_unif > 0 + AND phz.flux_h_unif > 0 + AND phz.phz_classification = 2 + AND phz.phz_flags = 0 + AND (phz.phz_90_int_2 - phz.phz_90_int_1) / (1 + phz.phz_median) < 0.20 + AND phz.phz_median BETWEEN {z_min} AND {z_max} +""" + +result_galaxies = Irsa.query_tap(adql, async_job=True).to_table() +print(f'Found {len(result_galaxies)} galaxies') result_galaxies[:5] ``` -```{warning} -Note that we use `to_table` above rather than `to_qtable`. While astropy's `QTable` is more powerful than its `Table`, as it e.g. handles the column units properly, we cannot use it here due to a known bug; it mishandles the large integer numbers in the `object_id` column and recast them as float during which process some precision is being lost. +## 3. Compare the photometric and spectroscopic redshifts + +The Euclid Deep Fields are also covered by NISP slitless spectroscopy, so many of these galaxies have a spectroscopic redshift in the DR1 SPE catalog (see the [SPE catalogs notebook](5_Euclid_intro_SPE_catalog.md)). +We query the best redshift estimate of each galaxy (`spe_rank = 0`) with its probability `spe_z_prob` and reliability `spe_z_rel`, and join the result to our table on `object_id`, keeping the galaxies without a spectroscopic redshift (`join_type='left'`): + +```{code-cell} ipython3 +adql_spe = f""" + SELECT object_id, spe_z, spe_z_prob, spe_z_rel + FROM euclid_dr1_spectro_z_spe_galaxy_candidates_deep + WHERE spe_rank = 0 + AND object_id IN ({', '.join(str(obj_id) for obj_id in result_galaxies['object_id'])}) +""" +result_spe = Irsa.query_tap(adql_spe, async_job=True).to_table() -Once the bug is fixed, we plan to update the code in this notebook and simplify some of the approaches below. +galaxies = join(result_galaxies, result_spe, keys='object_id', join_type='left') +has_spec_z = ~galaxies['spe_z'].mask + +print(f'{has_spec_z.sum()} of the {len(galaxies)} galaxies have a spectroscopic redshift') ``` -+++ +Not all of these spectroscopic redshifts can be trusted: most of the galaxies are faint, and when a single emission line is detected, it can be attributed to the wrong line, which gives a completely wrong redshift. +As in the [SPE catalogs notebook](5_Euclid_intro_SPE_catalog.md), we consider a spectroscopic redshift reliable when both its probability and its reliability are high (`spe_z_prob > 0.99` and `spe_z_rel > 0.05`): -## 3. Read in a cutout of the MER image from IRSA directly +```{code-cell} ipython3 +reliable = has_spec_z & (galaxies['spe_z_prob'].filled(0) > 0.99) & (galaxies['spe_z_rel'].filled(0) > 0.05) -+++ +print(f'{reliable.sum()} of the {has_spec_z.sum()} spectroscopic redshifts are reliable') +``` -Due to the large field of view of the MER mosaic, let's cut out a smaller section (5'x5') of the MER mosaic to inspect the image. +Let's compare the two redshifts: ```{code-cell} ipython3 -# Use fsspec to interact with the fits file without downloading the full file -hdu = fits.open(filename, use_fsspec=True) +fig, ax = plt.subplots(figsize=(6, 6)) +ax.scatter(galaxies['spe_z'][has_spec_z], galaxies['phz_median'][has_spec_z], + facecolors='none', edgecolors='grey', label='All spectroscopic redshifts') +ax.scatter(galaxies['spe_z'][reliable], galaxies['phz_median'][reliable], + color='tab:blue', label='Reliable spectroscopic redshifts') +ax.plot([0, 5], [0, 5], color='grey', linestyle='--', linewidth=0.8) +ax.axhspan(z_min, z_max, color='tab:red', alpha=0.1) +ax.set_xlabel('Spectroscopic redshift (spe_z)') +ax.set_ylabel('Photometric redshift (phz_median)') +_ = ax.legend(loc='upper left') +``` -# Store the header -header = hdu[0].header +The galaxies with a reliable spectroscopic redshift lie close to the one-to-one line, within the width of their photometric redshift intervals, while the unreliable ones are scattered over the whole redshift range. -# Read in the cutout of the image that you want -cutout_image = Cutout2D(hdu[0].section, position=coords_cutout, size=im_cutout, wcs=WCS(header)) +## 4. Read in a cutout of the VIS mosaic and overplot the galaxies + +We first search for the DR1 MER mosaics that cover our region. +The mosaics come in two collections, wide and deep tiles (see the [MER mosaics notebook](1_Euclid_intro_MER_images.md)); the Deep Survey catalogs are measured on the deep tiles, so we search the deep collection. + +```{code-cell} ipython3 +image_table = Irsa.query_sia(pos=(coords_cutout, 10 * u.arcsec), collection='euclid_dr1_deep_DpdMerBksMosaic', facility='Euclid') ``` +```{note} +The result lists all of the MER mosaic images available at this position: the Euclid VIS, Y, J, H mosaics, and the mosaics of the ground-based surveys that have been put on the same pixel scale. We use the ``facility`` argument above to query for Euclid images only. +``` + +Various image types are returned, so we keep the science images only: + ```{code-cell} ipython3 -cutout_image.data.shape +science_images = image_table[image_table['dataproduct_subtype'] == 'science'] +science_images['obs_id', 'energy_bandpassname', 'access_url'] ``` +We keep the VIS mosaic, which we use for the cutouts and in Firefly below; the first part of its `obs_id` is its tile ID: + ```{code-cell} ipython3 -norm = ImageNormalize(cutout_image.data, interval=PercentileInterval(99.9), stretch=AsinhStretch()) -_ = plt.imshow(cutout_image.data, cmap='gray', origin='lower', norm=norm) +vis_mosaic = science_images[science_images['energy_bandpassname'] == 'VIS'][0] +filename = vis_mosaic['access_url'] +tileID = vis_mosaic['obs_id'][:9] + +print(f'The MER tile ID of this region is {tileID}') ``` -## 4. Overplot the catalog on the MER mosaic image +The MER mosaics are large, so we read only a 5 x 5 arcmin cutout of the VIS mosaic directly from the cloud, without downloading the full file (see the [MER mosaics notebook](1_Euclid_intro_MER_images.md) for more details on the cloud access). +The `cloud_access` column of the SIA result gives the location of the mosaic in the cloud: -+++ +```{code-cell} ipython3 +cloud_info = json.loads(vis_mosaic['cloud_access']) +s3_path = f"s3://{cloud_info['aws']['bucket_name']}/{cloud_info['aws']['key']}" +s3_path +``` + +```{code-cell} ipython3 +# Open the file without downloading it, and read only the cutout +hdu = fits.open(s3_path, fsspec_kwargs={'anon': True}) +header = hdu[0].header + +cutout_image = Cutout2D(hdu[0].section, position=coords_cutout, size=im_cutout, wcs=WCS(header)) +cutout_image.data.shape +``` ```{tip} -We can rely on astropy's WCSAxes framework for making plots of Astronomical data in Matplotlib. Please note the usage of `projection` and `transform` arguments in the code example below. +We rely on astropy's WCSAxes framework for making plots of astronomical data in Matplotlib. Please note the usage of the `projection` and `transform` arguments in the code example below. For more info, please visit the [WCSAxes documentation](https://docs.astropy.org/en/stable/visualization/wcsaxes/index.html). ``` @@ -293,117 +354,117 @@ ax = plt.subplot(projection=cutout_image.wcs) ax.imshow(cutout_image.data, cmap='gray', origin='lower', norm=ImageNormalize(cutout_image.data, interval=PercentileInterval(99.9), stretch=LogStretch())) -plt.scatter(result_galaxies['ra'], result_galaxies['dec'], s=36, facecolors='none', edgecolors='red', - transform=ax.get_transform('world')) +ax.scatter(result_galaxies['ra'], result_galaxies['dec'], s=36, facecolors='none', edgecolors='red', + transform=ax.get_transform('world')) -_ = plt.title('Galaxies between z = 1.4 and 1.6') +_ = ax.set_title(f'Galaxies with photometric redshifts between {z_min} and {z_max}') ``` -## 5. Pull spectra for one of the brightest sources by object ID +## 5. Look at the spectrum and image of one of the brightest galaxies -```{code-cell} ipython3 -result_galaxies.sort(keys='flux_h_unif', reverse=True) -``` +Let's sort the galaxies with a reliable spectroscopic redshift by their H-band flux, and look at the five brightest ones: ```{code-cell} ipython3 -result_galaxies[:3] +brightest = galaxies[reliable] +brightest.sort(keys='flux_h_unif', reverse=True) +brightest['object_id', 'flux_h_unif', 'phz_median', 'phz_90_int_1', 'phz_90_int_2', 'spe_z', 'spe_z_prob', 'spe_z_rel'][:5] ``` -Let's pick one of these galaxies. Note that the table has been sorted above, we can use the same index here and below to access the data for this particular galaxy. +We pick the brightest one: ```{code-cell} ipython3 -index = 2 +galaxy = brightest[0] + +obj_id = galaxy['object_id'] +photo_z = galaxy['phz_median'] +spec_z = galaxy['spe_z'] +coords_galaxy = SkyCoord(galaxy['ra'], galaxy['dec'], unit='deg') -obj_id = result_galaxies['object_id'][index] -redshift = result_galaxies['phz_median'][index] +print(f'Object {obj_id}: photometric redshift {photo_z:.2f}, spectroscopic redshift {spec_z:.3f}') ``` -We will use TAP and an ASQL query to find the spectral data for this particular galaxy. +### Retrieve the spectra + +We search IRSA's Simple Spectral Access (SSA) service for the DR1 SIR 1D spectra at the position of the galaxy. +A galaxy in the Euclid Deep Fields can have several spectra, from the red and blue grisms and from the deep and wide tiles; as in the [SIR 1D spectra notebook](3_Euclid_intro_1D_spectra.md), we keep the spectrum with the most usable flux bins for each kind of grism. ```{code-cell} ipython3 -adql_object = f"SELECT * FROM {table_1dspectra} WHERE objectid = {obj_id}" +ssa_result = Irsa.query_ssa(pos=coords_galaxy, radius=1 * u.arcsec, collection='euclid_dr1_DpdSirCombinedSpectra') +spectra = [QTable.read(url) for url in ssa_result['access_url']] -# Pull the data on this particular galaxy -result_spectra = Irsa.query_tap(adql_object).to_table() -result_spectra +print(f'Found {len(spectra)} spectra') ``` -Pull out the file name from the ``result_spectra`` table: - ```{code-cell} ipython3 -spectrum_path = f"https://irsa.ipac.caltech.edu/{result_spectra['path'][0]}" -spectrum_path -``` +def bad_flux_bins(spectrum): + # Flux bins with an odd MASK value or a MASK value of 64 or more should not be used + mask = spectrum['MASK'].value + return (mask % 2 == 1) | (mask >= 64) -```{code-cell} ipython3 -spectrum = QTable.read(spectrum_path) -``` -### Now the data are read in, plot the spectrum +def grism_name(spectrum): + # Blue grism spectra start below 1.1 micron + return 'Blue grism' if spectrum['WAVELENGTH'].min() < 1.1 * u.micron else 'Red grism' -```{tip} -As we use astropy.visualization’s quantity_support, matplotlib automatically picks up the axis units from the quantities we plot. -``` -```{code-cell} ipython3 -quantity_support() +best = {} +for spectrum in sorted(spectra, key=lambda spectrum: np.sum(~bad_flux_bins(spectrum)), reverse=True): + best.setdefault(grism_name(spectrum), spectrum) +list(best) ``` -```{code-cell} ipython3 -plt.plot(spectrum['WAVELENGTH'].to(u.micron), spectrum['SIGNAL']) +### Plot the spectra -plt.xlim(1.25, 1.85) -plt.ylim(-0.5, 0.5) -_ = plt.title(f"Object {obj_id} with phz_median={redshift}") -``` - -Let's cut out a very small patch of the MER image to see what this galaxy looks like. Remember that we sorted the table above, so can reuse the same index to pick up the coordinates for the galaxy. Otherwise we could filter on the object ID. +We plot the usable flux bins of the best spectrum of each grism, and mark the strongest emission lines at the spectroscopic redshift, to see how well the photometric redshift agrees with it: ```{code-cell} ipython3 -result_galaxies[index] -``` +emission_lines = {'Hβ': 4862.7 * u.AA, '[OIII]': 5008.2 * u.AA, 'Hα': 6564.6 * u.AA} +grism_colors = {'Red grism': 'tab:red', 'Blue grism': 'tab:blue'} -```{code-cell} ipython3 -# Set the image cutout size for the selected galaxy -size_galaxy_cutout = 2.0 * u.arcsec -``` +fig, ax = plt.subplots(figsize=(12, 4.5)) +for name, spectrum in best.items(): + bad = bad_flux_bins(spectrum) + ax.plot(spectrum['WAVELENGTH'].to(u.micron), np.ma.masked_where(bad, spectrum['SIGNAL']), + color=grism_colors[name], alpha=0.8, label=name) -Use the `ra` and `dec` columns for the galaxy to create a `SkyCoord`. +for line, rest_wavelength in emission_lines.items(): + observed = (rest_wavelength * (1 + spec_z)).to(u.micron) + ax.axvline(observed, color='grey', linestyle='--', linewidth=0.8) + ax.text(observed.value, 0.98, f' {line}', transform=ax.get_xaxis_transform(), rotation=90, va='top', ha='right') -```{code-cell} ipython3 -coords_galaxy = SkyCoord(result_galaxies['ra'][index], result_galaxies['dec'][index], unit='deg') +ax.set_title(f'Object {obj_id}: phz_median = {photo_z:.2f}, spe_z = {spec_z:.3f}') +_ = ax.legend(loc='upper left') ``` -```{code-cell} ipython3 -coords_galaxy -``` +The Hα and [OIII] lines fall right at the spectroscopic redshift, which agrees well with the photometric redshift. +Not every feature is a line of this galaxy, though: slitless spectra are prone to contamination by neighboring sources, which is the likely origin of the narrow spike near 1.3 micron in the blue grism spectrum. -We haven't closed the image file above, so use `Cutout2D` again to cut out a section around the galaxy. +### Look at the galaxy on the VIS mosaic + +We haven't closed the image file above, so we use `Cutout2D` again to cut out a small region around the galaxy: ```{code-cell} ipython3 -cutout_galaxy = Cutout2D(hdu[0].section, position=coords_galaxy, size=size_galaxy_cutout, wcs=WCS(header)) -``` +size_galaxy_cutout = 3 * u.arcsec -Plot to show the cutout on the galaxy +cutout_galaxy = Cutout2D(hdu[0].section, position=coords_galaxy, size=size_galaxy_cutout, wcs=WCS(header)) -```{code-cell} ipython3 ax = plt.subplot(projection=cutout_galaxy.wcs) - ax.imshow(cutout_galaxy.data, cmap='gray', origin='lower', norm=ImageNormalize(cutout_galaxy.data, interval=PercentileInterval(99.9), stretch=AsinhStretch())) +_ = ax.set_title(f'Object {obj_id}') ``` ## 6. Load the image in Firefly for interactive exploration -First we intialize the Firefly client as explained [here](https://caltech-ipac.github.io/firefly_client/usage/initializing-vanilla.html) so that we can send our data to Firefly for visualization. +First we initialize the Firefly client as explained [here](https://caltech-ipac.github.io/firefly_client/usage/initializing-vanilla.html) so that we can send our data to Firefly for visualization. ```{code-cell} ipython3 fc = FireflyClient.make_client('https://irsa.ipac.caltech.edu/irsaviewer') ``` -### Vizualize the FITS image with Firefly +### Visualize the FITS image with Firefly -After initializing the client, we can directly use the FITS image file URL to visualise it in Firefy. +After initializing the client, we can directly use the FITS image file URL to visualize it in Firefly. Note this can take a while to upload the full MER image. @@ -419,13 +480,21 @@ You can click on a table row to inspect the particular galaxy in the image, or c ```{code-cell} ipython3 tbl_stream = io.BytesIO() -result_galaxies.write(tbl_stream, format='votable') +galaxies.write(tbl_stream, format='votable') fc.show_table(tbl_stream) ``` +## References + + + +- Euclid Collaboration: Tucci, M., et al. 2026, *Euclid Data Release 1 (DR1): The PHZ processing*, A&A, in prep. + ++++ + ## About this Notebook -**Updated**: 2026-04-29 +**Updated**: 2026-11-12 **Contact:** [the IRSA Helpdesk](https://irsa.ipac.caltech.edu/docs/help_desk.html) with questions or reporting problems. From 6283f67c48ba5dc8d9f04cfd32f92fea027c9559 Mon Sep 17 00:00:00 2001 From: xoubish Date: Fri, 6 Nov 2026 11:56:36 -0800 Subject: [PATCH 2/7] Select galaxies at z 0.85-1.85 and get their spectra from the association table Widen the photometric redshift range to the one where Halpha falls in the red grism, plot only the reliable spectroscopic redshifts against the photometric ones, and retrieve the spectra of the example galaxy through euclid.objectid_spectrafile_association_dr1 and the spectrumDM service, since the DR1 spectra will not be in SSA/CAOM at release. --- .../euclid/4_Euclid_intro_PHZ_catalog.md | 81 ++++++++----------- 1 file changed, 34 insertions(+), 47 deletions(-) diff --git a/tutorials/euclid/4_Euclid_intro_PHZ_catalog.md b/tutorials/euclid/4_Euclid_intro_PHZ_catalog.md index 2de96b70..69d57740 100644 --- a/tutorials/euclid/4_Euclid_intro_PHZ_catalog.md +++ b/tutorials/euclid/4_Euclid_intro_PHZ_catalog.md @@ -37,7 +37,7 @@ By the end of this tutorial, you will: - know how to join the PHZ and MER catalogs with ADQL to select galaxies with good photometric redshifts in a redshift range; - know how to compare the photometric redshifts with the spectroscopic redshifts of the DR1 SPE catalog; - know how to make a cutout of a MER mosaic and overplot the selected galaxies on it; -- know how to retrieve and plot the spectrum of one of the galaxies, and compare its photometric and spectroscopic redshifts; +- know how to find and plot the spectra of one of the galaxies from its object ID, and compare its photometric and spectroscopic redshifts; - know how to load the image and catalog into Firefly to inspect individual sources interactively. +++ @@ -129,32 +129,6 @@ conf.remote_timeout = 120 # DR1 is not public yet, so we query IRSA's development server. Irsa.tap_url = 'https://irsadev.ipac.caltech.edu/TAP' Irsa.sia_url = 'https://irsadev.ipac.caltech.edu/SIA' -Irsa.ssa_url = 'https://irsadev.ipac.caltech.edu/SSA' -``` - - - -```{code-cell} ipython3 -# TEMPORARY: the SSA service does not return DR1 spectra yet, so we emulate -# Irsa.query_ssa with a TAP query on IRSA's CAOM tables, which already have them. -from astropy.table import Table - - -def query_ssa_from_caom(*, pos, radius, collection): - query = f""" - SELECT a.uri - FROM euclid.plane_euclid_dr1 AS p - JOIN euclid.observation_euclid_dr1 AS o ON o.obsid = p.obsid - JOIN euclid.artifact_euclid_dr1 AS a ON a.planeid = p.planeid - WHERE o.collection = '{collection}' - AND CONTAINS(p.pt, CIRCLE('ICRS', {pos.ra.deg}, {pos.dec.deg}, {radius.to(u.deg).value})) = 1 - """ - uris = Irsa.query_tap(query, async_job=True).to_table()['uri'] - return Table({'s_ra': [pos.ra.deg] * len(uris), 's_dec': [pos.dec.deg] * len(uris), - 'access_url': [f'https://irsadev.ipac.caltech.edu/{uri}' for uri in uris]}) - - -Irsa.query_ssa = query_ssa_from_caom ``` ## 1. Find the PHZ catalog tables at IRSA @@ -201,14 +175,14 @@ A few highlights of the PHZ catalog columns: columns_info ``` -## 2. Select galaxies between z = 1.4 and 1.6 with good photometric redshifts +## 2. Select galaxies between z = 0.85 and 1.85 with good photometric redshifts We join the PHZ catalog with the MER catalog, which has the coordinates, and select galaxies in a 5 arcmin box around a position in the Euclid Deep Field North, with the following conditions: - the unified fluxes in the four Euclid bands are greater than zero, so the galaxies are detected in all four Euclid MER mosaics; - `phz_classification = 2`, so the source is accepted as a galaxy only; - `phz_flags = 0`, so the PHZ processing had no problems; - the 90% interval of the photometric redshift PDF is narrower than 20% of (1 + z), so the photometric redshift is well constrained; -- the median photometric redshift is between 1.4 and 1.6. +- the median photometric redshift is between 0.85 and 1.85, the range in which the Hα line falls in the NISP red grism spectra (1.2 to 1.9 micron), so that the galaxies can also have spectroscopic redshifts (see below). ```{code-cell} ipython3 ######################## User defined section ############################ @@ -218,7 +192,7 @@ dec_cutout = 66 im_cutout = 5 * u.arcmin # Redshift range -z_min, z_max = 1.4, 1.6 +z_min, z_max = 0.85, 1.85 ########################################################################## coords_cutout = SkyCoord(ra_cutout, dec_cutout, unit='deg', frame='icrs') @@ -278,22 +252,22 @@ reliable = has_spec_z & (galaxies['spe_z_prob'].filled(0) > 0.99) & (galaxies['s print(f'{reliable.sum()} of the {has_spec_z.sum()} spectroscopic redshifts are reliable') ``` -Let's compare the two redshifts: +Let's compare the two redshifts of the galaxies with a reliable spectroscopic redshift: ```{code-cell} ipython3 fig, ax = plt.subplots(figsize=(6, 6)) -ax.scatter(galaxies['spe_z'][has_spec_z], galaxies['phz_median'][has_spec_z], - facecolors='none', edgecolors='grey', label='All spectroscopic redshifts') -ax.scatter(galaxies['spe_z'][reliable], galaxies['phz_median'][reliable], - color='tab:blue', label='Reliable spectroscopic redshifts') -ax.plot([0, 5], [0, 5], color='grey', linestyle='--', linewidth=0.8) -ax.axhspan(z_min, z_max, color='tab:red', alpha=0.1) +ax.scatter(galaxies['spe_z'][reliable], galaxies['phz_median'][reliable], color='tab:blue') +ax.plot([0.5, 2.5], [0.5, 2.5], color='grey', linestyle='--', linewidth=0.8) +ax.axhspan(z_min, z_max, color='tab:red', alpha=0.1, label='Selected photometric redshift range') +ax.set_xlim(0.5, 2.5) +ax.set_ylim(0.5, 2.5) ax.set_xlabel('Spectroscopic redshift (spe_z)') ax.set_ylabel('Photometric redshift (phz_median)') _ = ax.legend(loc='upper left') ``` -The galaxies with a reliable spectroscopic redshift lie close to the one-to-one line, within the width of their photometric redshift intervals, while the unreliable ones are scattered over the whole redshift range. +Most of the galaxies lie close to the one-to-one line, within the width of their photometric redshift intervals. +Try plotting the unreliable spectroscopic redshifts as well: they are scattered over the whole redshift range, because a single detected line attributed to the wrong transition gives a completely wrong redshift. ## 4. Read in a cutout of the VIS mosaic and overplot the galaxies @@ -370,10 +344,23 @@ brightest.sort(keys='flux_h_unif', reverse=True) brightest['object_id', 'flux_h_unif', 'phz_median', 'phz_90_int_1', 'phz_90_int_2', 'spe_z', 'spe_z_prob', 'spe_z_rel'][:5] ``` -We pick the brightest one: +### Retrieve the spectra + +The DR1 SIR 1D spectra are stored in large FITS files with one extension (HDU) per spectrum. +The table `euclid.objectid_spectrafile_association_dr1` at IRSA lists, for each object ID, the file and the HDU of its spectra; we query it for the brightest of these galaxies. +Not every galaxy with a spectroscopic redshift has a spectrum listed in this table yet, so if none is found, we move on to the next brightest galaxy. ```{code-cell} ipython3 -galaxy = brightest[0] +for galaxy in brightest: + adql_spectra = f""" + SELECT objectid, tileid, path, hdu + FROM euclid.objectid_spectrafile_association_dr1 + WHERE objectid = {galaxy['object_id']} + """ + result_spectra = Irsa.query_tap(adql_spectra).to_table() + if len(result_spectra) > 0: + break + print(f"No spectra listed for object {galaxy['object_id']}, trying the next galaxy") obj_id = galaxy['object_id'] photo_z = galaxy['phz_median'] @@ -381,18 +368,18 @@ spec_z = galaxy['spe_z'] coords_galaxy = SkyCoord(galaxy['ra'], galaxy['dec'], unit='deg') print(f'Object {obj_id}: photometric redshift {photo_z:.2f}, spectroscopic redshift {spec_z:.3f}') +result_spectra ``` -### Retrieve the spectra +IRSA's spectrumDM service converts a given HDU of a spectrum file into a VOTable, which we read into an astropy table. +A galaxy in the Euclid Deep Fields can have spectra from both the red and the blue grism; as in the [SIR 1D spectra notebook](3_Euclid_intro_1D_spectra.md), we keep the spectrum with the most usable flux bins for each kind of grism. -We search IRSA's Simple Spectral Access (SSA) service for the DR1 SIR 1D spectra at the position of the galaxy. -A galaxy in the Euclid Deep Fields can have several spectra, from the red and blue grisms and from the deep and wide tiles; as in the [SIR 1D spectra notebook](3_Euclid_intro_1D_spectra.md), we keep the spectrum with the most usable flux bins for each kind of grism. + ```{code-cell} ipython3 -ssa_result = Irsa.query_ssa(pos=coords_galaxy, radius=1 * u.arcsec, collection='euclid_dr1_DpdSirCombinedSpectra') -spectra = [QTable.read(url) for url in ssa_result['access_url']] - -print(f'Found {len(spectra)} spectra') +spectrum_urls = [f"https://irsadev.ipac.caltech.edu/api/spectrumdm/convert/{row['path']}?dataset_id=euclid_combspec&hdu={row['hdu']}" + for row in result_spectra] +spectra = [QTable.read(url) for url in spectrum_urls] ``` ```{code-cell} ipython3 From e521701b4cbcf47c93aba6f053feafa4ff027171 Mon Sep 17 00:00:00 2001 From: xoubish Date: Fri, 6 Nov 2026 12:03:54 -0800 Subject: [PATCH 3/7] Build the spectrum URLs with IRSA's recipe Use the irsacloud spectrumDM endpoint with the euclid_combspec_dr1 dataset, and build the URL in the ADQL query on the association table, as recommended by IRSA. --- tutorials/euclid/4_Euclid_intro_PHZ_catalog.md | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/tutorials/euclid/4_Euclid_intro_PHZ_catalog.md b/tutorials/euclid/4_Euclid_intro_PHZ_catalog.md index 69d57740..1c348786 100644 --- a/tutorials/euclid/4_Euclid_intro_PHZ_catalog.md +++ b/tutorials/euclid/4_Euclid_intro_PHZ_catalog.md @@ -347,13 +347,16 @@ brightest['object_id', 'flux_h_unif', 'phz_median', 'phz_90_int_1', 'phz_90_int_ ### Retrieve the spectra The DR1 SIR 1D spectra are stored in large FITS files with one extension (HDU) per spectrum. -The table `euclid.objectid_spectrafile_association_dr1` at IRSA lists, for each object ID, the file and the HDU of its spectra; we query it for the brightest of these galaxies. -Not every galaxy with a spectroscopic redshift has a spectrum listed in this table yet, so if none is found, we move on to the next brightest galaxy. +The table `euclid.objectid_spectrafile_association_dr1` at IRSA lists, for each object ID, the file (`path`) and the HDU of its spectra. +IRSA's spectrumDM service converts a given HDU of a spectrum file into a VOTable; its URL is built from the file path and HDU, which we do directly in the query. +We query the table for the brightest of these galaxies; not every galaxy with a spectroscopic redshift has a spectrum listed in it yet, so if none is found, we move on to the next brightest galaxy. ```{code-cell} ipython3 for galaxy in brightest: adql_spectra = f""" - SELECT objectid, tileid, path, hdu + SELECT objectid, tileid, hdu, + 'https://irsacloud.ipac.caltech.edu/api/spectrumdm/convert/' || path + || '?dataset_id=euclid_combspec_dr1&hdu=' || hdu AS url FROM euclid.objectid_spectrafile_association_dr1 WHERE objectid = {galaxy['object_id']} """ @@ -371,15 +374,11 @@ print(f'Object {obj_id}: photometric redshift {photo_z:.2f}, spectroscopic redsh result_spectra ``` -IRSA's spectrumDM service converts a given HDU of a spectrum file into a VOTable, which we read into an astropy table. +Each URL returns a VOTable containing the spectrum, which we read into an astropy table. A galaxy in the Euclid Deep Fields can have spectra from both the red and the blue grism; as in the [SIR 1D spectra notebook](3_Euclid_intro_1D_spectra.md), we keep the spectrum with the most usable flux bins for each kind of grism. - - ```{code-cell} ipython3 -spectrum_urls = [f"https://irsadev.ipac.caltech.edu/api/spectrumdm/convert/{row['path']}?dataset_id=euclid_combspec&hdu={row['hdu']}" - for row in result_spectra] -spectra = [QTable.read(url) for url in spectrum_urls] +spectra = [QTable.read(url) for url in result_spectra['url']] ``` ```{code-cell} ipython3 From 663f31298f6d8dace7335e1a5bba8e94f5efa499 Mon Sep 17 00:00:00 2001 From: xoubish Date: Mon, 9 Nov 2026 16:21:32 -0800 Subject: [PATCH 4/7] Address review comments on notebook 4, check against the DR1 PHZ papers --- .../euclid/4_Euclid_intro_PHZ_catalog.md | 255 +++++++++++++----- 1 file changed, 188 insertions(+), 67 deletions(-) diff --git a/tutorials/euclid/4_Euclid_intro_PHZ_catalog.md b/tutorials/euclid/4_Euclid_intro_PHZ_catalog.md index 1c348786..2f2fc474 100644 --- a/tutorials/euclid/4_Euclid_intro_PHZ_catalog.md +++ b/tutorials/euclid/4_Euclid_intro_PHZ_catalog.md @@ -55,10 +55,11 @@ Euclid Data Release 1 (DR1) comprises about 1900 sq deg of imaging, spectroscopy DR1 also includes several photometric redshift and instrument calibration fields, such as COSMOS, CDFS, and AEGIS. It follows Quick Release 1 (Q1), which covered about 63 sq deg with a single pass over each of the three Euclid Deep Fields and the dark cloud LDN 1641; all of the DR1 data, including the areas observed in Q1, have been reprocessed with updated pipelines and calibrations. -Among the data products included in DR1 are the catalogs created by the PHZ processing function, which estimates the photometric redshift and the classification (star, galaxy, QSO) of every source in the MER catalog from its photometry. +Among the data products included in DR1 are the catalogs created by the PHZ processing function, which estimates the photometric redshift of every source in the MER catalog from its photometry, and classifies it as a star or a galaxy. +In DR1, the photometric redshifts are measured with a nearest-neighbour method (NNPZ), which compares the photometry of each source with that of a reference sample of galaxies with precise redshifts in the COSMOS field. This notebook provides an introduction to the main PHZ catalog, which contains the photometric redshift probability distribution, the unified fluxes, and the classification for each source. We follow a typical path through the catalog: we select galaxies with good photometric redshifts in a redshift range, look at them on a MER mosaic, and compare the photometric redshift of one of them with its spectrum and spectroscopic redshift. -The PHZ processing for DR1 is described in detail in Euclid Collaboration: Tucci et al. (2026), see the [References](#references) section below. +The PHZ processing for DR1 is described in detail in Euclid Collaboration: Hartley et al. (2026), and the galaxy physical properties it also provides in Euclid Collaboration: Enia et al. (2026); see the [References](#references) section below. If you have questions about this notebook, please contact the [IRSA helpdesk](https://irsa.ipac.caltech.edu/docs/help_desk.html). +++ @@ -69,7 +70,7 @@ Like the MER catalog (see the [MER catalogs notebook](2_Euclid_intro_MER_catalog * `euclid_dr1_phz_photo_z_wide` -- the main PHZ catalog of the **Euclid Wide Survey**: photometric redshifts, unified fluxes, and classification. * `euclid_dr1_phz_photo_z_deep_survey` -- the same for the **Euclid Deep Survey** (the deep tiles of the Euclid Deep Fields North, South, and Fornax). -* `euclid_dr1_phz_classification_wide` and `euclid_dr1_phz_classification_deep_survey` -- the detailed classification probabilities (star, galaxy, QSO, globular cluster) behind the `phz_classification` column of the main catalogs. +* `euclid_dr1_phz_classification_wide` and `euclid_dr1_phz_classification_deep_survey` -- the star and galaxy probabilities, and the thresholds above which a source is accepted as a star or a galaxy, behind the `phz_classification` column of the main catalogs. The PHZ tables have no coordinates; they are joined to the MER catalog of the same survey on `object_id`. In this notebook we work in the Euclid Deep Field North, so we use the deep survey tables. @@ -102,7 +103,7 @@ import matplotlib.pyplot as plt from astropy.coordinates import SkyCoord from astropy.io import fits from astropy.nddata import Cutout2D -from astropy.table import QTable, join +from astropy.table import QTable from astropy import units as u from astropy.utils.data import conf from astropy.visualization import ImageNormalize, PercentileInterval, AsinhStretch, LogStretch, quantity_support @@ -147,6 +148,7 @@ Irsa.list_catalogs(filter='euclid_dr1', include_metadata_tables=True) ```{code-cell} ipython3 table_mer = 'euclid_dr1_mer_catalogue_deep_survey' table_phz = 'euclid_dr1_phz_photo_z_deep_survey' +table_spe = 'euclid_dr1_spectro_z_spe_galaxy_candidates_deep' ``` ### Learn some information about the photo-z catalog @@ -164,10 +166,10 @@ A few highlights of the PHZ catalog columns: - `object_id`, the MER object ID used to join the PHZ catalog with the MER catalog; - `phz_median`, the median of the photometric redshift probability distribution (PDF); - `phz_90_int_1` and `phz_90_int_2`, the lower and upper values of the PDF interval containing 90% of the probability; -- `phz_mode_1` and `phz_mode_2`, the two highest modes of the PDF; -- `phz_classification`: 1 for a star, 2 for a galaxy, 4 for a QSO, 8 for a globular cluster (a sum, for example 3, when the source is accepted as several classes); -- `phz_flags`, the PHZ processing flags, 0 when all went well; -- `flux_vis_unif`, `flux_y_unif`, `flux_j_unif`, `flux_h_unif` and the ground-based bands: the "unified" fluxes, recomputed from the MER fluxes after correction for Galactic extinction and filter shifts, which are the fluxes used for the photometric redshifts. +- `phz_mode_1`, the mean redshift of the most probable peak of the PDF, and `phz_mode_2`, the mean redshift of the PDF of the closest neighbour in the reference sample (despite their names, neither is exactly the mode of the PDF); +- `phz_classification`: 1 for a star, 2 for a galaxy, and 3 when the source is accepted as both. The data model also defines 4 for a QSO and 8 for a globular cluster, but DR1 only classifies sources as stars or galaxies: QSO candidates are the point-like sources that are not accepted as stars; +- `phz_flags`, the PHZ processing flags, 0 when all went well (for example, 1 when no close neighbour was found in the reference sample); +- `flux_vis_unif`, `flux_y_unif`, `flux_j_unif`, `flux_h_unif` and the ground-based bands: the "unified" fluxes, the MER fluxes corrected for Galactic extinction and put on a common filter system (the DECam filters for the ground-based bands), so that they can be compared across the whole survey. They are computed for all sources, but should not be used for stars. ``` ```{code-cell} ipython3 @@ -178,12 +180,15 @@ columns_info ## 2. Select galaxies between z = 0.85 and 1.85 with good photometric redshifts We join the PHZ catalog with the MER catalog, which has the coordinates, and select galaxies in a 5 arcmin box around a position in the Euclid Deep Field North, with the following conditions: -- the unified fluxes in the four Euclid bands are greater than zero, so the galaxies are detected in all four Euclid MER mosaics; +- the unified fluxes in the four Euclid bands are greater than zero; - `phz_classification = 2`, so the source is accepted as a galaxy only; - `phz_flags = 0`, so the PHZ processing had no problems; - the 90% interval of the photometric redshift PDF is narrower than 20% of (1 + z), so the photometric redshift is well constrained; - the median photometric redshift is between 0.85 and 1.85, the range in which the Hα line falls in the NISP red grism spectra (1.2 to 1.9 micron), so that the galaxies can also have spectroscopic redshifts (see below). +In the same query, we also get the best spectroscopic redshift of each galaxy (`spe_rank = 0`), with its probability `spe_z_prob` and reliability `spe_z_rel`, from the DR1 SPE catalog (see the [SPE catalogs notebook](5_Euclid_intro_SPE_catalog.md)). +We use a left outer join, which keeps the galaxies without a spectroscopic redshift; the condition `spe_rank = 0` goes in the `ON` clause of the join rather than in the `WHERE` clause, which would drop them. + ```{code-cell} ipython3 ######################## User defined section ############################ # Set the center and size of the region to select galaxies from (and of the image cutout below) @@ -203,9 +208,11 @@ size_cutout = im_cutout.to(u.deg).value adql = f""" SELECT mer.object_id, mer.ra, mer.dec, phz.flux_vis_unif, phz.flux_y_unif, phz.flux_j_unif, phz.flux_h_unif, - phz.phz_classification, phz.phz_median, phz.phz_90_int_1, phz.phz_90_int_2 + phz.phz_classification, phz.phz_median, phz.phz_90_int_1, phz.phz_90_int_2, + spe.spe_z, spe.spe_z_prob, spe.spe_z_rel FROM {table_mer} AS mer JOIN {table_phz} AS phz ON mer.object_id = phz.object_id + LEFT OUTER JOIN {table_spe} AS spe ON spe.object_id = mer.object_id AND spe.spe_rank = 0 WHERE CONTAINS(POINT('ICRS', mer.ra, mer.dec), BOX('ICRS', {ra_cutout}, {dec_cutout}, {size_cutout}, {size_cutout})) = 1 AND phz.flux_vis_unif > 0 @@ -218,27 +225,18 @@ adql = f""" AND phz.phz_median BETWEEN {z_min} AND {z_max} """ -result_galaxies = Irsa.query_tap(adql, async_job=True).to_table() -print(f'Found {len(result_galaxies)} galaxies') -result_galaxies[:5] +galaxies = Irsa.query_tap(adql, async_job=True).to_table() +print(f'Found {len(galaxies)} galaxies') +galaxies[:5] ``` ## 3. Compare the photometric and spectroscopic redshifts -The Euclid Deep Fields are also covered by NISP slitless spectroscopy, so many of these galaxies have a spectroscopic redshift in the DR1 SPE catalog (see the [SPE catalogs notebook](5_Euclid_intro_SPE_catalog.md)). -We query the best redshift estimate of each galaxy (`spe_rank = 0`) with its probability `spe_z_prob` and reliability `spe_z_rel`, and join the result to our table on `object_id`, keeping the galaxies without a spectroscopic redshift (`join_type='left'`): +The Euclid Deep Fields are also covered by NISP slitless spectroscopy, so many of these galaxies have a spectroscopic redshift. +The galaxies without one have a masked `spe_z`: ```{code-cell} ipython3 -adql_spe = f""" - SELECT object_id, spe_z, spe_z_prob, spe_z_rel - FROM euclid_dr1_spectro_z_spe_galaxy_candidates_deep - WHERE spe_rank = 0 - AND object_id IN ({', '.join(str(obj_id) for obj_id in result_galaxies['object_id'])}) -""" -result_spe = Irsa.query_tap(adql_spe, async_job=True).to_table() - -galaxies = join(result_galaxies, result_spe, keys='object_id', join_type='left') -has_spec_z = ~galaxies['spe_z'].mask +has_spec_z = ~np.ma.getmaskarray(galaxies['spe_z']) print(f'{has_spec_z.sum()} of the {len(galaxies)} galaxies have a spectroscopic redshift') ``` @@ -266,7 +264,24 @@ ax.set_ylabel('Photometric redshift (phz_median)') _ = ax.legend(loc='upper left') ``` -Most of the galaxies lie close to the one-to-one line, within the width of their photometric redshift intervals. +Most of the galaxies lie close to the one-to-one line. +Let's quantify this with the usual metrics: the normalized median absolute deviation (NMAD) of (zphot - zspec) / (1 + zspec), and the fraction of outliers, for which this difference is larger than 0.15. +We also count how many spectroscopic redshifts fall within the 90% interval of the photometric redshift: + +```{code-cell} ipython3 +spec_z_reliable = np.asarray(galaxies['spe_z'][reliable], dtype=float) +dz = (np.asarray(galaxies['phz_median'][reliable]) - spec_z_reliable) / (1 + spec_z_reliable) +nmad = 1.48 * np.median(np.abs(dz - np.median(dz))) +inside_90 = ((spec_z_reliable >= galaxies['phz_90_int_1'][reliable]) + & (spec_z_reliable <= galaxies['phz_90_int_2'][reliable])) + +print(f'NMAD = {nmad:.3f}, outlier fraction = {np.mean(np.abs(dz) > 0.15):.1%}') +print(f'{inside_90.mean():.0%} of the spectroscopic redshifts are within the 90% interval of the photometric redshift') +``` + +The scatter and outlier fraction are similar to those found by Euclid Collaboration: Hartley et al. (2026) for the whole Euclid Deep Field North. +However, far fewer than 90% of the spectroscopic redshifts fall within the 90% interval: the photometric redshift PDFs of the deep fields are too narrow in DR1, because no systematic uncertainty was added to the fluxes in their processing (Euclid Collaboration: Hartley et al. 2026). +Keep this in mind when using the PDF intervals, as we did in section 2 to select galaxies with a well-constrained photometric redshift. Try plotting the unreliable spectroscopic redshifts as well: they are scattered over the whole redshift range, because a single detected line attributed to the wrong transition gives a completely wrong redshift. ## 4. Read in a cutout of the VIS mosaic and overplot the galaxies @@ -289,14 +304,11 @@ science_images = image_table[image_table['dataproduct_subtype'] == 'science'] science_images['obs_id', 'energy_bandpassname', 'access_url'] ``` -We keep the VIS mosaic, which we use for the cutouts and in Firefly below; the first part of its `obs_id` is its tile ID: +We keep the VIS mosaic, which we use for the cutouts and in Firefly below: ```{code-cell} ipython3 vis_mosaic = science_images[science_images['energy_bandpassname'] == 'VIS'][0] filename = vis_mosaic['access_url'] -tileID = vis_mosaic['obs_id'][:9] - -print(f'The MER tile ID of this region is {tileID}') ``` The MER mosaics are large, so we read only a 5 x 5 arcmin cutout of the VIS mosaic directly from the cloud, without downloading the full file (see the [MER mosaics notebook](1_Euclid_intro_MER_images.md) for more details on the cloud access). @@ -323,85 +335,186 @@ We rely on astropy's WCSAxes framework for making plots of astronomical data in For more info, please visit the [WCSAxes documentation](https://docs.astropy.org/en/stable/visualization/wcsaxes/index.html). ``` +We overplot the galaxies with a reliable spectroscopic redshift, colored by their spectroscopic redshift. +In section 5, we look at the spectrum of the brightest of them in the H band, which we mark with a star: + +```{code-cell} ipython3 +brightest = galaxies[reliable] +if len(brightest) == 0: + raise ValueError('No galaxy has a reliable spectroscopic redshift; try a larger region or a different redshift range.') +brightest.sort(keys='flux_h_unif', reverse=True) +``` + ```{code-cell} ipython3 +fig = plt.figure(figsize=(8, 7)) ax = plt.subplot(projection=cutout_image.wcs) ax.imshow(cutout_image.data, cmap='gray', origin='lower', norm=ImageNormalize(cutout_image.data, interval=PercentileInterval(99.9), stretch=LogStretch())) -ax.scatter(result_galaxies['ra'], result_galaxies['dec'], s=36, facecolors='none', edgecolors='red', - transform=ax.get_transform('world')) +# Open circles, with their edge colored by the spectroscopic redshift +spec_z_values = np.asarray(brightest['spe_z'], dtype=float) +norm = plt.Normalize(spec_z_values.min(), spec_z_values.max()) +cmap = plt.get_cmap('cool') +ax.scatter(brightest['ra'], brightest['dec'], s=60, facecolors='none', edgecolors=cmap(norm(spec_z_values)), + linewidths=1.5, transform=ax.get_transform('world')) +ax.scatter(brightest['ra'][0], brightest['dec'][0], s=300, marker='*', facecolors='none', edgecolors='yellow', + linewidths=1.5, transform=ax.get_transform('world')) -_ = ax.set_title(f'Galaxies with photometric redshifts between {z_min} and {z_max}') +fig.colorbar(plt.cm.ScalarMappable(norm=norm, cmap=cmap), ax=ax, label='Spectroscopic redshift (spe_z)') +ax.set_title('Galaxies with a reliable spectroscopic redshift') +plt.show() ``` ## 5. Look at the spectrum and image of one of the brightest galaxies -Let's sort the galaxies with a reliable spectroscopic redshift by their H-band flux, and look at the five brightest ones: +Here are the five brightest galaxies with a reliable spectroscopic redshift, sorted by their H-band flux: ```{code-cell} ipython3 -brightest = galaxies[reliable] -brightest.sort(keys='flux_h_unif', reverse=True) brightest['object_id', 'flux_h_unif', 'phz_median', 'phz_90_int_1', 'phz_90_int_2', 'spe_z', 'spe_z_prob', 'spe_z_rel'][:5] ``` ### Retrieve the spectra The DR1 SIR 1D spectra are stored in large FITS files with one extension (HDU) per spectrum. -The table `euclid.objectid_spectrafile_association_dr1` at IRSA lists, for each object ID, the file (`path`) and the HDU of its spectra. -IRSA's spectrumDM service converts a given HDU of a spectrum file into a VOTable; its URL is built from the file path and HDU, which we do directly in the query. -We query the table for the brightest of these galaxies; not every galaxy with a spectroscopic redshift has a spectrum listed in it yet, so if none is found, we move on to the next brightest galaxy. +The table `euclid.objectid_spectrafile_association_dr1` at IRSA lists, for each object ID, the file (`path`) and the HDU of each of its spectra. +To read a spectrum, we don't download its whole file: IRSA's spectrumDM service reads a single HDU of a spectrum file in the cloud and returns it as a VOTable following the IVOA Spectrum Data Model, which astropy can read directly. +Its URL is built from the file path and the HDU number (see the [SIR 1D spectra notebook](3_Euclid_intro_1D_spectra.md)): ```{code-cell} ipython3 -for galaxy in brightest: - adql_spectra = f""" - SELECT objectid, tileid, hdu, - 'https://irsacloud.ipac.caltech.edu/api/spectrumdm/convert/' || path - || '?dataset_id=euclid_combspec_dr1&hdu=' || hdu AS url +SPECTRUMDM_URL = 'https://irsacloud.ipac.caltech.edu/api/spectrumdm/convert/' +SPECTRUM_DATASET_ID = 'euclid_combspec_dr1' + + +def spectrum_url(path, hdu): + """Build the spectrumDM URL of one spectrum. + + Parameters + ---------- + path : str + Path of the spectrum file, from the ``path`` column of the association table. + hdu : int + Number of the HDU of the spectrum in the file. + + Returns + ------- + str + URL that returns the spectrum as a VOTable. + """ + return f'{SPECTRUMDM_URL}{path}?dataset_id={SPECTRUM_DATASET_ID}&hdu={hdu}' + + +def find_spectra(object_id): + """Find the spectra of an object in the spectrum association table. + + Parameters + ---------- + object_id : int + MER object ID of the source. + + Returns + ------- + astropy.table.Table + One row per spectrum, with the file path and HDU number of the spectrum, and its spectrumDM URL. + """ + query = f""" + SELECT objectid, tileid, path, hdu FROM euclid.objectid_spectrafile_association_dr1 - WHERE objectid = {galaxy['object_id']} + WHERE objectid = {object_id} """ - result_spectra = Irsa.query_tap(adql_spectra).to_table() + spectrum_files = Irsa.query_tap(query).to_table() + spectrum_files['url'] = [spectrum_url(path, hdu) for path, hdu in zip(spectrum_files['path'], spectrum_files['hdu'])] + return spectrum_files +``` + +We find the spectra of the brightest galaxy; if a galaxy has no spectrum listed, we move on to the next brightest one: + +```{code-cell} ipython3 +for bright_galaxy in brightest: + result_spectra = find_spectra(bright_galaxy['object_id']) if len(result_spectra) > 0: break - print(f"No spectra listed for object {galaxy['object_id']}, trying the next galaxy") + print(f"No spectra listed for object {bright_galaxy['object_id']}, trying the next galaxy") +else: + raise ValueError('None of the galaxies with a reliable spectroscopic redshift has a spectrum listed; ' + 'try a larger region or a different redshift range.') -obj_id = galaxy['object_id'] -photo_z = galaxy['phz_median'] -spec_z = galaxy['spe_z'] -coords_galaxy = SkyCoord(galaxy['ra'], galaxy['dec'], unit='deg') +obj_id = bright_galaxy['object_id'] +photo_z = bright_galaxy['phz_median'] +spec_z = bright_galaxy['spe_z'] +coords_galaxy = SkyCoord(bright_galaxy['ra'], bright_galaxy['dec'], unit='deg') print(f'Object {obj_id}: photometric redshift {photo_z:.2f}, spectroscopic redshift {spec_z:.3f}') -result_spectra +result_spectra['objectid', 'tileid', 'hdu'] ``` -Each URL returns a VOTable containing the spectrum, which we read into an astropy table. -A galaxy in the Euclid Deep Fields can have spectra from both the red and the blue grism; as in the [SIR 1D spectra notebook](3_Euclid_intro_1D_spectra.md), we keep the spectrum with the most usable flux bins for each kind of grism. +Each URL returns a VOTable containing the spectrum, which we read into an astropy table: ```{code-cell} ipython3 spectra = [QTable.read(url) for url in result_spectra['url']] ``` +A galaxy in the Euclid Deep Fields can have spectra from both the red and the blue grism; as in the [SIR 1D spectra notebook](3_Euclid_intro_1D_spectra.md), we use the MASK column to find the flux bins not to use, and keep the spectrum with the most usable flux bins for each kind of grism: + ```{code-cell} ipython3 def bad_flux_bins(spectrum): - # Flux bins with an odd MASK value or a MASK value of 64 or more should not be used + """Find the flux bins of a spectrum that should not be used. + + Parameters + ---------- + spectrum : astropy.table.QTable + Spectrum, with a MASK column. + + Returns + ------- + numpy.ndarray + Boolean array, True for the flux bins with an odd MASK value or a MASK value of 64 or more. + """ mask = spectrum['MASK'].value return (mask % 2 == 1) | (mask >= 64) def grism_name(spectrum): - # Blue grism spectra start below 1.1 micron + """Tell which kind of grism a spectrum comes from. + + Parameters + ---------- + spectrum : astropy.table.QTable + Spectrum, with a WAVELENGTH column. + + Returns + ------- + str + ``'Blue grism'`` if the spectrum starts below 1.1 micron, ``'Red grism'`` otherwise. + """ return 'Blue grism' if spectrum['WAVELENGTH'].min() < 1.1 * u.micron else 'Red grism' -best = {} -for spectrum in sorted(spectra, key=lambda spectrum: np.sum(~bad_flux_bins(spectrum)), reverse=True): - best.setdefault(grism_name(spectrum), spectrum) +def best_spectra(spectra): + """Pick the spectrum with the most usable flux bins for each kind of grism. + + Parameters + ---------- + spectra : list of astropy.table.QTable + Spectra of one object. + + Returns + ------- + dict + Best spectrum of each kind of grism, keyed by the name of the grism. + """ + best = {} + for spectrum in sorted(spectra, key=lambda spectrum: np.sum(~bad_flux_bins(spectrum)), reverse=True): + best.setdefault(grism_name(spectrum), spectrum) + return best + + +best = best_spectra(spectra) list(best) ``` ### Plot the spectra -We plot the usable flux bins of the best spectrum of each grism, and mark the strongest emission lines at the spectroscopic redshift, to see how well the photometric redshift agrees with it: +We plot the usable flux bins of the best spectrum of each grism, and mark the strongest emission lines that fall in the wavelength range of the usable flux bins at the spectroscopic redshift, to see how well the photometric redshift agrees with it: ```{code-cell} ipython3 emission_lines = {'Hβ': 4862.7 * u.AA, '[OIII]': 5008.2 * u.AA, 'Hα': 6564.6 * u.AA} @@ -413,17 +526,23 @@ for name, spectrum in best.items(): ax.plot(spectrum['WAVELENGTH'].to(u.micron), np.ma.masked_where(bad, spectrum['SIGNAL']), color=grism_colors[name], alpha=0.8, label=name) +usable_wavelengths = [spectrum['WAVELENGTH'][~bad_flux_bins(spectrum)] for spectrum in best.values()] +wave_min = min(wavelengths.min() for wavelengths in usable_wavelengths) +wave_max = max(wavelengths.max() for wavelengths in usable_wavelengths) for line, rest_wavelength in emission_lines.items(): observed = (rest_wavelength * (1 + spec_z)).to(u.micron) - ax.axvline(observed, color='grey', linestyle='--', linewidth=0.8) - ax.text(observed.value, 0.98, f' {line}', transform=ax.get_xaxis_transform(), rotation=90, va='top', ha='right') + if wave_min < observed < wave_max: + ax.axvline(observed, color='grey', linestyle='--', linewidth=0.8) + ax.text(observed.value, 0.98, f' {line}', transform=ax.get_xaxis_transform(), rotation=90, va='top', ha='right') ax.set_title(f'Object {obj_id}: phz_median = {photo_z:.2f}, spe_z = {spec_z:.3f}') -_ = ax.legend(loc='upper left') +ax.legend(loc='upper left', bbox_to_anchor=(1.01, 1), fontsize='small') +plt.show() ``` -The Hα and [OIII] lines fall right at the spectroscopic redshift, which agrees well with the photometric redshift. -Not every feature is a line of this galaxy, though: slitless spectra are prone to contamination by neighboring sources, which is the likely origin of the narrow spike near 1.3 micron in the blue grism spectrum. +This galaxy has a single spectrum, from the blue grism. +Its Hα line is clearly detected at 1.24 micron, right at the spectroscopic redshift, which agrees well with the photometric redshift. +At this redshift, the [OIII] line falls at about 0.95 micron, near the start of the blue grism spectrum, where the spectrum is noisy, and it is not detected. ### Look at the galaxy on the VIS mosaic @@ -473,9 +592,11 @@ fc.show_table(tbl_stream) ## References - + -- Euclid Collaboration: Tucci, M., et al. 2026, *Euclid Data Release 1 (DR1): The PHZ processing*, A&A, in prep. +- Euclid Collaboration: Hartley, W., Paltani, S., Dubath, F., et al. 2026, *Euclid Data Release 1 (DR1): Photometric-redshift pipeline overview and validation*, A&A, in prep. +- Euclid Collaboration: Enia, A., Hartley, W. G., Bolzonella, M., et al. 2026, *Euclid Data Release 1 (DR1): The PHZ processing function for the physical properties of galaxies*, A&A, in prep. +- Euclid Collaboration: Tucci, M., Paltani, S., Hartley, W. G., et al. 2026, A&A, 711, A5, which describes the PHZ processing of Q1. +++ From a9cfb339509a27c990d2eef073f483c10db8b0aa Mon Sep 17 00:00:00 2001 From: xoubish Date: Mon, 9 Nov 2026 16:59:19 -0800 Subject: [PATCH 5/7] Address review comments on notebook 4 --- tutorials/euclid/4_Euclid_intro_PHZ_catalog.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/tutorials/euclid/4_Euclid_intro_PHZ_catalog.md b/tutorials/euclid/4_Euclid_intro_PHZ_catalog.md index 2f2fc474..cf92c058 100644 --- a/tutorials/euclid/4_Euclid_intro_PHZ_catalog.md +++ b/tutorials/euclid/4_Euclid_intro_PHZ_catalog.md @@ -192,8 +192,8 @@ We use a left outer join, which keeps the galaxies without a spectroscopic redsh ```{code-cell} ipython3 ######################## User defined section ############################ # Set the center and size of the region to select galaxies from (and of the image cutout below) -ra_cutout = 267.8 -dec_cutout = 66 +ra_cutout = 269.46 +dec_cutout = 66.0 im_cutout = 5 * u.arcmin # Redshift range @@ -523,8 +523,9 @@ grism_colors = {'Red grism': 'tab:red', 'Blue grism': 'tab:blue'} fig, ax = plt.subplots(figsize=(12, 4.5)) for name, spectrum in best.items(): bad = bad_flux_bins(spectrum) + # Draw the red grism spectrum on top, as it is less noisy where the two grisms overlap ax.plot(spectrum['WAVELENGTH'].to(u.micron), np.ma.masked_where(bad, spectrum['SIGNAL']), - color=grism_colors[name], alpha=0.8, label=name) + color=grism_colors[name], alpha=0.8, label=name, zorder=3 if name == 'Red grism' else 2) usable_wavelengths = [spectrum['WAVELENGTH'][~bad_flux_bins(spectrum)] for spectrum in best.values()] wave_min = min(wavelengths.min() for wavelengths in usable_wavelengths) @@ -540,9 +541,9 @@ ax.legend(loc='upper left', bbox_to_anchor=(1.01, 1), fontsize='small') plt.show() ``` -This galaxy has a single spectrum, from the blue grism. -Its Hα line is clearly detected at 1.24 micron, right at the spectroscopic redshift, which agrees well with the photometric redshift. -At this redshift, the [OIII] line falls at about 0.95 micron, near the start of the blue grism spectrum, where the spectrum is noisy, and it is not detected. +This galaxy has spectra from both grisms. +At its spectroscopic redshift, which agrees well with the photometric redshift, the Hα line falls at 1.30 micron, where the two grisms overlap, and it is seen in both spectra. +This bright galaxy has a strong continuum, and its emission lines are relatively weak: the Hβ and [OIII] lines, near 1 micron in the blue grism spectrum, are not clearly detected. ### Look at the galaxy on the VIS mosaic From dda926287e0555260827352fb97da95282bed928 Mon Sep 17 00:00:00 2001 From: xoubish Date: Wed, 11 Nov 2026 08:14:24 -0800 Subject: [PATCH 6/7] Point the notebook to the production IRSA services for DR1 --- tutorials/euclid/4_Euclid_intro_PHZ_catalog.md | 8 -------- 1 file changed, 8 deletions(-) diff --git a/tutorials/euclid/4_Euclid_intro_PHZ_catalog.md b/tutorials/euclid/4_Euclid_intro_PHZ_catalog.md index cf92c058..05499ddf 100644 --- a/tutorials/euclid/4_Euclid_intro_PHZ_catalog.md +++ b/tutorials/euclid/4_Euclid_intro_PHZ_catalog.md @@ -124,14 +124,6 @@ quantity_support() conf.remote_timeout = 120 ``` - - -```{code-cell} ipython3 -# DR1 is not public yet, so we query IRSA's development server. -Irsa.tap_url = 'https://irsadev.ipac.caltech.edu/TAP' -Irsa.sia_url = 'https://irsadev.ipac.caltech.edu/SIA' -``` - ## 1. Find the PHZ catalog tables at IRSA The Euclid catalogs at IRSA can be listed with the `list_catalogs` method; here we filter on the DR1 tables. From a25368152e1276f6938b6f3c1349e01a458e86ef Mon Sep 17 00:00:00 2001 From: xoubish Date: Wed, 11 Nov 2026 12:32:07 -0800 Subject: [PATCH 7/7] Require astroquery 0.4.11 for include_metadata_tables --- tutorials/euclid/4_Euclid_intro_PHZ_catalog.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tutorials/euclid/4_Euclid_intro_PHZ_catalog.md b/tutorials/euclid/4_Euclid_intro_PHZ_catalog.md index 05499ddf..ec765a22 100644 --- a/tutorials/euclid/4_Euclid_intro_PHZ_catalog.md +++ b/tutorials/euclid/4_Euclid_intro_PHZ_catalog.md @@ -84,12 +84,12 @@ Some column names differ from the Q1 PHZ catalog: for example the redshift inter ## Imports ```{important} -We rely on ``astroquery`` features that have been recently added, so please make sure you have version v0.4.10 or newer installed. +We rely on ``astroquery`` features that have been recently added, so please make sure you have version v0.4.11 or newer installed. ``` ```{code-cell} ipython3 # Uncomment the next line to install dependencies if needed. -# !pip install numpy matplotlib 'astropy>=5.3' 'astroquery>=0.4.10' fsspec s3fs firefly_client +# !pip install numpy matplotlib 'astropy>=5.3' 'astroquery>=0.4.11' fsspec s3fs firefly_client ``` ```{code-cell} ipython3