Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
b3e9225
Ignore local venv directories
jkrick Oct 22, 2026
562c35d
Point spherex_cutouts at irsatest and filter to DR1
jkrick Oct 22, 2026
f12b0e2
Point spherex_psf at irsatest, filter to DR1, fix ePSF fallback guard
jkrick Oct 22, 2026
2500fc4
Update Explanatory Supplement link for DR1 naming
jkrick Oct 22, 2026
818fea2
Update Explanatory Supplement link for DR1 naming
jkrick Oct 22, 2026
54b40bf
Mention periodic Data Releases on SPHEREx landing page
jkrick Oct 22, 2026
db5747e
Point SDT TAP query at irsatest and filter to DR1
jkrick Oct 22, 2026
5c16dcf
Fold in #150 refactor and #238 header preservation fix
jkrick Oct 22, 2026
61b4e2a
Point spherex_intro.md at the DR1 SIA collection
jkrick Oct 22, 2026
0bf8d7c
Point Source Discovery Tool at DR1 SIA and DR1 solid angle pixel maps
jkrick Oct 22, 2026
85df160
Read QR3/DR1 FLAGS header keywords in Source Discovery Tool photometry
jkrick Oct 22, 2026
ebd02ec
Build the Source Discovery Tool's D4 mask from D4 flags
jkrick Oct 22, 2026
a4ffe0d
Describe the EPSF extension instead of the old PSF cube in spherex_in…
jkrick Oct 23, 2026
c15f929
Update spherex_intro.md flag text and FLAGS header lookup for DR1
jkrick Oct 23, 2026
1146463
Explain combined L2 N_* pixel-count keywords in spherex_intro.md
jkrick Oct 28, 2026
6615d7b
Use the Explanatory Supplement's bad-pixel flags in SPHEREx notebooks
jkrick Oct 28, 2026
352a519
Add a draft skill for SPHEREx data-release notebook updates
jkrick Oct 28, 2026
fdf15a6
Merge upstream/main (PR #396) into spherex-dr1
jkrick Oct 28, 2026
9caed7f
Let the SPHEREx release skill test SIA against a pre-release host
jkrick Oct 29, 2026
50018d2
Move the SPHEREx release skill to its own branch
jkrick Oct 29, 2026
245086e
Select the Source Discovery Tool's D4 image by detector name
jkrick Oct 29, 2026
1a8231a
Say QR3 and DR1 supersede QR2 in SPHEREx notebooks
jkrick Nov 4, 2026
36012f6
Say IRSA searches no longer return QR2 in the PSF notebook
jkrick Nov 5, 2026
8a83a9e
Use DR1 and QR3 together and query the public IRSA server
jkrick Nov 9, 2026
b94d6ff
Merge branch 'main' into spherex-dr1
DeekshaMohanty Nov 12, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,10 @@ _build/
#pycache
__pycache__

# Local Python virtual environments
.venv/
venv/

# Local working notes, not part of the tutorials
CLAUDE.md
OU_PR_REVIEW.md
Expand Down
3 changes: 2 additions & 1 deletion tutorials/spherex/spherex.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,8 @@
SPHEREx observes the sky from roughly 0.75–5.0 µm using a single instrument that provides low-resolution spectroscopy (R ≈ 40–150) in hundreds of spectral channels for every point on the sky.
Its science goals span cosmology, galaxy evolution, and the interstellar medium, enabling measurements of large-scale structure, the cosmic history of star formation, and the distribution of key molecules and ices in the Milky Way and nearby galaxies.

SPHEREx data releases include weekly [Quick Release spectral image products](https://caltech-ipac.github.io/spherex-archive-documentation/spherex-data-products/) (multi-extension FITS files containing calibrated near-infrared surface brightness, variance, flags, modeled backgrounds, PSFs, and wavelength WCS) along with ancillary calibration and metadata files such as gain matrices, dark current maps, solid angle pixel maps, and detailed spectral WCS products for each detector.
SPHEREx data releases include weekly [Quick Release spectral image products](https://caltech-ipac.github.io/spherex-archive-documentation/spherex-data-products/) as well as periodic Data Releases (e.g., DR1) (multi-extension FITS files containing calibrated near-infrared surface brightness, variance, flags, modeled backgrounds, PSFs, and wavelength WCS) along with ancillary calibration and metadata files such as gain matrices, dark current maps, solid angle pixel maps, and detailed spectral WCS products for each detector.
QR3 and DR1 supersede QR2, and SPHEREx recommends using QR3 and DR1 over QR2.

```{notebook-gallery} notebook_metadata.yml
tutorials/spherex/spherex_intro.md
Expand Down
6 changes: 4 additions & 2 deletions tutorials/spherex/spherex_cutouts.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ SPHEREx data will be used to:

The community will also mine SPHEREx data and combine it with synergistic data sets to address a variety of additional topics in astrophysics.

More information is available in the [SPHEREx Explanatory Supplement](https://irsa.ipac.caltech.edu/data/SPHEREx/docs/SPHEREx_Expsupp_QR.pdf).
More information is available in the [SPHEREx Explanatory Supplement](https://irsa.ipac.caltech.edu/data/SPHEREx/docs/SPHEREx_Expsupp.pdf).

## 3. Imports

Expand Down Expand Up @@ -104,6 +104,8 @@ output_filename = 'spherex_cutouts_mef.fits'
Here we show how to use `astroquery`'s IRSA TAP interface to retrieve all images that overlap with the position defined above.
This query will retrieve a table of URLs that link to the MEF cutouts.
Each row in the table corresponds to a single cutout and includes the data access URL and an observation timestamp.
The SPHEREx metadata tables include DR1 and QR3 images.
QR3 and DR1 supersede QR2, and SPHEREx recommends using QR3 and DR1 over QR2.
The results are sorted from oldest to newest.

```{code-cell} ipython3
Expand Down Expand Up @@ -166,7 +168,7 @@ def process_cutout(row, coord, cache):
with fits.open(row["uri"], cache=cache) as hdulist:
# There are seven HDUs:
# 0 contains minimal metadata in the header and no data.
# 1 through 6 are: IMAGE, FLAGS, VARIANCE, ZODI, PSF, WCS-WAVE
# 1 through 6 are: IMAGE, FLAGS, VARIANCE, ZODI, EPSF, WCS-WAVE
header = hdulist[1].header

# Compute pixel coordinates corresponding to cutout position.
Expand Down
33 changes: 18 additions & 15 deletions tutorials/spherex/spherex_intro.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,8 @@ The community will also mine SPHEREx data and combine it with synergistic data s

More information about SPHEREx, the data releases, and the data products is available in the [SPHEREx Archive at IRSA User Guide](https://caltech-ipac.github.io/spherex-archive-documentation).

This notebook can be used with both Quick Release 3 (QR3) and Quick Release 2 (QR2) data.
This notebook uses data from SPHEREx Data Release 1 (DR1).
QR3 and DR1 supersede the earlier Quick Release 2 (QR2), and SPHEREx recommends using QR3 and DR1 over QR2.

+++

Expand Down Expand Up @@ -109,15 +110,15 @@ The IRSA SIA collections can be listed using using the ``list_collections`` meth
+++

The collections are documented at [SPHEREx Data Access: Application Program Interfaces (APIs)](https://caltech-ipac.github.io/spherex-archive-documentation/spherex-data-access#application-program-interfaces-apis)
There are currently four collections available:
The Spectral Image MEFs are available in four collections:

* `'spherex_qr3'` -- Quick Release 3 Spectral Image MEFs that are part of the SPHEREx **Wide Survey**
* `'spherex_qr3_deep'` -- Quick Release 3 Spectral Image MEFs that are part of the SPHEREx **Deep Survey**
* `'spherex_qr2'` -- Quick Release 2 Spectral Image MEFs that are part of the SPHEREx **Wide Survey**
* `'spherex_qr2_deep'` -- Quick Release 2 Spectral Image MEFs that are part of the SPHEREx **Deep Survey**
* `'spherex_dr1'` -- DR1 Spectral Image MEFs that are part of the SPHEREx **Wide Survey**
* `'spherex_dr1_deep'` -- DR1 Spectral Image MEFs that are part of the SPHEREx **Deep Survey**
* `'spherex_qr3'` -- QR3 Spectral Image MEFs that are part of the SPHEREx **Wide Survey**
* `'spherex_qr3_deep'` -- QR3 Spectral Image MEFs that are part of the SPHEREx **Deep Survey**

```{code-cell} ipython3
results = Irsa.query_sia(pos=(coord, search_radius), collection='spherex_qr2')
results = Irsa.query_sia(pos=(coord, search_radius), collection=['spherex_dr1', 'spherex_qr3'])
```

:::{note}
Expand Down Expand Up @@ -188,7 +189,7 @@ You can see that the Level 2 Spectral Image files are multi-extension FITS files
* **FLAGS:** A bitmap with per-pixel status and processing flags.
* **VARIANCE:** A per-pixel estimate of the variance.
* **ZODI:** An model of the zodiacal dust background signal. Note that this is not subtracted from the IMAGE extension.
* **PSF:** An image cube (3D array) where each plane represents a Point Spread Function (PSF) for a cutout in over-sampled pixel space.
* **EPSF:** A table of effective Point Spread Functions (ePSFs), one for each of 441 regions of the detector, in over-sampled pixel space.
* **WCS-WAVE:** Spectral WCS lookup table that maps pixel coordinates to the central wavelength and bandwidth of each pixel.

We will tour each extension in the sections below.
Expand Down Expand Up @@ -362,7 +363,7 @@ Now let's take a look at some header keywords that provide information about how
spectral_image_header['L2 N_*']
```

There are 14 flags in total. Typically, most of the pixels are identified as SOURCE pixels, which are pixels mapped to a known source. The remaining flags are described in Table 8 of the [SPHEREx Explanatory Supplement](https://irsa.ipac.caltech.edu/data/SPHEREx/docs/SPHEREx_Expsupp_QR.pdf).
These keywords count the pixels carrying most of the flags defined in the FLAGS header (see section 9), plus a few combined counts: keywords ending in `_NB` count pixels that are not also flagged as bad, and those ending in `_NS` count pixels that are not also flagged SOURCE. Typically, most of the pixels are identified as SOURCE pixels, which are pixels mapped to a known source. The flags are described in Table 16 of the [SPHEREx Explanatory Supplement](https://irsa.ipac.caltech.edu/data/SPHEREx/docs/SPHEREx_Expsupp.pdf).

+++

Expand All @@ -382,10 +383,12 @@ for keyword in keywords_to_print:
print(f"{keyword}: {value}")
```

Conveniently, the definitions of the flags are also provided in the FLAGS header.
Conveniently, the definitions of the flags are also provided in the FLAGS header. For each bit number n, the keyword `MSKNnnnn` gives the flag name and `MSKDnnnn` gives its description.

```{code-cell} ipython3
flags_header['MP*']
for key in flags_header['MSKN*']:
bit = key[4:]
print(f"{int(bit):2d} {flags_header[key]:<13} {flags_header['MSKD' + bit]}")
```

Tip: In the Firefly visualization, if you are looking at the first extension (IMAGE), you can open up the layers icon (top right) to enable overlaying a visualization of the flags.
Expand Down Expand Up @@ -420,14 +423,14 @@ for keyword in keywords_to_print:
print(f"{keyword}: {value}")
```

## 12. Explore the fifth extension: PSF
## 12. Explore the fifth extension: EPSF

The fifth extension of the MEF contains 121 Point-spread functions (PSFs); each PSF is represented as a 101 x 101 image and all 121 are assembled together into a cube. Each of the 121 layers represents a "super-resolution" PSF estimate in a different region (defined by an 11x11 grid) of the detector. Each PSF is a two-dimensional array with size of 101 × 101 pixels. The PSFs are oversampled such that 10 PSF pixels cover the same spatial extent as one spectral image pixel (0.615 arcsec). Let's look at some specific header keywords for the PSF:
The fifth extension of the MEF contains the effective Point Spread Function (ePSF), stored as a binary table. The detector is divided into a 21 x 21 grid of regions (each about 97 x 97 spectral image pixels), and each of the 441 rows of the table holds the ePSF for one region, along with that region's position on the detector (`XCENTER`, `YCENTER`) and its central wavelength and bandwidth (`CWAVE`, `CBAND`). Each ePSF is a 33 x 33 array, oversampled by a factor of 5 relative to the spectral image pixels (header keywords `OVSMPX` and `OVSMPY`). Unlike an optical PSF, the ePSF is a model of the pixel-convolved PSF: it predicts the pixel values of a point source at any sub-pixel position. The [SPHEREx ePSF tutorial](spherex_psf.md) shows how to find and use the ePSF for a given position. Let's look at some specific header keywords for the EPSF:

```{code-cell} ipython3
psf_header = hdulist[5].header
psf_header = hdulist["EPSF"].header

keywords_to_print = ['EXTNAME', 'NAXIS1', 'NAXIS2', 'NAXIS3', 'BUNIT']
keywords_to_print = ['EXTNAME', 'NAXIS2', 'DETECTOR', 'OVSMPX', 'OVSMPY']

for keyword in keywords_to_print:
value = psf_header.get(keyword, 'Keyword not found')
Expand Down
2 changes: 1 addition & 1 deletion tutorials/spherex/spherex_mosaic.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ SPHEREx data will be used to:

The community will also mine SPHEREx data and combine it with synergistic data sets to address a variety of additional topics in astrophysics.

More information is available in the [SPHEREx Explanatory Supplement](https://irsa.ipac.caltech.edu/data/SPHEREx/docs/SPHEREx_Expsupp_QR.pdf).
More information is available in the [SPHEREx Explanatory Supplement](https://irsa.ipac.caltech.edu/data/SPHEREx/docs/SPHEREx_Expsupp.pdf).

+++

Expand Down
69 changes: 41 additions & 28 deletions tutorials/spherex/spherex_psf.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,9 @@ kernelspec:

```{note}
This notebook is not intended for use of QR-1 data.
QR3 and DR1 supersede QR2, and SPHEREx recommends using QR3 and DR1 over QR2.
IRSA searches, including the one in this notebook, no longer return QR2 images.
If you already have QR2 images and prefer not to switch to QR3 or DR1, Section 5.2 shows how to obtain the ePSF for them.
```

+++
Expand All @@ -48,7 +51,7 @@ SPHEREx data will be used to:

The community will also mine SPHEREx data and combine it with synergistic data sets to address a variety of additional topics in astrophysics.

More information is available in the [SPHEREx Explanatory Supplement](https://irsa.ipac.caltech.edu/data/SPHEREx/docs/SPHEREx_Expsupp_QR.pdf).
More information is available in the [SPHEREx Explanatory Supplement](https://irsa.ipac.caltech.edu/data/SPHEREx/docs/SPHEREx_Expsupp.pdf).

+++

Expand Down Expand Up @@ -180,7 +183,7 @@ for attempt in range(max_retries):
hdul.info()
```

The downloaded SPHEREx image cutout contains 5 FITS layers, which are described in the [SPHEREx Explanatory Supplement](https://irsa.ipac.caltech.edu/data/SPHEREx/docs/SPHEREx_Expsupp_QR.pdf).
The downloaded SPHEREx image cutout contains a primary HDU plus 6 extensions (IMAGE, FLAGS, VARIANCE, ZODI, EPSF, WCS-WAVE), which are described in the [SPHEREx Explanatory Supplement](https://irsa.ipac.caltech.edu/data/SPHEREx/docs/SPHEREx_Expsupp.pdf).

Let's also get the version of the image. The version is needed later to decide how to handle the PSF. For obtaining the version of the image, we define a handy function.

Expand Down Expand Up @@ -243,7 +246,7 @@ We will use these coordinates later to obtain the correct spatial PSF.

## 5. Obtain the ePSF Depending on Image Version

The PSFs included in the SPHEREx LVF images has been updated. Some of the images therefore still include the old PSF (referred to as "PSF"), while newer versions of the images include the new PSF (referred to as "ePSF"). Differently than the previously used PSF, the ePSF is *not* an optical PSF. Instead, the ePSF is a subpixel-resolved model of the pixel-convolved PSF, which predicts pixel values for a point source at arbitrary sub-pixel positions. Additionally, we implemented a simpler way to read the PSF layer - it is now simply a binary table with array columns.
The PSFs included in the SPHEREx LVF images has been updated. Older images therefore still include the old PSF (referred to as "PSF"), while newer versions of the images include the new PSF (referred to as "ePSF"). Differently than the previously used PSF, the ePSF is *not* an optical PSF. Instead, the ePSF is a subpixel-resolved model of the pixel-convolved PSF, which predicts pixel values for a point source at arbitrary sub-pixel positions. Additionally, we implemented a simpler way to read the PSF layer - it is now simply a binary table with array columns.

The differentiation is as follows:
* Versions earlier than `7.0` contain the old PSF (saved in the `PSF` FITS layer). These are QR-1 and QR-2 data.
Expand All @@ -267,12 +270,14 @@ Note that the ePSFs are stored in the form of a binary table, which makes it str
try:
epsf_hdu = image_hdul["EPSF"]
except KeyError:
epsf_in_image = False
print(
"This image does not contain the EPSF extension. "
"It was likely created with an older version. "
"Please proceed to Section 5.2."
"Proceeding to Section 5.2."
)
else:
epsf_in_image = True
epsf_bintable = Table(epsf_hdu.data)
epsf_header = epsf_hdu.header
print("ePSF loaded.")
Expand All @@ -284,14 +289,15 @@ We advice the users **not** to use the older PSFs, which are stored in the `PSF`
Since in these older versions the ePSFs are not provided in the FITS images, we have to obtain them from the respective calibration products.

```{note}
Eventually, after the reprocessing of the data products, the older version LVF images will be replaced by newer versions. Generally, we advise to download the newest images (which also contain improved flagging and calibrations). However, note that the effective PSF of the images is unchanged, but the ePSF provides a better modeling of it than the previous PSFs. It is therefore safe to just exchange the PSFs and use the new ePSF for the older version images.
QR3 and DR1 supersede QR2, and SPHEREx recommends using QR3 and DR1 over QR2 (the newer images also contain improved flagging and calibrations). IRSA searches no longer return QR2 images, so the query in Section 4.1 finds only DR1 and QR3 images and the code cells in this section are skipped. If you already have QR2 images and prefer not to switch to QR3 or DR1, note that the effective PSF of the images is unchanged, but the ePSF provides a better modeling of it than the previous PSFs. It is therefore safe to just exchange the PSFs and use the new ePSF for the older version images. To do this, set `spectral_image_url` in Section 4.1 to your QR2 image and rerun the notebook from there.
```

For this, we first have to figure out the detector on which the image was taken, because the PSF depends on the wavelength (hence detector).

```{code-cell} ipython3
this_detector = image_hdul['IMAGE'].header["DETECTOR"]
print(f"Detector is {this_detector}")
if not epsf_in_image:
this_detector = image_hdul['IMAGE'].header["DETECTOR"]
print(f"Detector is {this_detector}")
```

Next, we load the calibration file for that detector. Note that each detector has one ePSF calibration file as for now we do not assume that the calibration changes as a function of time.
Expand All @@ -303,37 +309,44 @@ Specifically, the ePSF calibration files are stored in the respective release di
We first construct a dictionary linking the detectors with the most current ePSFs in this bucket.

```{code-cell} ipython3
ePSF_calibs = {
"1": "https://nasa-irsa-spherex.s3.us-east-1.amazonaws.com/qr3/epsf/cal-epsf-v1-2026-191/1/epsf_D1_spx_cal-epsf-v1-2026-191.fits",
"2": "https://nasa-irsa-spherex.s3.us-east-1.amazonaws.com/qr3/epsf/cal-epsf-v1-2026-191/2/epsf_D2_spx_cal-epsf-v1-2026-191.fits",
"3": "https://nasa-irsa-spherex.s3.us-east-1.amazonaws.com/qr3/epsf/cal-epsf-v2-2026-191/3/epsf_D3_spx_cal-epsf-v2-2026-191.fits",
"4": "https://nasa-irsa-spherex.s3.us-east-1.amazonaws.com/qr3/epsf/cal-epsf-v1-2026-191/4/epsf_D4_spx_cal-epsf-v1-2026-191.fits",
"5": "https://nasa-irsa-spherex.s3.us-east-1.amazonaws.com/qr3/epsf/cal-epsf-v1-2026-191/5/epsf_D5_spx_cal-epsf-v1-2026-191.fits",
"6": "https://nasa-irsa-spherex.s3.us-east-1.amazonaws.com/qr3/epsf/cal-epsf-v1-2026-191/6/epsf_D6_spx_cal-epsf-v1-2026-191.fits"
}
if not epsf_in_image:
ePSF_calibs = {
"1": "https://nasa-irsa-spherex.s3.us-east-1.amazonaws.com/qr3/epsf/cal-epsf-v1-2026-191/1/epsf_D1_spx_cal-epsf-v1-2026-191.fits",
"2": "https://nasa-irsa-spherex.s3.us-east-1.amazonaws.com/qr3/epsf/cal-epsf-v1-2026-191/2/epsf_D2_spx_cal-epsf-v1-2026-191.fits",
"3": "https://nasa-irsa-spherex.s3.us-east-1.amazonaws.com/qr3/epsf/cal-epsf-v2-2026-191/3/epsf_D3_spx_cal-epsf-v2-2026-191.fits",
"4": "https://nasa-irsa-spherex.s3.us-east-1.amazonaws.com/qr3/epsf/cal-epsf-v1-2026-191/4/epsf_D4_spx_cal-epsf-v1-2026-191.fits",
"5": "https://nasa-irsa-spherex.s3.us-east-1.amazonaws.com/qr3/epsf/cal-epsf-v1-2026-191/5/epsf_D5_spx_cal-epsf-v1-2026-191.fits",
"6": "https://nasa-irsa-spherex.s3.us-east-1.amazonaws.com/qr3/epsf/cal-epsf-v1-2026-191/6/epsf_D6_spx_cal-epsf-v1-2026-191.fits"
}
```

Now we can rad in the correct calibration product ePSF corresponding to the detector in which the image was taken.
Now we can read in the correct calibration product ePSF corresponding to the detector in which the image was taken.

```{code-cell} ipython3
with fits.open(ePSF_calibs[str(this_detector)]) as hdul:
hdul.info()
epsf_bintable = Table(hdul['EPSF'].data)
epsf_header = hdul['EPSF'].header
epsf_mosaic = hdul['EPSF_MOSAIC'].data
if not epsf_in_image:
with fits.open(ePSF_calibs[str(this_detector)]) as hdul:
hdul.info()
epsf_bintable = Table(hdul['EPSF'].data)
epsf_header = hdul['EPSF'].header
epsf_mosaic = hdul['EPSF_MOSAIC'].data
else:
print("ePSF was already loaded from the image in Section 5.1; skipping the calibration-file fallback.")
```

The calibration product contains the same `EPSF` extension as the SPHEREx LVF image files.
In addition, the calibration products contains an `EPSF_MOSAIC` layer, which is *not* available in the LVF images. The ePSF mosaic allows a quick visualization of all the ePSFs on the detector.

```{code-cell} ipython3
fig = plt.figure(figsize=(16,10))
ax1 = fig.add_subplot(1,2,1)

im1 = ax1.imshow(epsf_mosaic, origin='lower')
ax1.set_xlabel("$x$")
ax1.set_ylabel("$y$")
plt.show()
if not epsf_in_image:
fig = plt.figure(figsize=(16,10))
ax1 = fig.add_subplot(1,2,1)

im1 = ax1.imshow(epsf_mosaic, origin='lower')
ax1.set_xlabel("$x$")
ax1.set_ylabel("$y$")
plt.show()
else:
print("EPSF_MOSAIC is only available in the calibration product (Section 5.2), not in the LVF image itself.")
```

```{note}
Expand Down
Loading
Loading