Skip to content

Update SPHEREx tutorials for DR1 - #401

Merged
DeekshaMohanty merged 25 commits into
Caltech-IPAC:mainfrom
jkrick:spherex-dr1
Nov 12, 2026
Merged

DeekshaMohanty merged 25 commits into
Caltech-IPAC:mainfrom
jkrick:spherex-dr1

Conversation

@jkrick

@jkrick jkrick commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Closes #392 (IRSA-8059). related to #374 ( I think this PR closes that issue, but waiting for Andreas to confirm). Draft until SPHEREx DR1 is released (2026-11-12) -- do not merge before then.

Summary

All five SPHEREx tutorials were reviewed and tested for DR1:

  • spherex_intro.md: query the DR1 and QR3 SIA collections (spherex_dr1, spherex_qr3) and list all four DR1/QR3 Wide and Deep collections; describe the EPSF extension instead of the old PSF cube; read MSKN* FLAGS keywords (22 flags) and cite the Explanatory Supplement's bad-pixel list; explain the combined-L2 N_* pixel-count keywords; Explanatory Supplement link renamed.
  • spherex_cutouts.md: the TAP query returns DR1 and QR3 images (IRSA's metadata tables no longer hold QR2 after release); Explanatory Supplement link renamed.
  • spherex_psf.md: the TAP query returns DR1 and QR3 images; guard the sec. 5.2 QR ePSF fallback so it no longer overwrites a DR1 image's own ePSF; fix the extension-count text; Explanatory Supplement link renamed.
  • spherex_source_discovery_tool_demo.md (Source Discovery Tool): DR1 and QR3 SIA collections and DR1 solid-angle pixel maps (cal-sapm-v3-2026-191); read QR3/DR1 FLAGS header keywords in aperture_photometry.py; build the D4 mask from D4 flags; bad_flags = the supplement's CONTAINS_BAD_PIXEL list; select the D4 image by detector name (groups[3] picked D5 when a detector has no images at the search position); sec. 12 TAP query returns DR1 and QR3 images; Explanatory Supplement link renamed.
  • spherex_mosaic.md: Explanatory Supplement link renamed only. The M101 demo cube is a static file and is intentionally not regenerated with DR1.

Also: the spherex.md landing page now mentions periodic Data Releases alongside the weekly Quick Releases.

Before merging (release day, 2026-11-12 or later)

This PR was written and tested before DR1 was public. Per review, the notebooks already point at the public IRSA server, so they only find DR1 images once DR1 is public. Do the remaining steps in order once DR1 is public, and push the changes to this PR's branch.

  • 1. Confirm DR1 is public on IRSA. All of these should work from outside the IPAC network:
    • In Python, from astroquery.ipac.irsa import Irsa; Irsa.list_collections(servicetype='SIA', filter='spherex') lists spherex_dr1 and spherex_dr1_deep (alongside spherex_qr3 and spherex_qr3_deep). If the names differ, update them in spherex_intro.md (sec. 5, the collection list and the query_sia call) and in the Source Discovery Tool (sec. 5, data_release / sia_collection).
    • https://irsa.ipac.caltech.edu/ibe/data/spherex/dr1/ opens in a browser.
    • https://irsa.ipac.caltech.edu/data/SPHEREx/docs/SPHEREx_Expsupp.pdf opens (the notebooks link to it). If the supplement ends up at a different URL, update the link in all five notebooks: grep -rn "SPHEREx_Expsupp.pdf" tutorials/ finds all of them.
    • aws s3 ls --no-sign-request s3://nasa-irsa-spherex/dr1/ lists folders (the Source Discovery Tool reads images from there).
  • 2. Point the notebooks back at the public IRSA server. Done before release, per review (8a83a9e): the # TESTING ONLY setup (IRSA_HOST, Irsa.tap_url) and the DR1-only URI filter were removed from spherex_cutouts.md, spherex_psf.md, and the Source Discovery Tool; the TAP queries use https://irsa.ipac.caltech.edu. grep -rn "irsatest\|irsadev\|IRSA_HOST" tutorials/ finds nothing.
  • 3. Update the data citation. In the Source Discovery Tool's Acknowledgements, replace the QR2 DOI sentence ("The DOI for SPHEREx QR2 data is 10.26131/IRSA629 ...") with the DR1 DOI, which IRSA will publish at release.
  • 4. Update the dates. Set **Updated:** (in "About this Notebook" at the bottom) to the release date in all five SPHEREx notebooks.
  • 5. Run all five notebooks against the public server and check they finish without errors and the plots look reasonable: spherex_intro.md, spherex_cutouts.md, spherex_psf.md, spherex_mosaic.md, and the Source Discovery Tool. For example, jupytext --to notebook --execute spherex_intro.md.
    • The Source Discovery Tool needs SExtractor; its sec. 3.1 explains the conda environment (conda-spherex_sdt.yml).
    • The Firefly cells open a viewer tab; check the images show up there.
    • In the Source Discovery Tool, sec. 12.1 measures a hard-coded source, selected_ids = [65]. Its number comes from SExtractor and can change if the public search returns different images. Check that the spectrum plot at the end looks like a real source (before release it had ~276 points across 0.75-5 um). If not, pick another source number from the sec. 11 plots.
  • 6. Update the SPHEREx data-release skill PR (#SKILL_PR). That PR adds
    .claude/skills/spherex-data-release/ on the branch spherex-release-skill, which documents how
    this release was done. Once steps 1-5 are finished, commit to that branch (not this one):
    • in references/touchpoints.md, add the DR1 DOI, and correct the SIA collection names or the
      Explanatory Supplement URL if either changed at release;
    • add anything that went differently or took extra work on release day to
      references/decisions.md;
    • mark that PR ready for review after this one merges.

Testing so far

  • spherex_cutouts.md, spherex_psf.md: executed against irsatest with real DR1 data.
  • spherex_intro.md: executed against the pre-release DR1 SIA (179 results, 7 HDUs, EPSF 441 rows). Firefly displays the DR1 image (checked with a pre-release viewer); the production viewer gets a final check on release day.
  • Source Discovery Tool: executed end to end (secs. 4-12, including SExtractor, bokeh, and Firefly) against pre-release DR1 SIA + IBE, with SExtractor 2.29.0. Source 65 spectrum: 276 points, 0.75-5.0 um, median S/N ~11.
  • spherex_mosaic.md: link-only change; executed end to end (2026-10-29), no errors.
  • After review (8a83a9e): on irsatest, spherex.artifact has no QR2 rows; the PSF query returns 345 DR1 + 109 QR3 images with a DR1 image first; QR3 images include the EPSF extension. Notebooks not re-run since, because public IRSA doesn't serve DR1 yet (covered by step 5).

CI may fail until release day: the notebooks now query public IRSA, which doesn't serve DR1 yet.
🤖 Generated with Claude Code

jkrick and others added 21 commits October 22, 2026 10:51
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ation fix

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Replace the QR2/QR3 collection list with the DR1 wide/deep collections and
query spherex_dr1. Collection names are expected, not yet confirmed: DR1 is
not in SIA anywhere until the Nov 12 release.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
data_release qr2 -> dr1 (also sets the SIA collection and S3 SAPM paths),
and SAPM calibration cal-sapm-v2-2025-164 -> cal-sapm-v3-2026-191, the
version shipped under dr1/ (verified via irsatest IBE).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
QR3 and DR1 name flag bit n in MSKNnnnn instead of QR2's MP_<name> = n.
build_phot_table only looked for MP_ keys, so on DR1 it recorded no flags
and grab_star raised KeyError: 'TRANSIENT'. Read MSKN keys, falling back to
MP_ for QR2. Also drop PHANTOM and REFERENCE from the notebook's bad_flags
list; neither flag exists in DR1.

Tested against DR1 on irsatest (secs. 5, 7, 8, 12): 282 cutouts, 22 flags
recorded, spectrum spans 0.74-5.0 um.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mask4 checked flags_data2 == 2 (the D2 image's flags) instead of
flags_data4 == 2.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tro.md

DR1 (like QR3) replaces the 121-plane PSF image cube with an EPSF binary
table: 441 rows (21 x 21 detector regions), each a 33 x 33 ePSF oversampled
5x. Update the extension list, section 12 text, and the header keywords the
section 12 cell prints, and link to the ePSF tutorial. Checked against a DR1
file on irsatest.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DR1 (like QR3) defines flags as MSKNnnnn (name) / MSKDnnnn (description)
instead of MP_*, so flags_header['MP*'] printed nothing. Print a bit/name/
description table from the MSKN/MSKD keys instead. Also 14 -> 22 flags
(counted in DR1 FLAGS headers), and point the Explanatory Supplement link
at the renamed SPHEREx_Expsupp.pdf, dropping the QR-era table number.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Source Discovery Tool: bad_flags now matches the supplement's
CONTAINS_BAD_PIXEL flag list (drops FULLSAMPLE, PHANMISS and OUTLIER;
adds the new DR1 artifact flags), and grab_star skips flags that an
image's FLAGS header does not define. Intro: reword the L2 N_* paragraph
and cite the supplement's flag table.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Captures the DR1 update as a repeatable workflow: release-dependent spots
in each notebook, settled decisions and gotchas, a release-availability
check script, and a parameterized Source Discovery Tool test harness.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Conflicts in spherex_cutouts.md and spherex_psf.md: kept the DR1 test-host
block and release filter, took main's reviewed Caltech-IPAC#150/Caltech-IPAC#238 changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
run_sdt_release_test.py gains --sia-host to run the notebook's own SIA
cells against irsadev; decisions.md notes how to test SIA notebooks there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The skill gets its own PR (branch spherex-release-skill) so the DR1
tutorial PR covers only notebook changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The D4 image was taken from groups[3], which is D4 only when all six
detectors have images at the search position. At the demo position,
DR1 returns only D1, D2, D4 and D5, so groups[3] silently picked D5.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jkrick jkrick self-assigned this Oct 30, 2026
@jkrick jkrick added the content: spherex Content related issues/PRs for notebooks with SPHEREx relevance label Oct 30, 2026
@jkrick
jkrick requested a review from troyraen October 30, 2026 23:33
@jkrick
jkrick requested a review from DeekshaMohanty November 4, 2026 22:09
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jkrick

jkrick commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

Narrative in all SPHEREx notebooks and the landing page now states that QR3 and DR1 supersede QR2 and that SPHEREx recommends them; QR2 material (e.g. psf sec. 5.2) is kept for users still analyzing QR2.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jkrick

jkrick commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

Updated spherex_psf.md for the recent clarification that IRSA searches won't return QR2 after the Nov 12 DR1 release:

  • Top note and Section 5.2 note now say IRSA searches no longer return QR2 images, and address users who already have QR2 images
    and prefer not to switch to DR1 (without explaining how to find QR2 data).
  • Section 5.2 note tells those users to set spectral_image_url in Section 4.1 to their QR2 image and rerun from there.
  • Section 5 intro now says "Older images" include the old PSF, instead of "Some of the images".

No code changes needed: the Section 4.1 query returns only DR1 images, and Section 5.2 gets the ePSF from the QR3 calibration
files on S3 using fixed URLs. I ran Sections 4.2–6.2 on a QR2 cutout opened directly by URL and Section 5.2 worked.

Also checked the other SPHEREx notebooks: none of them search for QR2 (intro and SDT use spherex_dr1, cutouts and SDT TAP
queries filter to DR1), and the existing QR2 wording is still accurate.

@troyraen troyraen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @jkrick. My suggestions below are to use DR1 and QR3 together and to revert the irsatest testing setup.

Comment thread tutorials/spherex/spherex_cutouts.md Outdated
Comment on lines +112 to +118
# TESTING ONLY -- DR1 is not yet on production. Point at irsatest for now and
# revert IRSA_HOST to "https://irsa.ipac.caltech.edu" before merging at DR1 release.
# Note: `Irsa.tap_url` must be set directly -- `astroquery.ipac.irsa.conf.tap_url` is only read
# once when the module is first imported, so changing it afterward has no effect.
IRSA_HOST = "https://irsatest.ipac.caltech.edu"
Irsa.tap_url = f"{IRSA_HOST}/TAP"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# TESTING ONLY -- DR1 is not yet on production. Point at irsatest for now and
# revert IRSA_HOST to "https://irsa.ipac.caltech.edu" before merging at DR1 release.
# Note: `Irsa.tap_url` must be set directly -- `astroquery.ipac.irsa.conf.tap_url` is only read
# once when the module is first imported, so changing it afterward has no effect.
IRSA_HOST = "https://irsatest.ipac.caltech.edu"
Irsa.tap_url = f"{IRSA_HOST}/TAP"

Comment thread tutorials/spherex/spherex_cutouts.md Outdated
Comment on lines +123 to +133
# spherex.plane.datarelease is a date, not a release name, so DR1 rows are
# selected by matching the DR1 path in the artifact URI instead.
query = f"""
SELECT
'https://irsa.ipac.caltech.edu/' || a.uri || '?center={coord.ra.deg},{coord.dec.deg}d&size={size.value}' AS uri,
'{IRSA_HOST}/' || a.uri || '?center={coord.ra.deg},{coord.dec.deg}d&size={size.value}' AS uri,
p.time_bounds_lower
FROM spherex.artifact a
JOIN spherex.plane p ON a.planeid = p.planeid
WHERE 1 = CONTAINS(POINT('ICRS', {coord.ra.deg}, {coord.dec.deg}), p.poly)
AND p.energy_bandpassname = '{bandpass}'
AND a.uri LIKE 'ibe/data/spherex/dr1/%'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If accepted, line 108 ("this query keeps only DR1 images") needs updating.

Suggested change
# spherex.plane.datarelease is a date, not a release name, so DR1 rows are
# selected by matching the DR1 path in the artifact URI instead.
query = f"""
SELECT
'https://irsa.ipac.caltech.edu/' || a.uri || '?center={coord.ra.deg},{coord.dec.deg}d&size={size.value}' AS uri,
'{IRSA_HOST}/' || a.uri || '?center={coord.ra.deg},{coord.dec.deg}d&size={size.value}' AS uri,
p.time_bounds_lower
FROM spherex.artifact a
JOIN spherex.plane p ON a.planeid = p.planeid
WHERE 1 = CONTAINS(POINT('ICRS', {coord.ra.deg}, {coord.dec.deg}), p.poly)
AND p.energy_bandpassname = '{bandpass}'
AND a.uri LIKE 'ibe/data/spherex/dr1/%'
query = f"""
SELECT
'https://irsa.ipac.caltech.edu/' || a.uri || '?center={coord.ra.deg},{coord.dec.deg}d&size={size.value}' AS uri,
p.time_bounds_lower
FROM spherex.artifact a
JOIN spherex.plane p ON a.planeid = p.planeid
WHERE 1 = CONTAINS(POINT('ICRS', {coord.ra.deg}, {coord.dec.deg}), p.poly)
AND p.energy_bandpassname = '{bandpass}'

Comment thread tutorials/spherex/spherex_psf.md Outdated
Comment on lines +121 to +127
# TESTING ONLY -- DR1 is not yet on production. Point at irsatest for now and
# revert IRSA_HOST to "https://irsa.ipac.caltech.edu" before merging at DR1 release.
# Note: `Irsa.tap_url` must be set directly -- `astroquery.ipac.irsa.conf.tap_url` is only read
# once when the module is first imported, so changing it afterward has no effect.
IRSA_HOST = "https://irsatest.ipac.caltech.edu"
Irsa.tap_url = f"{IRSA_HOST}/TAP"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# TESTING ONLY -- DR1 is not yet on production. Point at irsatest for now and
# revert IRSA_HOST to "https://irsa.ipac.caltech.edu" before merging at DR1 release.
# Note: `Irsa.tap_url` must be set directly -- `astroquery.ipac.irsa.conf.tap_url` is only read
# once when the module is first imported, so changing it afterward has no effect.
IRSA_HOST = "https://irsatest.ipac.caltech.edu"
Irsa.tap_url = f"{IRSA_HOST}/TAP"

Comment thread tutorials/spherex/spherex_psf.md Outdated
Comment on lines +132 to +141
# spherex.plane.datarelease is a date, not a release name, so DR1 rows are
# selected by matching the DR1 path in the artifact URI instead.
query = f"""
SELECT
'https://irsa.ipac.caltech.edu/' || a.uri || '?center={coord.ra.deg},{coord.dec.deg}d&size={size.value}' AS uri,
'{IRSA_HOST}/' || a.uri || '?center={coord.ra.deg},{coord.dec.deg}d&size={size.value}' AS uri,
p.time_bounds_lower, p.energy_bandpassname, p.time_bounds_lower, p.time_bounds_upper, p.provenance_version
FROM spherex.artifact a
JOIN spherex.plane p ON a.planeid = p.planeid
WHERE 1 = CONTAINS(POINT('ICRS', {coord.ra.deg}, {coord.dec.deg}), p.poly)
AND a.uri LIKE 'ibe/data/spherex/dr1/%'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# spherex.plane.datarelease is a date, not a release name, so DR1 rows are
# selected by matching the DR1 path in the artifact URI instead.
query = f"""
SELECT
'https://irsa.ipac.caltech.edu/' || a.uri || '?center={coord.ra.deg},{coord.dec.deg}d&size={size.value}' AS uri,
'{IRSA_HOST}/' || a.uri || '?center={coord.ra.deg},{coord.dec.deg}d&size={size.value}' AS uri,
p.time_bounds_lower, p.energy_bandpassname, p.time_bounds_lower, p.time_bounds_upper, p.provenance_version
FROM spherex.artifact a
JOIN spherex.plane p ON a.planeid = p.planeid
WHERE 1 = CONTAINS(POINT('ICRS', {coord.ra.deg}, {coord.dec.deg}), p.poly)
AND a.uri LIKE 'ibe/data/spherex/dr1/%'
query = f"""
SELECT
'https://irsa.ipac.caltech.edu/' || a.uri || '?center={coord.ra.deg},{coord.dec.deg}d&size={size.value}' AS uri,
p.time_bounds_lower, p.energy_bandpassname, p.time_bounds_lower, p.time_bounds_upper, p.provenance_version
FROM spherex.artifact a
JOIN spherex.plane p ON a.planeid = p.planeid
WHERE 1 = CONTAINS(POINT('ICRS', {coord.ra.deg}, {coord.dec.deg}), p.poly)

Comment thread tutorials/spherex/spherex_intro.md Outdated
Comment on lines +113 to +116
The DR1 Spectral Image MEFs are available in two collections:

* `'spherex_qr3'` -- Quick Release 3 Spectral Image MEFs that are part of the SPHEREx **Wide Survey**
* `'spherex_qr3_deep'` -- Quick Release 3 Spectral Image MEFs that are part of the SPHEREx **Deep Survey**
* `'spherex_qr2'` -- Quick Release 2 Spectral Image MEFs that are part of the SPHEREx **Wide Survey**
* `'spherex_qr2_deep'` -- Quick Release 2 Spectral Image MEFs that are part of the SPHEREx **Deep Survey**
* `'spherex_dr1'` -- DR1 Spectral Image MEFs that are part of the SPHEREx **Wide Survey**
* `'spherex_dr1_deep'` -- DR1 Spectral Image MEFs that are part of the SPHEREx **Deep Survey**

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why delete QR3?

Suggested change
The DR1 Spectral Image MEFs are available in two collections:
* `'spherex_qr3'` -- Quick Release 3 Spectral Image MEFs that are part of the SPHEREx **Wide Survey**
* `'spherex_qr3_deep'` -- Quick Release 3 Spectral Image MEFs that are part of the SPHEREx **Deep Survey**
* `'spherex_qr2'` -- Quick Release 2 Spectral Image MEFs that are part of the SPHEREx **Wide Survey**
* `'spherex_qr2_deep'` -- Quick Release 2 Spectral Image MEFs that are part of the SPHEREx **Deep Survey**
* `'spherex_dr1'` -- DR1 Spectral Image MEFs that are part of the SPHEREx **Wide Survey**
* `'spherex_dr1_deep'` -- DR1 Spectral Image MEFs that are part of the SPHEREx **Deep Survey**
The Spectral Image MEFs are available in four collections:
* `'spherex_dr1'` -- DR1 Spectral Image MEFs that are part of the SPHEREx **Wide Survey**
* `'spherex_dr1_deep'` -- DR1 Spectral Image MEFs that are part of the SPHEREx **Deep Survey**
* `'spherex_qr3'` -- QR3 Spectral Image MEFs that are part of the SPHEREx **Wide Survey**
* `'spherex_qr3_deep'` -- QR3 Spectral Image MEFs that are part of the SPHEREx **Deep Survey**

Comment thread tutorials/spherex/spherex_intro.md Outdated

```{code-cell} ipython3
results = Irsa.query_sia(pos=(coord, search_radius), collection='spherex_qr2')
results = Irsa.query_sia(pos=(coord, search_radius), collection='spherex_dr1')

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
results = Irsa.query_sia(pos=(coord, search_radius), collection='spherex_dr1')
results = Irsa.query_sia(pos=(coord, search_radius), collection=['spherex_dr1', 'spherex_qr3'])

data_release = "dr1"

# Spectral image queries
sia_collection = f"spherex_{data_release}"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If accepted, line 214 ("we have used the collection spherex_dr1") needs updating.

Suggested change
sia_collection = f"spherex_{data_release}"
sia_collection = [f"spherex_{data_release}", "spherex_qr3"]

Comment on lines +672 to +678
# TESTING ONLY -- DR1 is not yet on production. Point at irsatest for now and
# revert IRSA_HOST to "https://irsa.ipac.caltech.edu" before merging at DR1 release.
IRSA_HOST = "https://irsatest.ipac.caltech.edu"

# Define the service endpoint for IRSA's Table Access Protocol (TAP)
# so that we can query SPHEREx metadata tables.
service = pyvo.dal.TAPService("https://irsa.ipac.caltech.edu/TAP")
service = pyvo.dal.TAPService(f"{IRSA_HOST}/TAP")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# TESTING ONLY -- DR1 is not yet on production. Point at irsatest for now and
# revert IRSA_HOST to "https://irsa.ipac.caltech.edu" before merging at DR1 release.
IRSA_HOST = "https://irsatest.ipac.caltech.edu"
# Define the service endpoint for IRSA's Table Access Protocol (TAP)
# so that we can query SPHEREx metadata tables.
service = pyvo.dal.TAPService("https://irsa.ipac.caltech.edu/TAP")
service = pyvo.dal.TAPService(f"{IRSA_HOST}/TAP")
# Define the service endpoint for IRSA's Table Access Protocol (TAP)
# so that we can query SPHEREx metadata tables.
service = pyvo.dal.TAPService("https://irsa.ipac.caltech.edu/TAP")

Comment on lines +687 to +696
# spherex.plane.datarelease is a date, not a release name, so DR1 rows are
# selected by matching the DR1 path in the artifact URI instead.
query = f"""
SELECT
'https://irsa.ipac.caltech.edu/' || a.uri || '?center={matched_stars['ra'][0]},{matched_stars['dec'][0]}d&size={size.value}' AS uri,
'{IRSA_HOST}/' || a.uri || '?center={matched_stars['ra'][0]},{matched_stars['dec'][0]}d&size={size.value}' AS uri,
p.time_bounds_lower
FROM spherex.artifact a
JOIN spherex.plane p ON a.planeid = p.planeid
WHERE 1 = CONTAINS(POINT('ICRS', {matched_stars['ra'][0]}, {matched_stars['dec'][0]}), p.poly)
AND a.uri LIKE 'ibe/data/spherex/dr1/%'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Revert IRSA_HOST change and drop DR1 filter so this returns DR1 + QR3.

Suggested change
# spherex.plane.datarelease is a date, not a release name, so DR1 rows are
# selected by matching the DR1 path in the artifact URI instead.
query = f"""
SELECT
'https://irsa.ipac.caltech.edu/' || a.uri || '?center={matched_stars['ra'][0]},{matched_stars['dec'][0]}d&size={size.value}' AS uri,
'{IRSA_HOST}/' || a.uri || '?center={matched_stars['ra'][0]},{matched_stars['dec'][0]}d&size={size.value}' AS uri,
p.time_bounds_lower
FROM spherex.artifact a
JOIN spherex.plane p ON a.planeid = p.planeid
WHERE 1 = CONTAINS(POINT('ICRS', {matched_stars['ra'][0]}, {matched_stars['dec'][0]}), p.poly)
AND a.uri LIKE 'ibe/data/spherex/dr1/%'
query = f"""
SELECT
'https://irsa.ipac.caltech.edu/' || a.uri || '?center={matched_stars['ra'][0]},{matched_stars['dec'][0]}d&size={size.value}' AS uri,
p.time_bounds_lower
FROM spherex.artifact a
JOIN spherex.plane p ON a.planeid = p.planeid
WHERE 1 = CONTAINS(POINT('ICRS', {matched_stars['ra'][0]}, {matched_stars['dec'][0]}), p.poly)

Per review on Caltech-IPAC#401:
- Remove the irsatest TESTING ONLY setup (IRSA_HOST, Irsa.tap_url)
  from the cutouts, PSF, and Source Discovery Tool notebooks.
- Drop the DR1-only URI filter from the TAP queries so they return
  DR1 and QR3 images.
- Query the spherex_dr1 and spherex_qr3 SIA collections in the intro
  and Source Discovery Tool; list all four DR1/QR3 collections.
- Update the narrative to match.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jkrick

jkrick commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @troyraen -- all suggestions applied in 8a83a9e:

  • Removed the irsatest TESTING ONLY setup (IRSA_HOST, Irsa.tap_url) from cutouts, psf, and the Source Discovery Tool; TAP queries now use https://irsa.ipac.caltech.edu.
  • Dropped the DR1-only URI filter, so the TAP queries return DR1 + QR3.
  • Intro and SDT now query ['spherex_dr1', 'spherex_qr3']; the intro lists all four DR1/QR3 Wide and Deep collections.
  • Updated the narrative to match (cutouts sec. 5, SDT sec. 6, psf sec. 5.2 note, and "prefer not to switch to QR3 or DR1" in psf).

Checked on irsatest: spherex.artifact has no QR2 rows, the psf query returns 345 DR1 + 109 QR3 images with a DR1 image first, and QR3 images carry the EPSF extension, so the sec. 5.2 fallback is still skipped. Notebooks will get a full run against production on release day (checklist step 5).

@jkrick
jkrick marked this pull request as ready for review November 12, 2026 19:39
@DeekshaMohanty
DeekshaMohanty merged commit cb30dcb into Caltech-IPAC:main Nov 12, 2026
4 of 14 checks passed
github-actions Bot pushed a commit that referenced this pull request Nov 12, 2026
Update SPHEREx tutorials for DR1 cb30dcb
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

content: spherex Content related issues/PRs for notebooks with SPHEREx relevance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Update SPHEREx notebooks for DR1

3 participants