Repository navigation
Update SPHEREx tutorials for DR1 - #401
Conversation
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ation fix Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Replace the QR2/QR3 collection list with the DR1 wide/deep collections and query spherex_dr1. Collection names are expected, not yet confirmed: DR1 is not in SIA anywhere until the Nov 12 release. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
data_release qr2 -> dr1 (also sets the SIA collection and S3 SAPM paths), and SAPM calibration cal-sapm-v2-2025-164 -> cal-sapm-v3-2026-191, the version shipped under dr1/ (verified via irsatest IBE). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
QR3 and DR1 name flag bit n in MSKNnnnn instead of QR2's MP_<name> = n. build_phot_table only looked for MP_ keys, so on DR1 it recorded no flags and grab_star raised KeyError: 'TRANSIENT'. Read MSKN keys, falling back to MP_ for QR2. Also drop PHANTOM and REFERENCE from the notebook's bad_flags list; neither flag exists in DR1. Tested against DR1 on irsatest (secs. 5, 7, 8, 12): 282 cutouts, 22 flags recorded, spectrum spans 0.74-5.0 um. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mask4 checked flags_data2 == 2 (the D2 image's flags) instead of flags_data4 == 2. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tro.md DR1 (like QR3) replaces the 121-plane PSF image cube with an EPSF binary table: 441 rows (21 x 21 detector regions), each a 33 x 33 ePSF oversampled 5x. Update the extension list, section 12 text, and the header keywords the section 12 cell prints, and link to the ePSF tutorial. Checked against a DR1 file on irsatest. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DR1 (like QR3) defines flags as MSKNnnnn (name) / MSKDnnnn (description) instead of MP_*, so flags_header['MP*'] printed nothing. Print a bit/name/ description table from the MSKN/MSKD keys instead. Also 14 -> 22 flags (counted in DR1 FLAGS headers), and point the Explanatory Supplement link at the renamed SPHEREx_Expsupp.pdf, dropping the QR-era table number. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Source Discovery Tool: bad_flags now matches the supplement's CONTAINS_BAD_PIXEL flag list (drops FULLSAMPLE, PHANMISS and OUTLIER; adds the new DR1 artifact flags), and grab_star skips flags that an image's FLAGS header does not define. Intro: reword the L2 N_* paragraph and cite the supplement's flag table. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Captures the DR1 update as a repeatable workflow: release-dependent spots in each notebook, settled decisions and gotchas, a release-availability check script, and a parameterized Source Discovery Tool test harness. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Conflicts in spherex_cutouts.md and spherex_psf.md: kept the DR1 test-host block and release filter, took main's reviewed Caltech-IPAC#150/Caltech-IPAC#238 changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
run_sdt_release_test.py gains --sia-host to run the notebook's own SIA cells against irsadev; decisions.md notes how to test SIA notebooks there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The skill gets its own PR (branch spherex-release-skill) so the DR1 tutorial PR covers only notebook changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The D4 image was taken from groups[3], which is D4 only when all six detectors have images at the search position. At the demo position, DR1 returns only D1, D2, D4 and D5, so groups[3] silently picked D5. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Narrative in all SPHEREx notebooks and the landing page now states that QR3 and DR1 supersede QR2 and that SPHEREx recommends them; QR2 material (e.g. psf sec. 5.2) is kept for users still analyzing QR2. |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Updated spherex_psf.md for the recent clarification that IRSA searches won't return QR2 after the Nov 12 DR1 release:
No code changes needed: the Section 4.1 query returns only DR1 images, and Section 5.2 gets the ePSF from the QR3 calibration Also checked the other SPHEREx notebooks: none of them search for QR2 (intro and SDT use spherex_dr1, cutouts and SDT TAP |
| # TESTING ONLY -- DR1 is not yet on production. Point at irsatest for now and | ||
| # revert IRSA_HOST to "https://irsa.ipac.caltech.edu" before merging at DR1 release. | ||
| # Note: `Irsa.tap_url` must be set directly -- `astroquery.ipac.irsa.conf.tap_url` is only read | ||
| # once when the module is first imported, so changing it afterward has no effect. | ||
| IRSA_HOST = "https://irsatest.ipac.caltech.edu" | ||
| Irsa.tap_url = f"{IRSA_HOST}/TAP" | ||
|
|
There was a problem hiding this comment.
| # TESTING ONLY -- DR1 is not yet on production. Point at irsatest for now and | |
| # revert IRSA_HOST to "https://irsa.ipac.caltech.edu" before merging at DR1 release. | |
| # Note: `Irsa.tap_url` must be set directly -- `astroquery.ipac.irsa.conf.tap_url` is only read | |
| # once when the module is first imported, so changing it afterward has no effect. | |
| IRSA_HOST = "https://irsatest.ipac.caltech.edu" | |
| Irsa.tap_url = f"{IRSA_HOST}/TAP" |
| # spherex.plane.datarelease is a date, not a release name, so DR1 rows are | ||
| # selected by matching the DR1 path in the artifact URI instead. | ||
| query = f""" | ||
| SELECT | ||
| 'https://irsa.ipac.caltech.edu/' || a.uri || '?center={coord.ra.deg},{coord.dec.deg}d&size={size.value}' AS uri, | ||
| '{IRSA_HOST}/' || a.uri || '?center={coord.ra.deg},{coord.dec.deg}d&size={size.value}' AS uri, | ||
| p.time_bounds_lower | ||
| FROM spherex.artifact a | ||
| JOIN spherex.plane p ON a.planeid = p.planeid | ||
| WHERE 1 = CONTAINS(POINT('ICRS', {coord.ra.deg}, {coord.dec.deg}), p.poly) | ||
| AND p.energy_bandpassname = '{bandpass}' | ||
| AND a.uri LIKE 'ibe/data/spherex/dr1/%' |
There was a problem hiding this comment.
If accepted, line 108 ("this query keeps only DR1 images") needs updating.
| # spherex.plane.datarelease is a date, not a release name, so DR1 rows are | |
| # selected by matching the DR1 path in the artifact URI instead. | |
| query = f""" | |
| SELECT | |
| 'https://irsa.ipac.caltech.edu/' || a.uri || '?center={coord.ra.deg},{coord.dec.deg}d&size={size.value}' AS uri, | |
| '{IRSA_HOST}/' || a.uri || '?center={coord.ra.deg},{coord.dec.deg}d&size={size.value}' AS uri, | |
| p.time_bounds_lower | |
| FROM spherex.artifact a | |
| JOIN spherex.plane p ON a.planeid = p.planeid | |
| WHERE 1 = CONTAINS(POINT('ICRS', {coord.ra.deg}, {coord.dec.deg}), p.poly) | |
| AND p.energy_bandpassname = '{bandpass}' | |
| AND a.uri LIKE 'ibe/data/spherex/dr1/%' | |
| query = f""" | |
| SELECT | |
| 'https://irsa.ipac.caltech.edu/' || a.uri || '?center={coord.ra.deg},{coord.dec.deg}d&size={size.value}' AS uri, | |
| p.time_bounds_lower | |
| FROM spherex.artifact a | |
| JOIN spherex.plane p ON a.planeid = p.planeid | |
| WHERE 1 = CONTAINS(POINT('ICRS', {coord.ra.deg}, {coord.dec.deg}), p.poly) | |
| AND p.energy_bandpassname = '{bandpass}' |
| # TESTING ONLY -- DR1 is not yet on production. Point at irsatest for now and | ||
| # revert IRSA_HOST to "https://irsa.ipac.caltech.edu" before merging at DR1 release. | ||
| # Note: `Irsa.tap_url` must be set directly -- `astroquery.ipac.irsa.conf.tap_url` is only read | ||
| # once when the module is first imported, so changing it afterward has no effect. | ||
| IRSA_HOST = "https://irsatest.ipac.caltech.edu" | ||
| Irsa.tap_url = f"{IRSA_HOST}/TAP" | ||
|
|
There was a problem hiding this comment.
| # TESTING ONLY -- DR1 is not yet on production. Point at irsatest for now and | |
| # revert IRSA_HOST to "https://irsa.ipac.caltech.edu" before merging at DR1 release. | |
| # Note: `Irsa.tap_url` must be set directly -- `astroquery.ipac.irsa.conf.tap_url` is only read | |
| # once when the module is first imported, so changing it afterward has no effect. | |
| IRSA_HOST = "https://irsatest.ipac.caltech.edu" | |
| Irsa.tap_url = f"{IRSA_HOST}/TAP" |
| # spherex.plane.datarelease is a date, not a release name, so DR1 rows are | ||
| # selected by matching the DR1 path in the artifact URI instead. | ||
| query = f""" | ||
| SELECT | ||
| 'https://irsa.ipac.caltech.edu/' || a.uri || '?center={coord.ra.deg},{coord.dec.deg}d&size={size.value}' AS uri, | ||
| '{IRSA_HOST}/' || a.uri || '?center={coord.ra.deg},{coord.dec.deg}d&size={size.value}' AS uri, | ||
| p.time_bounds_lower, p.energy_bandpassname, p.time_bounds_lower, p.time_bounds_upper, p.provenance_version | ||
| FROM spherex.artifact a | ||
| JOIN spherex.plane p ON a.planeid = p.planeid | ||
| WHERE 1 = CONTAINS(POINT('ICRS', {coord.ra.deg}, {coord.dec.deg}), p.poly) | ||
| AND a.uri LIKE 'ibe/data/spherex/dr1/%' |
There was a problem hiding this comment.
| # spherex.plane.datarelease is a date, not a release name, so DR1 rows are | |
| # selected by matching the DR1 path in the artifact URI instead. | |
| query = f""" | |
| SELECT | |
| 'https://irsa.ipac.caltech.edu/' || a.uri || '?center={coord.ra.deg},{coord.dec.deg}d&size={size.value}' AS uri, | |
| '{IRSA_HOST}/' || a.uri || '?center={coord.ra.deg},{coord.dec.deg}d&size={size.value}' AS uri, | |
| p.time_bounds_lower, p.energy_bandpassname, p.time_bounds_lower, p.time_bounds_upper, p.provenance_version | |
| FROM spherex.artifact a | |
| JOIN spherex.plane p ON a.planeid = p.planeid | |
| WHERE 1 = CONTAINS(POINT('ICRS', {coord.ra.deg}, {coord.dec.deg}), p.poly) | |
| AND a.uri LIKE 'ibe/data/spherex/dr1/%' | |
| query = f""" | |
| SELECT | |
| 'https://irsa.ipac.caltech.edu/' || a.uri || '?center={coord.ra.deg},{coord.dec.deg}d&size={size.value}' AS uri, | |
| p.time_bounds_lower, p.energy_bandpassname, p.time_bounds_lower, p.time_bounds_upper, p.provenance_version | |
| FROM spherex.artifact a | |
| JOIN spherex.plane p ON a.planeid = p.planeid | |
| WHERE 1 = CONTAINS(POINT('ICRS', {coord.ra.deg}, {coord.dec.deg}), p.poly) |
| The DR1 Spectral Image MEFs are available in two collections: | ||
|
|
||
| * `'spherex_qr3'` -- Quick Release 3 Spectral Image MEFs that are part of the SPHEREx **Wide Survey** | ||
| * `'spherex_qr3_deep'` -- Quick Release 3 Spectral Image MEFs that are part of the SPHEREx **Deep Survey** | ||
| * `'spherex_qr2'` -- Quick Release 2 Spectral Image MEFs that are part of the SPHEREx **Wide Survey** | ||
| * `'spherex_qr2_deep'` -- Quick Release 2 Spectral Image MEFs that are part of the SPHEREx **Deep Survey** | ||
| * `'spherex_dr1'` -- DR1 Spectral Image MEFs that are part of the SPHEREx **Wide Survey** | ||
| * `'spherex_dr1_deep'` -- DR1 Spectral Image MEFs that are part of the SPHEREx **Deep Survey** |
There was a problem hiding this comment.
Why delete QR3?
| The DR1 Spectral Image MEFs are available in two collections: | |
| * `'spherex_qr3'` -- Quick Release 3 Spectral Image MEFs that are part of the SPHEREx **Wide Survey** | |
| * `'spherex_qr3_deep'` -- Quick Release 3 Spectral Image MEFs that are part of the SPHEREx **Deep Survey** | |
| * `'spherex_qr2'` -- Quick Release 2 Spectral Image MEFs that are part of the SPHEREx **Wide Survey** | |
| * `'spherex_qr2_deep'` -- Quick Release 2 Spectral Image MEFs that are part of the SPHEREx **Deep Survey** | |
| * `'spherex_dr1'` -- DR1 Spectral Image MEFs that are part of the SPHEREx **Wide Survey** | |
| * `'spherex_dr1_deep'` -- DR1 Spectral Image MEFs that are part of the SPHEREx **Deep Survey** | |
| The Spectral Image MEFs are available in four collections: | |
| * `'spherex_dr1'` -- DR1 Spectral Image MEFs that are part of the SPHEREx **Wide Survey** | |
| * `'spherex_dr1_deep'` -- DR1 Spectral Image MEFs that are part of the SPHEREx **Deep Survey** | |
| * `'spherex_qr3'` -- QR3 Spectral Image MEFs that are part of the SPHEREx **Wide Survey** | |
| * `'spherex_qr3_deep'` -- QR3 Spectral Image MEFs that are part of the SPHEREx **Deep Survey** |
|
|
||
| ```{code-cell} ipython3 | ||
| results = Irsa.query_sia(pos=(coord, search_radius), collection='spherex_qr2') | ||
| results = Irsa.query_sia(pos=(coord, search_radius), collection='spherex_dr1') |
There was a problem hiding this comment.
| results = Irsa.query_sia(pos=(coord, search_radius), collection='spherex_dr1') | |
| results = Irsa.query_sia(pos=(coord, search_radius), collection=['spherex_dr1', 'spherex_qr3']) |
| data_release = "dr1" | ||
|
|
||
| # Spectral image queries | ||
| sia_collection = f"spherex_{data_release}" |
There was a problem hiding this comment.
If accepted, line 214 ("we have used the collection spherex_dr1") needs updating.
| sia_collection = f"spherex_{data_release}" | |
| sia_collection = [f"spherex_{data_release}", "spherex_qr3"] |
| # TESTING ONLY -- DR1 is not yet on production. Point at irsatest for now and | ||
| # revert IRSA_HOST to "https://irsa.ipac.caltech.edu" before merging at DR1 release. | ||
| IRSA_HOST = "https://irsatest.ipac.caltech.edu" | ||
|
|
||
| # Define the service endpoint for IRSA's Table Access Protocol (TAP) | ||
| # so that we can query SPHEREx metadata tables. | ||
| service = pyvo.dal.TAPService("https://irsa.ipac.caltech.edu/TAP") | ||
| service = pyvo.dal.TAPService(f"{IRSA_HOST}/TAP") |
There was a problem hiding this comment.
| # TESTING ONLY -- DR1 is not yet on production. Point at irsatest for now and | |
| # revert IRSA_HOST to "https://irsa.ipac.caltech.edu" before merging at DR1 release. | |
| IRSA_HOST = "https://irsatest.ipac.caltech.edu" | |
| # Define the service endpoint for IRSA's Table Access Protocol (TAP) | |
| # so that we can query SPHEREx metadata tables. | |
| service = pyvo.dal.TAPService("https://irsa.ipac.caltech.edu/TAP") | |
| service = pyvo.dal.TAPService(f"{IRSA_HOST}/TAP") | |
| # Define the service endpoint for IRSA's Table Access Protocol (TAP) | |
| # so that we can query SPHEREx metadata tables. | |
| service = pyvo.dal.TAPService("https://irsa.ipac.caltech.edu/TAP") |
| # spherex.plane.datarelease is a date, not a release name, so DR1 rows are | ||
| # selected by matching the DR1 path in the artifact URI instead. | ||
| query = f""" | ||
| SELECT | ||
| 'https://irsa.ipac.caltech.edu/' || a.uri || '?center={matched_stars['ra'][0]},{matched_stars['dec'][0]}d&size={size.value}' AS uri, | ||
| '{IRSA_HOST}/' || a.uri || '?center={matched_stars['ra'][0]},{matched_stars['dec'][0]}d&size={size.value}' AS uri, | ||
| p.time_bounds_lower | ||
| FROM spherex.artifact a | ||
| JOIN spherex.plane p ON a.planeid = p.planeid | ||
| WHERE 1 = CONTAINS(POINT('ICRS', {matched_stars['ra'][0]}, {matched_stars['dec'][0]}), p.poly) | ||
| AND a.uri LIKE 'ibe/data/spherex/dr1/%' |
There was a problem hiding this comment.
Revert IRSA_HOST change and drop DR1 filter so this returns DR1 + QR3.
| # spherex.plane.datarelease is a date, not a release name, so DR1 rows are | |
| # selected by matching the DR1 path in the artifact URI instead. | |
| query = f""" | |
| SELECT | |
| 'https://irsa.ipac.caltech.edu/' || a.uri || '?center={matched_stars['ra'][0]},{matched_stars['dec'][0]}d&size={size.value}' AS uri, | |
| '{IRSA_HOST}/' || a.uri || '?center={matched_stars['ra'][0]},{matched_stars['dec'][0]}d&size={size.value}' AS uri, | |
| p.time_bounds_lower | |
| FROM spherex.artifact a | |
| JOIN spherex.plane p ON a.planeid = p.planeid | |
| WHERE 1 = CONTAINS(POINT('ICRS', {matched_stars['ra'][0]}, {matched_stars['dec'][0]}), p.poly) | |
| AND a.uri LIKE 'ibe/data/spherex/dr1/%' | |
| query = f""" | |
| SELECT | |
| 'https://irsa.ipac.caltech.edu/' || a.uri || '?center={matched_stars['ra'][0]},{matched_stars['dec'][0]}d&size={size.value}' AS uri, | |
| p.time_bounds_lower | |
| FROM spherex.artifact a | |
| JOIN spherex.plane p ON a.planeid = p.planeid | |
| WHERE 1 = CONTAINS(POINT('ICRS', {matched_stars['ra'][0]}, {matched_stars['dec'][0]}), p.poly) |
Per review on Caltech-IPAC#401: - Remove the irsatest TESTING ONLY setup (IRSA_HOST, Irsa.tap_url) from the cutouts, PSF, and Source Discovery Tool notebooks. - Drop the DR1-only URI filter from the TAP queries so they return DR1 and QR3 images. - Query the spherex_dr1 and spherex_qr3 SIA collections in the intro and Source Discovery Tool; list all four DR1/QR3 collections. - Update the narrative to match. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks @troyraen -- all suggestions applied in 8a83a9e:
Checked on irsatest: |
Update SPHEREx tutorials for DR1 cb30dcb
Closes #392 (IRSA-8059). related to #374 ( I think this PR closes that issue, but waiting for Andreas to confirm). Draft until SPHEREx DR1 is released (2026-11-12) -- do not merge before then.
Summary
All five SPHEREx tutorials were reviewed and tested for DR1:
spherex_dr1,spherex_qr3) and list all four DR1/QR3 Wide and Deep collections; describe the EPSF extension instead of the old PSF cube; readMSKN*FLAGS keywords (22 flags) and cite the Explanatory Supplement's bad-pixel list; explain the combined-L2N_*pixel-count keywords; Explanatory Supplement link renamed.cal-sapm-v3-2026-191); read QR3/DR1 FLAGS header keywords inaperture_photometry.py; build the D4 mask from D4 flags;bad_flags= the supplement's CONTAINS_BAD_PIXEL list; select the D4 image by detector name (groups[3]picked D5 when a detector has no images at the search position); sec. 12 TAP query returns DR1 and QR3 images; Explanatory Supplement link renamed.Also: the spherex.md landing page now mentions periodic Data Releases alongside the weekly Quick Releases.
Before merging (release day, 2026-11-12 or later)
This PR was written and tested before DR1 was public. Per review, the notebooks already point at the public IRSA server, so they only find DR1 images once DR1 is public. Do the remaining steps in order once DR1 is public, and push the changes to this PR's branch.
from astroquery.ipac.irsa import Irsa; Irsa.list_collections(servicetype='SIA', filter='spherex')listsspherex_dr1andspherex_dr1_deep(alongsidespherex_qr3andspherex_qr3_deep). If the names differ, update them inspherex_intro.md(sec. 5, the collection list and thequery_siacall) and in the Source Discovery Tool (sec. 5,data_release/sia_collection).grep -rn "SPHEREx_Expsupp.pdf" tutorials/finds all of them.aws s3 ls --no-sign-request s3://nasa-irsa-spherex/dr1/lists folders (the Source Discovery Tool reads images from there).# TESTING ONLYsetup (IRSA_HOST,Irsa.tap_url) and the DR1-only URI filter were removed fromspherex_cutouts.md,spherex_psf.md, and the Source Discovery Tool; the TAP queries usehttps://irsa.ipac.caltech.edu.grep -rn "irsatest\|irsadev\|IRSA_HOST" tutorials/finds nothing.**Updated:**(in "About this Notebook" at the bottom) to the release date in all five SPHEREx notebooks.spherex_intro.md,spherex_cutouts.md,spherex_psf.md,spherex_mosaic.md, and the Source Discovery Tool. For example,jupytext --to notebook --execute spherex_intro.md.conda-spherex_sdt.yml).selected_ids = [65]. Its number comes from SExtractor and can change if the public search returns different images. Check that the spectrum plot at the end looks like a real source (before release it had ~276 points across 0.75-5 um). If not, pick another source number from the sec. 11 plots..claude/skills/spherex-data-release/on the branchspherex-release-skill, which documents howthis release was done. Once steps 1-5 are finished, commit to that branch (not this one):
references/touchpoints.md, add the DR1 DOI, and correct the SIA collection names or theExplanatory Supplement URL if either changed at release;
references/decisions.md;Testing so far
spherex.artifacthas no QR2 rows; the PSF query returns 345 DR1 + 109 QR3 images with a DR1 image first; QR3 images include the EPSF extension. Notebooks not re-run since, because public IRSA doesn't serve DR1 yet (covered by step 5).CI may fail until release day: the notebooks now query public IRSA, which doesn't serve DR1 yet.
🤖 Generated with Claude Code