[tkislan]

@coderabbitai ignore

Summary

Phase 2 of the "Full Dataframe storage" RFC (§3.6.4): storing a block's result automatically. Stacked on #135, which adds artifacts.store_dataframe and the writer; this PR only adds the hook on top of it, so review that one first. The diff against it is the hook, its tests, one manifest field and the query-preview test builders, now shared.

When a Python or SQL block has a storage setting, the webapp puts the setting's frame name in the execute_request metadata (deepnote.dataframeStorage = {name, blockId}). A post_run_cell hook then stores the cell's result under that name, as if the block ended with artifacts.store_dataframe(result, name), and records the block in the manifest as block_id. A later call to store_dataframe writes no block_id, so a frame that code stored last belongs to no block.

• deepnote_toolkit/artifacts/dataframe_storage.py: the hook, registered as one more guarded step in init_deepnote_runtime(), the metadata reader, the error reports, and is_storable_frame, which decides which results are stored.
• deepnote_toolkit/artifacts/dataframe_storage_manifest.py: Manifest has an optional block_id field, which write_manifest takes and leaves out of the JSON when it is absent.
• DEEPNOTE_DATAFRAME_STORAGE_ENABLED switches the hook on. It does not affect store_dataframe.
• No code is added to user cells.

Out of scope (other repos): the block setting, webapp and executor metadata, the feature flag, the read endpoint, and the cleanup that deletes a block's frame when its setting is cleared.

Behaviour worth reviewing

• Never raises, never prints into the cell. Warnings are suppressed. Failures are reported to the webapp once per kernel session per errno or exception type, as TOOLKIT_RUNTIME_ERROR with code: DATAFRAME_STORAGE_WRITE_FAILED (the type toolkit/errors sends to Sentry, grouped by code), with requests like the other userpod-API callers. An error status from the webapp is logged to the file logger as well. The report carries the exception type, never its message, because pyarrow quotes the failing value.
• Stop during a write. IPython would catch a re-raised KeyboardInterrupt, print "Error in callback" and keep the cell successful, so the hook sets result.error_in_exec and calls showtraceback(). ipykernel builds the reply from the result after the hook, so the reply carries KeyboardInterrupt and the executor stops its queue. No SIGINT handler is installed: a cell with top-level await runs post_run_cell under a handler that only queues the interrupt, so Stop during that write is lost and the run goes on, as with any blocking code at the end of such a cell.
• Untrusted inputs. name and blockId from request metadata must match [A-Za-z0-9_-]{1,128}, because the name becomes a directory.
• Skipped: a read-only mount (its first write fails with EROFS before anything is converted), SQL query previews, PySpark and pandas-on-Spark frames, non-DataFrame results, failed or interrupted cells, and helper executions. Helpers are recognised by result.info.store_history, not execution_count: IPython 9 sets execution_count even when no history is stored, so the RFC's earlier "no execution_count" premise only holds on IPython 8.
• Env var: enabled for 1, true, yes, on (case-insensitive), the same set as _to_bool in deepnote_core/config/loader.py.

Changes since the previous revision of this PR

The RFC moved from per-block storage to named frames, and its first part (the function) moved to #135:

• Frames are written to deepnote_dataframes/<name>/, not .deepnote/dataframes/<notebookId>/<blockId>/.
• The request metadata is {name, blockId}. enabled and notebookId are gone: the webapp adds the field only for executions that should store.
• The writer, manifest, compression, read-only handling and project-root logic are in feat(artifacts): add store_dataframe and delete_dataframe #135.
• No SIGINT handling: the handler swap around the write (_sigint_raises) is gone, as is feat(artifacts): add store_dataframe and delete_dataframe #135's deferral of Stop during the manifest update.
• Merged feat(artifacts): add store_dataframe and delete_dataframe #135's latest changes. The manifest is a pydantic model, so block_id is a field of it. store_frame takes the typed frame instead of a kind, so the hook checks results with is_storable_frame instead of classify_frame. Cleanup checks previous where it unlinks it. The hook tests follow feat(artifacts): add store_dataframe and delete_dataframe #135's test cuts: the case that asserted whatever pyarrow does is gone, and one of the five unserializable frames is left.

Not verified here

• Python 3.10-3.12 and 3.14, and pyarrow 16.1. Only 3.13 / pyarrow 24.0 / pandas 2.2.3 / polars 1.39 ran locally.
• A real s3fs mounter, and whether polars writing to a Python file object surfaces a failed upload (fallback: flush plus fsync).
• A real kernel on the top-level-await path. IPython 9.12 (what the lock pins for Python 3.14) was run locally against the new tests, in a scratch install, not through a real kernel.
• The project root copies set_notebook_path()'s precedence, which uses home_dir without a /work suffix. Confirm on a pod that it equals the project mount.
• The DEEPNOTE_DATAFRAME_STORAGE_ENABLED value comes from the sandbox manager, which appends a project's own variables after the system ones, so a project can override it (RFC §3.6.2). Not fixed here.

Test plan

• poetry run python -m pytest tests/unit/artifacts tests/unit/test_runtime_initialization.py: 182 passed (the 107 tests of feat(artifacts): add store_dataframe and delete_dataframe #135 and 75 for the hook)
• At the previous revision, the same tests under IPython 9.12.0, installed to a scratch dir (the lock's 3.14 pin): 229 passed
• At the previous revision, broke the hook on purpose in 20 places (flag, name and block-id validation, helper-execution and failed-cell checks, SIGINT handling, report de-duplication, raise_for_status, warning suppression, read-only and EROFS handling, block_id, registration at startup); every break failed a test
• black, isort, flake8 clean on the touched files; mypy deepnote_toolkit/ clean
• Full tests/unit: 1461 passed, 9 skipped
• CI across the Python / pyarrow matrix
• Pod test: mount upload on close, project root

🤖 Generated with Claude Code

https://claude.ai/code/session_01B441SpUqRn7QTzcnWxjNa5

[github-actions]

📦 Python package built successfully!

• Version: 2.9.1.dev3+6a04fa8
• Wheel: deepnote_toolkit-2.9.1.dev3+6a04fa8-py3-none-any.whl
• Install:

  pip install "deepnote-toolkit @ https://deepnote-staging-runtime-artifactory.s3.amazonaws.com/deepnote-toolkit-packages/2.9.1.dev3%2B6a04fa8/deepnote_toolkit-2.9.1.dev3%2B6a04fa8-py3-none-any.whl"

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.16%. Comparing base (6523327) to head (dee4404).
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #131      +/-   ##
==========================================
+ Coverage   78.43%   79.16%   +0.72%     
==========================================
  Files         121      122       +1     
  Lines        6920     7013      +93     
  Branches     1005     1017      +12     
==========================================
+ Hits         5428     5552     +124     
+ Misses       1182     1151      -31     
  Partials      310      310              

Flag	Coverage Δ
combined	79.16% <100.00%> (+0.72%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

[deepnote-bot]

🚀 Review App Deployment Started

📝 Description	🌐 Link / Info
🌍 Review application	ra-131
🔑 Sign-in URL	Click to sign-in
📊 Application logs	View logs
🔄 Actions	Click to redeploy
🚀 ArgoCD deployment	View deployment
⏰ Last deployed	2026-11-10 17:36:01 (UTC)
📜 Deployed commit	1fc4eaa51e5c43c733279780cdab051b25e1a780
🛠️ Toolkit version	6a04fa8