Skip to content

Append to the cache of a dataset without shape without copying it - #8785

Open
Jakob Holz (Jajaho) wants to merge 2 commits into
microsoft:mainfrom
Jajaho:fix/cache-append-without-copy
Open

Jakob Holz (Jajaho) wants to merge 2 commits into
microsoft:mainfrom
Jajaho:fix/cache-append-without-copy

Conversation

@Jajaho

@Jajaho Jakob Holz (Jajaho) commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

For a dataset without a shape, _insert_into_data_dict appended each result to the cache with np.append, which copies all data collected so far. The time of add_result therefore grew with the length of the run, and a run of N results copied O(N²) data. This applies to DataSet and DataSetInMem alike, since both fill the cache.

Measured with two array parameters of 16384 values and one setpoint array per result (a scope recording), median time of add_result:

Result DataSetInMem before DataSet before DataSetInMem after DataSet after
0 4.4 ms 4.4 ms 0.3 ms 0.8 ms
400 70 ms 76 ms 0.3 ms 0.4 ms
750 128 ms 120 ms 0.3 ms 0.4 ms

On a Red Pitaya scope, a recording of 1000 captures went from 7.4 captures/s on average (54 ms per capture at the start, 186 ms after 800) to a constant 19 captures/s, limited by the instrument read.

The cache now stores such arrays as views of the beginning of a larger buffer and appends into its spare rows; a full buffer is replaced by one with 1.5 times the rows required, so the copies amount to a few times the data in total. The arrays returned by cache.data() have the values, dtype and shape np.append gave them.

  • Appends that np.append would promote to another dtype or another shape after the first axis, and arrays of objects, still go through np.append.
  • Only buffers allocated by _append_rows are written to: a module-level map from the id of each buffer to its filled rows (removed by weakref.finalize when the buffer is collected) tells them apart from arrays passed to add_result, which may themselves be views of a larger array.
  • The shaped path is unchanged.
  • The cache takes up to 1.5 times the memory of its data.

Tests:

  • test_cache_append_without_shape_does_not_copy_collected_data (DataSet and DataSetInMem): 100 appends copy the data at most 12 times; it fails without the change (99 copies).
  • test_cache_append_without_shape_promotes_dtype_and_keeps_input_arrays: an integer column followed by floats becomes float64 as with np.append, and an array whose views were passed as values is left unchanged.
  • The existing tests in tests/dataset/test_data_set_cache.py, which compare the cache with the database for many layouts, pass.

Without a shape, _insert_into_data_dict appended each result with np.append,
which copies all data collected so far, so the time of add_result grew with
the length of the run. With two arrays of 16384 values per result, add_result
took 4 ms at the start and 128 ms after 750 results, for DataSet and
DataSetInMem alike.

The cache now stores such arrays as views of the beginning of a larger
buffer and appends into its spare rows; a full buffer is replaced by one of
1.5 times the rows required. Appends that np.append would promote to another
dtype or shape, and arrays of objects, still go through np.append. Arrays
passed to add_result are never written to.
@Jajaho
Jakob Holz (Jajaho) requested a review from a team as a code owner November 20, 2026 02:30
@Jajaho

Copy link
Copy Markdown
Contributor Author

Some background on where this came up.

Our lab's bench software is built on QCoDeS. It records oscilloscope traces (a Red Pitaya scope, 2 × 16384 samples per capture) as runs. Each capture is one add_result on a Measurement: the capture number, the time since the first capture, the time axis and both traces as paramtype="array". A recording is either a set number of captures or runs until the operator stops it, which can mean thousands of captures. It is stored either as a DataSetInMem, exported to NetCDF at the end, or crash-safe as a DataSet.

Long recordings got slower and slower: 47 ms per capture at the start, 265 ms after 1300 captures. We traced that to the np.append in _insert_into_data_dict.

We looked at declaring a shape with Measurement.set_shapes(), which avoids the copy, but it doesn't fit this case:

  • A recording that runs until stopped has no number of captures to declare.
  • A DataSetInMem that stops before its shape is filled cannot be exported. export() raises ValueError: Dependency 'x' does not define an axis of length 10: found 4 unique values, so an aborted in-memory recording would lose its data.
  • A shape preallocates, and fills with NaN, the full size of the run at the start.

So unshaped runs are the normal case for this kind of acquisition. Until a release contains this change, we install the same append as a replacement for _insert_into_data_dict in our own code. We would much rather drop that.

A related observation, outside this PR: in the crash-safe DataSet, the cache keeps a second copy of everything already written to the database. With this change it no longer costs time, but it still grows with the run, about 1 GB after 1300 captures of two traces. For these runs we read the data back from the database or the exported NetCDF, and nothing reads the cache. (In a DataSetInMem the cache is the data, of course.) Would an opt-in setting to not keep a cache for a DataSet be welcome as a separate PR?

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The fixed 16-row minimum exceeds the stated 1.5× memory bound and can cause allocation failures, while the ownership regression test does not actually exercise caller-buffer protection.

2 open issues
What changed in this PR

This PR reduces repeated copying when appending results to QCoDeS dataset caches without a declared shape, benefiting both DataSet and DataSetInMem.

Changes:

  • Adds reusable, geometrically growing buffers for compatible arrays, retaining np.append fallbacks.
  • Adds regression tests for copy frequency, dtype promotion, and input preservation.
  • Documents the performance improvement in a newsfragment.
File Description
tests/​dataset/​test_data_set_cache.py Adds append-performance and data-integrity regression tests.
src/​qcodes/​dataset/​data_set_cache.py Implements buffer-backed appends and tracks buffer ownership.
docs/​changes/​newsfragments/​8785.improved Announces faster appends for datasets without a declared shape.

Review effort: Balanced

and buffer.shape[0] >= n_rows
):
buffer = np.empty(
(max(n_rows + n_rows // 2, 16), *existing_values.shape[1:]),
assert data["y"].dtype == np.float64
np.testing.assert_array_equal(
data["y"].ravel(), np.repeat([i if i < 10 else i + 0.5 for i in range(20)], 4)
)
@codecov

codecov Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 77.77%. Comparing base (29fe8a6) to head (7739845).
⚠️ Report is 11 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8785      +/-   ##
==========================================
+ Coverage   77.76%   77.77%   +0.01%     
==========================================
  Files         308      308              
  Lines       32855    32873      +18     
==========================================
+ Hits        25549    25567      +18     
  Misses       7306     7306              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@jenshnielsen

Copy link
Copy Markdown
Collaborator

Thanks Jakob Holz (@Jajaho) Looks like a worthwhile fix. I will need a bit of time looking at it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants