Repository navigation
Append to the cache of a dataset without shape without copying it - #8785
Jakob Holz (Jajaho) wants to merge 2 commits into
Conversation
Without a shape, _insert_into_data_dict appended each result with np.append, which copies all data collected so far, so the time of add_result grew with the length of the run. With two arrays of 16384 values per result, add_result took 4 ms at the start and 128 ms after 750 results, for DataSet and DataSetInMem alike. The cache now stores such arrays as views of the beginning of a larger buffer and appends into its spare rows; a full buffer is replaced by one of 1.5 times the rows required. Appends that np.append would promote to another dtype or shape, and arrays of objects, still go through np.append. Arrays passed to add_result are never written to.
|
Some background on where this came up. Our lab's bench software is built on QCoDeS. It records oscilloscope traces (a Red Pitaya scope, 2 × 16384 samples per capture) as runs. Each capture is one add_result on a Measurement: the capture number, the time since the first capture, the time axis and both traces as paramtype="array". A recording is either a set number of captures or runs until the operator stops it, which can mean thousands of captures. It is stored either as a DataSetInMem, exported to NetCDF at the end, or crash-safe as a DataSet. Long recordings got slower and slower: 47 ms per capture at the start, 265 ms after 1300 captures. We traced that to the np.append in _insert_into_data_dict. We looked at declaring a shape with Measurement.set_shapes(), which avoids the copy, but it doesn't fit this case:
So unshaped runs are the normal case for this kind of acquisition. Until a release contains this change, we install the same append as a replacement for _insert_into_data_dict in our own code. We would much rather drop that. A related observation, outside this PR: in the crash-safe DataSet, the cache keeps a second copy of everything already written to the database. With this change it no longer costs time, but it still grows with the run, about 1 GB after 1300 captures of two traces. For these runs we read the data back from the database or the exported NetCDF, and nothing reads the cache. (In a DataSetInMem the cache is the data, of course.) Would an opt-in setting to not keep a cache for a DataSet be welcome as a separate PR? |
There was a problem hiding this comment.
🟡 Changes recommended
The fixed 16-row minimum exceeds the stated 1.5× memory bound and can cause allocation failures, while the ownership regression test does not actually exercise caller-buffer protection.
2 open issues
What changed in this PR
This PR reduces repeated copying when appending results to QCoDeS dataset caches without a declared shape, benefiting both DataSet and DataSetInMem.
Changes:
- Adds reusable, geometrically growing buffers for compatible arrays, retaining
np.appendfallbacks. - Adds regression tests for copy frequency, dtype promotion, and input preservation.
- Documents the performance improvement in a newsfragment.
| File | Description |
|---|---|
tests/dataset/test_data_set_cache.py |
Adds append-performance and data-integrity regression tests. |
src/qcodes/dataset/data_set_cache.py |
Implements buffer-backed appends and tracks buffer ownership. |
docs/changes/newsfragments/8785.improved |
Announces faster appends for datasets without a declared shape. |
Review effort: Balanced
| and buffer.shape[0] >= n_rows | ||
| ): | ||
| buffer = np.empty( | ||
| (max(n_rows + n_rows // 2, 16), *existing_values.shape[1:]), |
| assert data["y"].dtype == np.float64 | ||
| np.testing.assert_array_equal( | ||
| data["y"].ravel(), np.repeat([i if i < 10 else i + 0.5 for i in range(20)], 4) | ||
| ) |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #8785 +/- ##
==========================================
+ Coverage 77.76% 77.77% +0.01%
==========================================
Files 308 308
Lines 32855 32873 +18
==========================================
+ Hits 25549 25567 +18
Misses 7306 7306 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Thanks Jakob Holz (@Jajaho) Looks like a worthwhile fix. I will need a bit of time looking at it. |

For a dataset without a shape,
_insert_into_data_dictappended each result to the cache withnp.append, which copies all data collected so far. The time ofadd_resulttherefore grew with the length of the run, and a run of N results copied O(N²) data. This applies toDataSetandDataSetInMemalike, since both fill the cache.Measured with two array parameters of 16384 values and one setpoint array per result (a scope recording), median time of
add_result:DataSetInMembeforeDataSetbeforeDataSetInMemafterDataSetafterOn a Red Pitaya scope, a recording of 1000 captures went from 7.4 captures/s on average (54 ms per capture at the start, 186 ms after 800) to a constant 19 captures/s, limited by the instrument read.
The cache now stores such arrays as views of the beginning of a larger buffer and appends into its spare rows; a full buffer is replaced by one with 1.5 times the rows required, so the copies amount to a few times the data in total. The arrays returned by
cache.data()have the values, dtype and shapenp.appendgave them.np.appendwould promote to another dtype or another shape after the first axis, and arrays of objects, still go throughnp.append._append_rowsare written to: a module-level map from the id of each buffer to its filled rows (removed byweakref.finalizewhen the buffer is collected) tells them apart from arrays passed toadd_result, which may themselves be views of a larger array.Tests:
test_cache_append_without_shape_does_not_copy_collected_data(DataSetandDataSetInMem): 100 appends copy the data at most 12 times; it fails without the change (99 copies).test_cache_append_without_shape_promotes_dtype_and_keeps_input_arrays: an integer column followed by floats becomes float64 as withnp.append, and an array whose views were passed as values is left unchanged.tests/dataset/test_data_set_cache.py, which compare the cache with the database for many layouts, pass.