Repository navigation
[wip] Optimize test durations - #8634
Jens Hedegaard Nielsen (jenshnielsen) wants to merge 2 commits into
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #8634 +/- ##
==========================================
+ Coverage 77.76% 77.78% +0.01%
==========================================
Files 308 308
Lines 32855 32896 +41
==========================================
+ Hits 25549 25587 +38
- Misses 7306 7309 +3 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
f78cf6c to
e1333b2
Compare
e1333b2 to
e8b6b67
Compare
e8b6b67 to
e7ee265
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Renaming returned structured-array fields mutates cached dtypes and affects subsequent reads across datasets, while malformed headers can retain arbitrarily large byte strings in the cache.
2 open issues
What changed in this PR
This PR targets faster QCoDeS dataset loading by reducing repeated work in SQLite value conversion.
Changes:
- Cache parsed array headers and reuse the optimized array reader for complex values.
- Simplify numeric conversion while preserving integer, infinity, and NaN handling.
- Add converter tests and a release note describing the performance improvement.
| File | Description |
|---|---|
tests/dataset/test_sqlite_converters.py |
Adds array compatibility, invalid-input, complex, and numeric conversion tests. |
src/qcodes/dataset/sqlite/database.py |
Introduces cached array-header parsing and streamlines complex and numeric conversion. |
docs/changes/newsfragments/8634.improved |
Documents faster dataset loading. |
Review effort: Balanced
Parsing the npy header of every stored value dominated the time it takes to load array and complex data. Cache the parsed header, since all rows of a parameter share it, and fall back to numpy's read_array for anything the fast path does not handle.
Use float.is_integer, which is False for NaN and infinity, rather than separately checking for finite values and comparing with the value converted to int.
e7ee265 to
c3323a3
Compare
There was a problem hiding this comment.
🟡 Changes recommended
The fast path bypasses NumPy validation, accepting unsupported minor versions and malformed negative dimensions; both need fallback handling and regression tests.
2 open issues
2 resolved since last review
Review effort: Balanced
| """ | ||
| if len(text) < 12 or not text.startswith(_NPY_MAGIC_PREFIX): | ||
| return None | ||
| major_version = text[6] |
| # later reads, so those are left to numpy. | ||
| if dtype.hasobject or dtype.itemsize == 0 or dtype.names is not None: | ||
| return None | ||
| return shape, fortran_order, dtype |


No description provided.