Skip to content

[wip] Optimize test durations - #8634

Draft
Jens Hedegaard Nielsen (jenshnielsen) wants to merge 2 commits into
microsoft:mainfrom
jenshnielsen:qcodes_faster_tests
Draft

Jens Hedegaard Nielsen (jenshnielsen) wants to merge 2 commits into
microsoft:mainfrom
jenshnielsen:qcodes_faster_tests

Conversation

@jenshnielsen

Copy link
Copy Markdown
Collaborator

No description provided.

@codecov

codecov Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.23077% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 77.78%. Comparing base (29fe8a6) to head (c3323a3).

Files with missing lines Patch % Lines
src/qcodes/dataset/sqlite/database.py 94.23% 3 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8634      +/-   ##
==========================================
+ Coverage   77.76%   77.78%   +0.01%     
==========================================
  Files         308      308              
  Lines       32855    32896      +41     
==========================================
+ Hits        25549    25587      +38     
- Misses       7306     7309       +3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Renaming returned structured-array fields mutates cached dtypes and affects subsequent reads across datasets, while malformed headers can retain arbitrarily large byte strings in the cache.

2 open issues
What changed in this PR

This PR targets faster QCoDeS dataset loading by reducing repeated work in SQLite value conversion.

Changes:

  • Cache parsed array headers and reuse the optimized array reader for complex values.
  • Simplify numeric conversion while preserving integer, infinity, and NaN handling.
  • Add converter tests and a release note describing the performance improvement.
File Description
tests/​dataset/​test_sqlite_converters.py Adds array compatibility, invalid-input, complex, and numeric conversion tests.
src/​qcodes/​dataset/​sqlite/​database.py Introduces cached array-header parsing and streamlines complex and numeric conversion.
docs/​changes/​newsfragments/​8634.improved Documents faster dataset loading.

Review effort: Balanced

Comment thread src/qcodes/dataset/sqlite/database.py Resolved
Comment thread src/qcodes/dataset/sqlite/database.py Resolved
Parsing the npy header of every stored value dominated the time it
takes to load array and complex data. Cache the parsed header, since
all rows of a parameter share it, and fall back to numpy's read_array
for anything the fast path does not handle.
Use float.is_integer, which is False for NaN and infinity, rather
than separately checking for finite values and comparing with the
value converted to int.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The fast path bypasses NumPy validation, accepting unsupported minor versions and malformed negative dimensions; both need fallback handling and regression tests.

2 open issues
2 resolved since last review

Review effort: Balanced

"""
if len(text) < 12 or not text.startswith(_NPY_MAGIC_PREFIX):
return None
major_version = text[6]
# later reads, so those are left to numpy.
if dtype.hasobject or dtype.itemsize == 0 or dtype.names is not None:
return None
return shape, fortran_order, dtype

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants